Mapping-based DAG tracing sampling method, system, device and storage medium

Abstract

The present invention discloses a mapping-based DAG tracing sampling method, system, device and storage medium, the method comprising: obtaining the input field and output field of the operator node, generating a mapping map; obtaining all the parent nodes of the operator node and storing them as DAG; using a recursive algorithm through DAG to query in sequence whether the parent node is the original table, if so, searching the corresponding original table field through the mapping map, and obtaining the corresponding sample data through the database according to the original table field; returning the obtained sample data and the mapping map. The method proposed by the present invention can find the source more correctly through DAG tracing when making field mapping recommendations, and can quickly and accurately obtain the original table sample data for recommendation; no manual filling is required, avoiding data disorder caused by multiple data tables, high accuracy, and improved user experience.

Description

Technical Field

[0001] The present invention relates to the field of data governance technology, and in particular to a mapping-based DAG tracing sampling method, system, device and storage medium. Background Art

[0002] In data governance, many intermediate tables, dependencies, and mapping relationships are generated from the original table to the corresponding target table, forming a DAG graph. When building the DAG, when an operator needs sample data from the original table, field mapping recommendations are made. Since the intermediate table has not been governed and has no corresponding data, recursive tracing is required to find the relevant data of the original table for recommended mapping. The existing method is to manually carry the relevant information of the original table in the data for governance during the DAG construction process, but this method will cause data disorder when there are many data tables, and the accuracy of manual information is low. Summary of the invention

[0003] In view of the technical problems that the above-mentioned method of manually carrying information for data management has low accuracy and may cause data disorder, the present invention proposes a mapping-based DAG tracing sampling method, system, device and storage medium.

[0004] In a first aspect, an embodiment of the present application provides a mapping-based DAG tracing sampling method, comprising:

[0005] Map generation step: Get the input field and output field of the operator node and generate a mapping map;

[0006] DAG acquisition step: acquiring all parent nodes of the operator node and storing them as DAG;

[0007] Parent node query step: using the DAG to use a recursive algorithm to query in sequence whether the parent node is the original table, if so, searching the corresponding original table field through the mapping map, and obtaining the corresponding sample data through the database according to the original table field;

[0008] Data return step: return the acquired sample data and the mapping map.

[0009] In the above-mentioned mapping-based DAG tracing sampling method, the parent node query step further includes:

[0010] Mapping operator query step: If not, query whether the type of the parent node is a mapping operator. If the type is the mapping operator, and the output field of the field mapping stored in the database is the same as the key value in the mapping map, then store the input field of the field mapping stored in the database in the mapping map.

[0011] In the above-mentioned mapping-based DAG tracing sampling method, the mapping map generation step includes:

[0012] Field acquisition step: by connecting to the database, obtain the input field and the output field of the operator node and store them in Map<String,List> middle;

[0013] Field sorting step: sorting the input fields and the output fields of the operator node, and storing them in inputMap and OutputMap respectively, to generate the mapping map.

[0014] In the above-mentioned mapping-based DAG tracing sampling method, in the mapping map, the key is the field id and the value is "table type#field id".

[0015] The above-mentioned mapping-based DAG tracing sampling method further includes:

[0016] Field mapping recommendation step: perform field mapping recommendation based on the acquired sample data.

[0017] In a second aspect, an embodiment of the present application provides a mapping-based DAG tracing sampling system, including:

[0018] Map generation unit: obtains the input field and output field of the operator node and generates a map;

[0019] DAG acquisition unit: acquires all parent nodes of the operator node and stores them as DAG;

[0020] A parent node query unit: uses a recursive algorithm to query the parent node in turn through the DAG to see if it is an original table. If so, the corresponding original table field is searched through the mapping map, and the corresponding sample data is obtained through the database according to the original table field.

[0021] Data return unit: returns the acquired sample data and the mapping map;

[0022] Field mapping recommendation unit: performs field mapping recommendation based on the acquired sample data.

[0023] In the above-mentioned mapping-based DAG tracing sampling system, the parent node query unit further includes:

[0024] Mapping operator query module: If the parent node is not the original table, query whether the type of the parent node is a mapping operator. If the type is the mapping operator, and the output field of the field mapping stored in the database is the same as the key value in the mapping map, then store the input field of the field mapping stored in the database in the mapping map.

[0025] In the above-mentioned mapping-based DAG traceability sampling system, the mapping map generation unit includes:

[0026] Field acquisition module: by connecting to the database, obtain the input field and the output field of the operator node and store them in Map<String,List> middle;

[0027] Field sorting module: sorts the input fields and the output fields of the operator node, and stores them in inputMap and OutputMap respectively, to generate the mapping map.

[0028] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the mapping-based DAG tracing sampling method as described in the first aspect above is implemented.

[0029] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the mapping-based DAG provenance sampling method as described in the first aspect above.

[0030] Compared with the prior art, the advantages and positive effects of the present invention are:

[0031] 1. When making field mapping recommendations, the method proposed in the present invention can more accurately find the source through DAG tracing, and can quickly and accurately obtain the original table sample data for recommendation;

[0032] 2. No need to fill in manually, avoiding data disorder caused by multiple data tables, high accuracy, improved user experience, and enhanced data management capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A schematic diagram of the steps of a mapping-based DAG tracing sampling method provided by the present invention;

[0034] Figure 2 The present invention provides Figure 1 Schematic diagram of the process of step S1;

[0035] Figure 3A schematic diagram of a process flow of a mapping-based DAG tracing sampling method provided by the present invention;

[0036] Figure 4 A flowchart of a specific embodiment of a mapping-based DAG tracing sampling method provided by the present invention;

[0037] Figure 5 A framework diagram of a mapping-based DAG tracing sampling system provided by the present invention;

[0038] Figure 6 A framework diagram of a computer device according to an embodiment of the present application.

[0039] Wherein, the accompanying drawings are marked as follows:

[0040] 1. Mapping map generation unit; 11. Field acquisition module; 12. Field sorting module; 2. DAG acquisition unit; 3. Parent node query unit; 31. Mapping operator query module; 4. Data return unit; 5. Field mapping recommendation unit; 81. Processor; 82. Memory; 83. Communication interface; 80. Bus. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0042] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0043] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0044] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0045] The present invention is described in detail below in conjunction with the various embodiments shown in the accompanying drawings, but it should be noted that these embodiments are not limitations of the present invention, and any equivalent transformations or substitutions in functions, methods, or structures made by ordinary technicians in the field based on these embodiments are all within the scope of protection of the present invention.

[0046] Before elaborating on each embodiment of the present invention in detail, the core inventive concept of the present invention is summarized and elaborated in detail through the following several embodiments.

[0047] The present invention proposes a method for performing DAG recursion to find sample data for field recommendation based on field mapping between tables. Through this method, the original table sample data can be quickly and accurately obtained for recommendation.

[0048] Embodiment 1:

[0049] Figure 1 A schematic diagram of the steps of a mapping-based DAG tracing sampling method provided by the present invention. Figure 1 As shown, this embodiment discloses a specific implementation of a mapping-based DAG tracing sampling method (hereinafter referred to as "method").

[0050] Specifically, the method disclosed in this embodiment mainly includes the following steps:

[0051] Step S1: Get the input field and output field of the operator node and generate a mapping map;

[0052] like Figure 2 As shown, step S1 specifically includes the following contents:

[0053] Step S11: By connecting to the database, obtain the input field and the output field of the operator node and store them in Map<String,List> middle;

[0054] Step S12: Arrange the input fields and output fields of the operator node, and store them in inputMap and OutputMap respectively to generate the mapping map. In the mapping map, the key is the field id, and the value is "table type#field id".

[0055] Step S2: Obtain all parent nodes of the operator node and store them as DAG;

[0056] Step S3: using the DAG and a recursive algorithm to query in turn whether the parent node is the original table, if so, searching for the corresponding original table field through the mapping map, and obtaining the corresponding sample data through the database according to the original table field;

[0057] If not, query whether the type of the parent node is a mapping operator. If the type is the mapping operator, and the output field of the field mapping stored in the database is the same as the key value in the mapping map, then store the input field of the field mapping stored in the database in the mapping map.

[0058] Step S4: Return the acquired sample data and the mapping map.

[0059] Step S5: Recommend field mapping based on the acquired sample data.

[0060] The purpose of the present invention is to quickly and accurately obtain the original table sample data for recommendation when making field mapping recommendations. Figure 3As shown, the present invention proposes a method for recursively finding sample data for field recommendation based on DAG field mapping between tables, which relies on a database storing mapping and DAG. The process is as follows:

[0061] 1. Assume that there is an operator node in the database. The input and output of this node have n and m fields respectively, and no matching has been performed yet;

[0062] 2. Connect to the database, obtain the input and output fields of the node and store them in the Map<String,List> In , get all the parent nodes of the node and store them as DAG;

[0063] 3. Arrange the input and output fields of the operator and store them in inputMap and OutputMap respectively, generate a mapping map, the key is the field id, and the value is "table type#field id";

[0064] 4. Get the parent node list of the previous level from DAG and take one of the nodes x;

[0065] 5. If x is the original table, the corresponding original table field will be searched through the mapping map to obtain the sample data and execute the next step. Otherwise, check whether the type of x is a mapping operator. If so, the field mapping input field stored in the database will be compared with the output field and the same key value in the mapping map and stored in the mapping map;

[0066] 6. If the parent node list has not been queried, execute steps 4-5, otherwise return the mapping map and sample data list;

[0067] 7. Recommend inter-operator fields based on sample data.

[0068] Please refer to the following Figure 4 The application process of this method is described as follows in conjunction with a specific embodiment:

[0069] like Figure 4 As shown in the figure, taking DAG as an example, suppose there is a mapping operator node x, with input fields: "name", "sex"; output fields: "person_name", "person_sex". Now you need to query the database and match the input field "name" with the output field "person_name" according to the sample data content.

[0070] The specific implementation method is carried out in the following steps:

[0071] 1. Connect to the database, obtain the input and output fields of the node and store them in the Map<String,List> In the above example, all parent nodes of the node are obtained and stored as DAG.

[0072] 2. Arrange the input and output fields of the operator and store them in inputMap and OutputMap respectively. Generate a mapping map with the key as the field id and the value as "table type#field id".

[0073] 3. Get the parent node list of the previous level from DAG, take the previous node person. If person is neither the original table nor the mapping operator, then continue to get the node list of the previous level, take a node y, and repeat this step.

[0074] 4. y is the mapping operator. The field mapping stored in the database is: "teacher_name"->"name", "teacher_sex"->"sex". The input field is stored in the mapping map by comparing the output field with the same key value in the mapping map. At this time, the map is "teacher_name"->"name", "teacher_sex"->"sex". Continue to get the node list of the previous level, take a node teacher_1, and repeat this step.

[0075] 5. After y and the predecessor node have executed the above steps, they return to the parent node list of person and find that there is still node z that has not been executed. Then repeat step 4 until the original table student is reached. Student is the original table. Find the corresponding original table field through mapping map, get the sample data, find the corresponding mapping in map: "student_name"->"name", "student_sex"->"sex", and get the sample data.

[0076] 6. Recommend field mapping based on the sample data obtained.

[0077] Embodiment 2:

[0078] In combination with the mapping-based DAG provenance sampling method disclosed in the first embodiment, this embodiment discloses a specific implementation example of a mapping-based DAG provenance sampling system (hereinafter referred to as “system”).

[0079] Reference Figure 5 As shown, the system comprises:

[0080] Map generation unit 1: obtains the input field and output field of the operator node and generates a mapping map;

[0081] DAG acquisition unit 2: acquires all parent nodes of the operator node and stores them as DAG;

[0082] Parent node query unit 3: using the DAG to use a recursive algorithm to query in sequence whether the parent node is the original table, if so, searching for the corresponding original table field through the mapping map, and obtaining the corresponding sample data through the database according to the original table field;

[0083] Data returning unit 4: returns the acquired sample data and the mapping map;

[0084] Field mapping recommendation unit 5: performs field mapping recommendation according to the acquired sample data.

[0085] Specifically, the mapping map generation unit 1 includes:

[0086] Field acquisition module 11: by connecting to the database, obtain the input field and the output field of the operator node and store them in Map<String,List> middle;

[0087] Field sorting module 12: sorts the input fields and the output fields of the operator node, and stores them in inputMap and OutputMap respectively, to generate the mapping map.

[0088] Specifically, the parent node query unit 3 also includes:

[0089] Mapping operator query module 31: If the parent node is not the original table, query whether the type of the parent node is a mapping operator. If the type is the mapping operator, and the output field of the field mapping stored in the database is the same as the key value in the mapping map, then the input field of the field mapping stored in the database is stored in the mapping map.

[0090] The technical solutions of the mapping-based DAG traceability sampling system disclosed in this embodiment and the mapping-based DAG traceability sampling method disclosed in the first embodiment are the same as those in the first embodiment, and will not be repeated here.

[0091] Embodiment three:

[0092] Combination Figure 6 As shown, this embodiment discloses a specific implementation of a computer device. The computer device may include a processor 81 and a memory 82 storing computer program instructions.

[0093] Specifically, the processor 81 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0094] Among them, the memory 82 may include a large-capacity memory for data or instructions. By way of example and not limitation, the memory 82 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 82 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 82 may be inside or outside the data processing device. In a specific embodiment, the memory 82 is a non-volatile memory. In a specific embodiment, the memory 82 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (Programmable Read-Only Memory, PROM for short), an erasable PROM (Erasable ProgrammableRead-Only Memory, EPROM for short), an electrically erasable PROM (Electrically Erasable ProgrammableRead-Only Memory, EEPROM for short), an electrically alterable ROM (Electrically Alterable Read-Only Memory, EAROM for short) or a flash memory (FLASH) or a combination of two or more of these. Under appropriate circumstances, the RAM can be a static random access memory (SRAM) or a dynamic random access memory (DRAM), wherein the DRAM can be a fast page mode dynamic random access memory (FPMDRAM), an extended data output dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0095] The memory 82 may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 81 .

[0096] The processor 81 reads and executes computer program instructions stored in the memory 82 to implement any one of the mapping-based DAG provenance sampling methods in the above embodiments.

[0097] In some of these embodiments, the computer device may further include a communication interface 83 and a bus 80. Figure 6 As shown, the processor 81, the memory 82, and the communication interface 83 are connected via a bus 80 and communicate with each other.

[0098] The communication interface 83 is used to implement communication between the modules, devices, units and / or equipment in the embodiment of the present application. The communication port 83 can also implement data communication with other components such as: external devices, image / data acquisition equipment, databases, external storage, and image / data processing workstations.

[0099] The bus 80 includes hardware, software or both, and couples the components of the computer device to each other. The bus 80 includes but is not limited to at least one of the following: a data bus, an address bus, a control bus, an expansion bus, and a local bus. By way of example and not limitation, bus 80 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, bus 80 may include one or more buses. Although embodiments of the present application describe and illustrate a particular bus, the present application contemplates any suitable bus or interconnect.

[0100] In addition, in combination with the mapping-based DAG tracing sampling method in the above embodiments, the embodiments of the present application may provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any one of the mapping-based DAG tracing sampling methods in the above embodiments is implemented.

[0101] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0102] To sum up, the beneficial effect of the present invention lies in that the method proposed in the present invention can find the source more correctly through DAG tracing when making field mapping recommendations, and can quickly and accurately obtain the original table sample data for recommendation; no manual filling is required, which avoids data disorder caused by many data tables, has high accuracy, improves user experience, and improves data governance capabilities.

[0103] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A mapping-based DAG tracing sampling method, characterized in that: include: Map generation step: Get the input field and output field of the operator node and generate a mapping map; DAG acquisition step: acquiring all parent nodes of the operator node and storing them as DAG; Parent node query step: using the DAG to use a recursive algorithm to query in sequence whether the parent node is the original table, if so, searching the corresponding original table field through the mapping map, and obtaining the corresponding sample data through the database according to the original table field; Data returning step: returning the acquired sample data and the mapping map; Wherein, the parent node query step further includes: Mapping operator query step: if not, query whether the type of the parent node is a mapping operator; if the type is the mapping operator, and the output field of the field mapping stored in the database is the same as the key value in the mapping map, store the input field of the field mapping stored in the database into the mapping map; The mapping map generation step includes: Field acquisition step: by connecting to the database, obtain the input field and the output field of the operator node and store the Map<String,List> middle; Field sorting step: sorting the input fields and the output fields of the operator node, and storing them in inputMap and OutputMap respectively, to generate the mapping map.

2. The mapping-based DAG tracing sampling method according to claim 1 is characterized in that: In the mapping map, the key is the field id and the value is "table type#field id".

3. The mapping-based DAG tracing sampling method according to claim 1 is characterized in that: Also includes: Field mapping recommendation step: perform field mapping recommendation based on the acquired sample data.

4. A mapping-based DAG tracing sampling system, characterized in that: include: Map generation unit: obtains the input field and output field of the operator node and generates a map; DAG acquisition unit: acquires all parent nodes of the operator node and stores them as DAG; A parent node query unit: uses a recursive algorithm to query the parent node in turn through the DAG to see if it is an original table. If so, the corresponding original table field is searched through the mapping map, and the corresponding sample data is obtained through the database according to the original table field. Data return unit: returns the acquired sample data and the mapping map; Field mapping recommendation unit: recommends field mapping according to the acquired sample data; Wherein, the parent node query unit also includes: Mapping operator query module: if the parent node is not the original table, query whether the type of the parent node is a mapping operator; if the type is the mapping operator, and the output field of the field mapping stored in the database is the same as the key value in the mapping map, store the input field of the field mapping stored in the database into the mapping map; Wherein, the mapping map generating unit includes: Field acquisition module: by connecting to the database, obtain the input field and the output field of the operator node and store them in Map<String,List> middle; Field sorting module: sorts the input fields and the output fields of the operator node, and stores them in inputMap and OutputMap respectively, to generate the mapping map.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the mapping-based DAG tracing sampling method according to any one of claims 1 to 3 is implemented.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the mapping-based DAG tracing sampling method according to any one of claims 1 to 3 is implemented.

Images