Field blood relationship acquisition method and system, electronic equipment and storage medium

Abstract

The invention relates to a field blood relationship acquisition method and system, electronic equipment and a storage medium. The method comprises the steps that a query statement is received, and a logic plan is obtained by analyzing the query statement; the logic plan is traversed to obtain source table information, target table information and a field conversion relation, the source table information comprises a source table field expression ID and a source table field name, and the target table information comprises a target table field expression ID and a target table field name; the field conversion relationship is a conversion relationship between a target table field expression ID and a source table field expression ID in the logic plan; the field blood relationship information is obtained according to the source table information, the target table information, the field conversion relation, a first conversion rule and a second conversion rule, the first conversion rule is a simple indirect conversion relation, and the second conversion rule is a complex indirect conversion relation. By adopting the method, the accuracy of the field blood relationship information can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a field lineage acquisition method, system, electronic device and storage medium. Background Art

[0002] With the development of computers, the amount of data is also growing. When processing and handling large amounts of data, if there is a problem with the data, it is necessary to verify the data at all levels, which will lead to a huge waste of resources and time. Therefore, it is necessary to perform lineage analysis on data fields to reduce the time of data verification.

[0003] In the related art, it is necessary to analyze Sql (Structured Query Language) to obtain the corresponding abstract syntax analysis tree, and then recursively traverse the abstract syntax analysis tree to obtain the data source field of the final display field. However, when extracting field information through the abstract syntax analysis tree, it is impossible to verify whether the information actually exists, which reduces the accuracy and reliability of the data field lineage analysis. In addition, the related art must fully define the lexical analyzer and syntax analyzer, and its development cost is often high and the scalability is poor. Moreover, the current field-level lineage cannot handle the field lineage relationship of the view.

[0004] Therefore, there is an urgent need to propose a field lineage acquisition method, system, electronic device and storage medium that can improve the accuracy of field lineage information. Summary of the Invention

[0005] Based on this, it is necessary to provide a field lineage acquisition method, system, electronic device and storage medium that can improve the accuracy of field lineage information in response to the above technical problems.

[0006] In a first aspect, a method for obtaining field lineage is provided, the method comprising:

[0007] Receiving a query statement and obtaining a logical plan by parsing the query statement;

[0008] Traversing the logical plan to obtain source table information, target table information, and a field conversion relationship, wherein the source table information includes a source table field expression ID and a corresponding source table field name, the target table information includes a target table field expression ID and a corresponding target table field name, and the field conversion relationship is a conversion relationship between the target table field expression ID and the source table field expression ID in the logical plan;

[0009] Field lineage information is obtained according to the source table information, target table information, field conversion relationship, first conversion rule and second conversion rule, wherein the first conversion rule is a simple indirect conversion rule and the second conversion rule is a complex indirect conversion rule.

[0010] In one embodiment, traversing the logical plan to obtain source table information, target table information, and field conversion relationships includes:

[0011] Traversing the logical plan to obtain at least one level of logical nodes, wherein the at least one level of logical nodes includes a start node, an intermediate node, and an end node, the start node includes source table information, the intermediate node includes intermediate table information, and the end node includes target table information;

[0012] Sequentially parsing the logic nodes of the at least one level to obtain field information of each logic node, wherein the field information includes a field expression ID and a field name;

[0013] The field information corresponding to each adjacent logical node is associated and matched to obtain the field conversion relationship.

[0014] In one embodiment, obtaining the field lineage information according to the source table information, the target table information, the field conversion relationship, the first conversion rule, and the second conversion rule includes:

[0015] Traversing the target table information and determining whether there is a first remaining element, wherein the first remaining element is a target table field expression ID that does not have the field conversion relationship and a corresponding target table field name;

[0016] In response to the presence of a first remaining element in the target table information, searching for corresponding intermediate table information according to the target table field expression ID of the first remaining element and according to the first conversion rule and the second conversion rule, wherein the intermediate table information includes the intermediate table field expression ID and the corresponding intermediate table field name;

[0017] According to the intermediate table field expression ID, determining whether the source table information includes the intermediate table information;

[0018] In response to the source table information including the intermediate table information, the field lineage information is obtained according to the intermediate table information and the target table information.

[0019] In one embodiment, searching for corresponding intermediate table information according to the first conversion rule and the second conversion rule based on the target table field expression ID of the first remaining element includes:

[0020] Determine corresponding first intermediate table information according to the first conversion rule and the target table field expression ID of the first remaining element;

[0021] According to the second conversion rule and the target table field expression ID of the first remaining element, corresponding second intermediate table information is determined.

[0022] In one embodiment, before determining whether the source table information includes the intermediate table information based on the intermediate table field expression ID, the method further includes:

[0023] Combining the first intermediate table information, the second intermediate table information, and the target table information to obtain comprehensive table information;

[0024] Determine whether there is a second remaining element in the comprehensive table information, wherein the second remaining element is a field expression ID and a field name that does not have a field conversion relationship, a first conversion rule, and a second conversion rule;

[0025] In response to the absence of the second remaining element in the comprehensive table information, the next step is to determine whether the source table information includes intermediate table information based on the intermediate table field expression ID.

[0026] In one embodiment, after obtaining the field lineage information, the method further includes:

[0027] updating the sensitive table information according to the field lineage information to obtain new sensitive table information, wherein the new sensitive table information includes sensitive field names;

[0028] Matching the field lineage information with the new sensitive table information to obtain sensitive fields in the field lineage information, wherein the field lineage information includes the source table field name, the intermediate table field name, and the target table field name;

[0029] The sensitive fields in the field lineage information are desensitized, wherein the desensitization includes at least one of rounding, offsetting, rearrangement, encryption, average value, invalidation, random value and data replacement.

[0030] In one embodiment, the method is applied to a field lineage acquisition system, the system including a message queue module, a metadata service, and a computing engine module. After obtaining the field lineage information, the method further includes:

[0031] Writing the field lineage information into the message queue module;

[0032] Consuming the field lineage information in the message queue module through the metadata service to write the field lineage information into the field lineage acquisition system in real time;

[0033] receiving a bloodline link location request, and determining the bloodline link according to the writing method of the bloodline information in the field;

[0034] In response to the fact that the writing mode of the field lineage information is real-time writing, it is determined that the lineage links are, in order of writing, the computing engine module, the message queue module, and the metadata service.

[0035] In a second aspect, a field lineage acquisition system is provided, including a lineage application module, a computing engine module, a first physical machine, a second physical machine, and a cloud container. The first physical machine includes a message queue module and a database module. The second physical machine includes a search engine module. The cloud container includes a front-end service, a lineage plug-in, a metadata service, a query service, and a registry module.

[0036] The output end of the computing engine module is connected to the first input end of the message queue module;

[0037] The first input end of the metadata service is connected to the first output end of the message queue module, the second input end of the metadata service is connected to the output end of the front-end service, and the third input end of the metadata service is connected to the output end of the query service;

[0038] The first output terminal of the metadata service is connected to the second input terminal of the message queue module, the second output terminal of the metadata service is connected to the input terminal of the search engine module, and the third output terminal of the metadata service is connected to the input terminal of the database module;

[0039] The second output end of the message queue module is connected to the input end of the bloodline plug-in, and the third output end of the message queue module is connected to the input end of the registry module;

[0040] The output end of the bloodline plug-in is connected to the input end of the bloodline application module.

[0041] In a third aspect, an electronic device is provided, comprising one or more processors; and a memory associated with the one or more processors, the memory being used to store program instructions, which, when read and executed by the one or more processors, execute the steps of the method described in any one of the first aspects above.

[0042] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in any one of the above-mentioned first aspects are performed.

[0043] In a fifth aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the steps of the method described in any one of the first aspects above.

[0044] The above-mentioned field lineage acquisition method, system, electronic device and storage medium obtain a logical plan by parsing the query statement; traverse the logical plan to obtain source table information, target table information and field conversion relationship to obtain part of the compiled field lineage information; obtain field lineage information based on the source table information, target table information, field conversion relationship, first conversion rule and second conversion rule to obtain all field lineage information, thereby improving the accuracy of the field lineage information. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A flowchart of a method for obtaining field lineage in one embodiment;

[0046] Figure 2 A specific flow chart of the steps for obtaining field lineage information in one embodiment;

[0047] Figure 3 An example diagram of a first conversion rule and a second conversion rule in one embodiment;

[0048] Figure 4 A schematic diagram of a process for obtaining sensitive fields in field lineage information in one embodiment;

[0049] Figure 5 A connection diagram of a field lineage acquisition system in one embodiment;

[0050] Figure 6 FIG. 1 is a diagram showing the internal structure of an electronic device in one embodiment. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0052] Example 1

[0053] In one embodiment, Figure 1 As shown, a method for obtaining field lineage is provided, the method comprising:

[0054] Receiving a query statement and obtaining a logical plan by parsing the query statement;

[0055] Traversing the logical plan to obtain source table information, target table information, and a field conversion relationship, wherein the source table information includes a source table field expression ID and a corresponding source table field name, the target table information includes a target table field expression ID and a corresponding target table field name, and the field conversion relationship is a conversion relationship between the target table field expression ID and the source table field expression ID in the logical plan;

[0056] Field lineage information is obtained according to the source table information, target table information, field conversion relationship, first conversion rule and second conversion rule, wherein the first conversion rule is a simple indirect conversion rule and the second conversion rule is a complex indirect conversion rule.

[0057] Specifically, a logical plan is obtained by parsing the query statement; the source table information, target table information and field conversion relationship are traversed through the logical plan to obtain the compiled partial field lineage information; the field lineage information is obtained according to the source table information, target table information, field conversion relationship, first conversion rule and second conversion rule to obtain all field lineage information, thereby improving the accuracy of the field lineage information.

[0058] In one embodiment, traversing the logical plan to obtain source table information, target table information, and field conversion relationships includes:

[0059] Traversing the logical plan to obtain at least one level of logical nodes, wherein the at least one level of logical nodes includes a start node, an intermediate node, and an end node, the start node includes source table information, the intermediate node includes intermediate table information, and the end node includes target table information;

[0060] Sequentially parsing the logic nodes of the at least one level to obtain field information of each logic node, wherein the field information includes a field expression ID and a field name;

[0061] The field information corresponding to each adjacent logical node is associated and matched to obtain the field conversion relationship.

[0062] Specifically, the source table information and the target table information are obtained by traversing the logical plan, and then a complex matching algorithm is used to obtain the field conversion relationship from the target table information to the source table information in the logical plan.

[0063] In one embodiment, Figure 2 As shown, the field lineage information is obtained according to the source table information, the target table information, the field conversion relationship, the first conversion rule, and the second conversion rule, including:

[0064] Traversing the target table information and determining whether there is a first remaining element, wherein the first remaining element is a target table field expression ID that does not have the field conversion relationship and a corresponding target table field name;

[0065] In response to the presence of a first remaining element in the target table information, searching for corresponding intermediate table information according to the target table field expression ID of the first remaining element and according to the first conversion rule and the second conversion rule, wherein the intermediate table information includes the intermediate table field expression ID and the corresponding intermediate table field name;

[0066] According to the intermediate table field expression ID, determining whether the source table information includes the intermediate table information;

[0067] In response to the source table information including the intermediate table information, the field lineage information is obtained according to the intermediate table information and the target table information.

[0068] Specifically, in this embodiment, "not having XXX" means that XXX is not found. In response to the absence of the first remaining element in the target table information, field lineage information is obtained based on the source table information, the target table information, and the field conversion relationship. In response to the absence of intermediate table information in the source table information, whether the source table information includes intermediate table information is again determined based on the intermediate table field expression ID. Based on the first conversion rule and the second conversion rule, the intermediate node of the first remaining element is found to obtain the field lineage information of the first remaining element.

[0069] In one embodiment, Figure 3 As shown, searching for corresponding intermediate table information according to the first conversion rule and the second conversion rule based on the target table field expression ID of the first remaining element includes:

[0070] Determine corresponding first intermediate table information according to the first conversion rule and the target table field expression ID of the first remaining element;

[0071] According to the second conversion rule and the target table field expression ID of the first remaining element, corresponding second intermediate table information is determined.

[0072] In one embodiment, before determining whether the source table information includes the intermediate table information based on the intermediate table field expression ID, the method further includes:

[0073] Combining the first intermediate table information, the second intermediate table information, and the target table information to obtain comprehensive table information;

[0074] Determine whether there is a second remaining element in the comprehensive table information, wherein the second remaining element is a field expression ID and a field name that does not have a field conversion relationship, a first conversion rule, and a second conversion rule;

[0075] In response to the absence of the second remaining element in the comprehensive table information, the next step is to determine whether the source table information includes intermediate table information based on the intermediate table field expression ID.

[0076] Specifically, in response to the existence of the second remaining element in the comprehensive table information, the next step is to traverse the target table information and determine whether the first remaining element exists.

[0077] In one embodiment, Figure 4 As shown, after obtaining the field lineage information, the method further includes:

[0078] updating the sensitive table information according to the field lineage information to obtain new sensitive table information, wherein the new sensitive table information includes sensitive field names;

[0079] Matching the field lineage information with the new sensitive table information to obtain sensitive fields in the field lineage information, wherein the field lineage information includes the source table field name, the intermediate table field name, and the target table field name;

[0080] The sensitive fields in the field lineage information are desensitized, wherein the desensitization includes at least one of rounding, offsetting, rearrangement, encryption, average value, invalidation, random value and data replacement.

[0081] Specifically, before writing the field lineage relationship into the field lineage acquisition system, the old sensitive table information is updated first, and the sensitive fields in the field lineage information are judged based on the new sensitive table information to maintain the security of the field lineage information and realize the application of field lineage to the field of sensitive table identification.

[0082] In one embodiment, the method is applied to a field lineage acquisition system, the system including a message queue module, a metadata service, and a computing engine module. After obtaining the field lineage information, the method further includes:

[0083] Writing the field lineage information into the message queue module;

[0084] Consuming the field lineage information in the message queue module through the metadata service to write the field lineage information into the field lineage acquisition system in real time;

[0085] receiving a bloodline link location request, and determining the bloodline link according to the writing method of the bloodline information in the field;

[0086] In response to the fact that the writing mode of the field lineage information is real-time writing, it is determined that the lineage links are, in order of writing, the computing engine module, the message queue module, and the metadata service.

[0087] Specifically, the writing methods of field lineage information include real-time writing and offline writing. By storing the acquired field lineage information into the field lineage acquisition system, the problem of high real-time requirements and large data volume of field lineage is solved.

[0088] Real-time writing: After the task runs, the acquired field lineage information is packaged into a message body and written to the message queue module. The metadata service then consumes the message body data in the message queue to store the field lineage information in the second physical machine and database module of the field lineage acquisition system. The lineage chain is: compute engine module - message queue module - metadata service.

[0089] For offline writing, the lineage system obtains table structure data from the data warehouse through active detection and stores it as data entities in the second physical machine and database module. At this time, the lineage link is: query service - metadata service.

[0090] It should be understood that although Figure 1 、 Figure 2 、 Figure 4 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 、 Figure 2 、 Figure 4 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0091] Example 2

[0092] In one embodiment, Figure 5 As shown, a field lineage acquisition system is provided, including a lineage application module, a computing engine module, a first physical machine, a second physical machine, and a cloud container. The first physical machine includes a message queue module and a database module, the second physical machine includes a search engine module, and the cloud container includes a front-end service, a lineage plug-in, a metadata service, a query service, and a registry module.

[0093] The output end of the computing engine module is connected to the first input end of the message queue module;

[0094] The first input end of the metadata service is connected to the first output end of the message queue module, the second input end of the metadata service is connected to the output end of the front-end service, and the third input end of the metadata service is connected to the output end of the query service;

[0095] The first output terminal of the metadata service is connected to the second input terminal of the message queue module, the second output terminal of the metadata service is connected to the input terminal of the search engine module, and the third output terminal of the metadata service is connected to the input terminal of the database module;

[0096] The second output end of the message queue module is connected to the input end of the bloodline plug-in, and the third output end of the message queue module is connected to the input end of the registry module;

[0097] The output end of the bloodline plug-in is connected to the input end of the bloodline application module.

[0098] Specifically, the search engine module is deployed on a self-operated physical machine (the second physical machine). The lineage application module is used to apply field lineage information to sensitive field analysis and data development efficiency improvement. The computing engine module, its field lineage information comes from the query statement. The query statement is run through the computing engine module. At the same time, through the plug-in, after the query statement is run, the actual field lineage information is obtained through the field lineage acquisition algorithm. The message queue module is deployed on the physical machine (the first physical machine) and is responsible for receiving real-time field lineage information. The front-end service is deployed in the cloud container to provide a web page viewing service for field lineage information. The lineage plug-in is used to respond to field lineage information and trigger specific execution actions. The metadata service can modify the object information on the metadata graph on the one hand, and obtain the underlying storage model that constitutes the lineage information graph on the other hand. At the same time, it can consume messages in the message queue module externally, and can also obtain all table information in the data warehouse through the query service.

[0099] For the specific definition of the field lineage acquisition system, please refer to the definition of the field lineage acquisition method above, which will not be repeated here. Each module in the above-mentioned field lineage acquisition system can be implemented in whole or in part by software, hardware, or a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the electronic device in the form of hardware, or can be stored in the memory of the electronic device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0100] Example 3

[0101] In one embodiment, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0102] Receiving a query statement and obtaining a logical plan by parsing the query statement;

[0103] Traversing the logical plan to obtain source table information, target table information, and a field conversion relationship, wherein the source table information includes a source table field expression ID and a corresponding source table field name, the target table information includes a target table field expression ID and a corresponding target table field name, and the field conversion relationship is a conversion relationship between the target table field expression ID and the source table field expression ID in the logical plan;

[0104] Field lineage information is obtained according to the source table information, target table information, field conversion relationship, first conversion rule and second conversion rule, wherein the first conversion rule is a simple indirect conversion rule and the second conversion rule is a complex indirect conversion rule.

[0105] When the program instructions are read and executed by the one or more processors, the program instructions may also perform operations corresponding to the various steps in the above method embodiments. Please refer to the above description and will not be repeated here. Figure 6 , which exemplarily shows the architecture of an electronic device, which may specifically include a processor 610, a video display adapter 611, a disk drive 612, an input / output interface 613, a network interface 614, and a memory 620. The processor 610, video display adapter 611, disk drive 612, input / output interface 613, network interface 614, and memory 620 may be communicatively connected via a communication bus 630.

[0106] Among them, the processor 610 can be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in this application.

[0107] The memory 620 can be implemented in the form of a read-only memory (ROM), a random access memory (RAM), a static storage device, a dynamic storage device, etc. The memory 620 can store an operating system 621 for controlling the operation of the electronic device 600, and a basic input and output system (BIOS) 622 for controlling the low-level operations of the electronic device 600. In addition, a web browser 623, a data storage management 624, and an icon font processing system 625, etc. can also be stored. The above-mentioned icon font processing system 625 can be an application program that specifically implements the operations of the aforementioned steps in the embodiment of the present application. In short, when the technical solution provided by the present application is implemented by software or firmware, the relevant program code is stored in the memory 620 and is called and executed by the processor 610.

[0108] The input / output interface 613 is used to connect to input / output modules to implement information input and output. The input / output modules can be configured as components within the device (not shown in the figure) or can be externally connected to the device to provide corresponding functions. Input devices may include a keyboard, mouse, touch screen, microphone, various sensors, etc., and output devices may include a display, speaker, vibrator, indicator light, etc.

[0109] The network interface 614 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via a wired method (such as USB, network cable, etc.) or a wireless method (such as mobile network, WiFi, Bluetooth, etc.).

[0110] The bus 630 comprises a pathway for transmitting information between the various components of the device (eg, the processor 610 , the video display adapter 611 , the disk drive 612 , the input / output interface 613 , the network interface 614 , and the memory 620 ).

[0111] In addition, the electronic device 600 can also obtain information on specific collection conditions from the virtual resource object collection condition information database 641 for use in condition judgment, etc.

[0112] It should be noted that although the electronic device 600 shown above only includes a processor 610, a video display adapter 611, a disk drive 612, an input / output interface 613, a network interface 614, a memory 620, a bus 630, etc., in a specific implementation, the electronic device may also include other components necessary for normal operation. In addition, it will be understood by those skilled in the art that the above device may only include the components necessary to implement the solution of the present application, and does not necessarily include all the components shown in the figure.

[0113] Through the description of the above implementation methods, it can be seen that those skilled in the art can clearly understand that the present application can be implemented by means of software plus the necessary general hardware platform. Based on this understanding, the technical solution of the present application can be essentially or in other words, the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling an electronic device (which can be a personal computer, a cloud server, or a network device, etc.) to execute the methods described in various embodiments of the present application or certain parts of the embodiments.

[0114] Example 4

[0115] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0116] Receiving a query statement and obtaining a logical plan by parsing the query statement;

[0117] Traversing the logical plan to obtain source table information, target table information, and a field conversion relationship, wherein the source table information includes a source table field expression ID and a corresponding source table field name, the target table information includes a target table field expression ID and a corresponding target table field name, and the field conversion relationship is a conversion relationship between the target table field expression ID and the source table field expression ID in the logical plan;

[0118] Field lineage information is obtained according to the source table information, target table information, field conversion relationship, first conversion rule and second conversion rule, wherein the first conversion rule is a simple indirect conversion rule and the second conversion rule is a complex indirect conversion rule.

[0119] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0120] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0121] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

[0122] Example 5

[0123] In one embodiment, a computer program product is provided, wherein a computer program is stored on the product, and when the computer program is executed by a processor, the following steps are performed:

[0124] Receiving a query statement and obtaining a logical plan by parsing the query statement;

[0125] Traversing the logical plan to obtain source table information, target table information, and a field conversion relationship, wherein the source table information includes a source table field expression ID and a corresponding source table field name, the target table information includes a target table field expression ID and a corresponding target table field name, and the field conversion relationship is a conversion relationship between the target table field expression ID and the source table field expression ID in the logical plan;

[0126] Field lineage information is obtained according to the source table information, target table information, field conversion relationship, first conversion rule and second conversion rule, wherein the first conversion rule is a simple indirect conversion rule and the second conversion rule is a complex indirect conversion rule.

[0127] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through a computer program. The computer program can be stored in a computer program product. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods.

[0128] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0129] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A method for obtaining field lineage, characterized in that: The method comprises: Receiving a query statement, and obtaining a logical plan by parsing the query statement; Traversing the logical plan to obtain source table information, target table information, and field conversion relationship, wherein the source table information includes a source table field expression ID and a corresponding source table field name, the target table information includes a target table field expression ID and a corresponding target table field name, and the field conversion relationship is a conversion relationship between the target table field expression ID and the source table field expression ID in the logical plan; Field lineage information is obtained according to the source table information, the target table information, the field conversion relationship, the first conversion rule and the second conversion rule, wherein the first conversion rule is a simple indirect conversion rule and the second conversion rule is a complex indirect conversion rule.

2. The method according to claim 1, characterized in that: The traversing the logical plan to obtain source table information, target table information, and field conversion relationships includes: Traversing the logical plan to obtain at least one level of logical nodes, wherein the at least one level of logical nodes includes a start node, an intermediate node, and an end node, the start node includes source table information, the intermediate node includes intermediate table information, and the end node includes target table information; The logic nodes of the at least one level are parsed in sequence to obtain field information of each logic node, wherein the field information includes a field expression ID and a field name; The field information corresponding to each adjacent logical node is associated and matched to obtain the field conversion relationship.

3. The method according to claim 1, characterized in that: The obtaining of field lineage information according to the source table information, the target table information, the field conversion relationship, the first conversion rule and the second conversion rule includes: Traversing the target table information, and determining whether there is a first remaining element, wherein the first remaining element is a target table field expression ID that does not have the field conversion relationship and a corresponding target table field name; In response to the existence of a first remaining element in the target table information, searching for corresponding intermediate table information according to the target table field expression ID of the first remaining element and according to the first conversion rule and the second conversion rule, wherein the intermediate table information includes the intermediate table field expression ID and the corresponding intermediate table field name; According to the intermediate table field expression ID, determining whether the source table information includes the intermediate table information; In response to the source table information including the intermediate table information, the field lineage information is obtained according to the intermediate table information and the target table information.

4. The method according to claim 3, characterized in that: The searching corresponding intermediate table information according to the target table field expression ID of the first remaining element and the first conversion rule and the second conversion rule includes: Determine corresponding first intermediate table information according to the first conversion rule and the target table field expression ID of the first remaining element; According to the second conversion rule and the target table field expression ID of the first remaining element, corresponding second intermediate table information is determined.

5. The method according to claim 4, characterized in that: Before judging whether the source table information includes the intermediate table information according to the intermediate table field expression ID, the method further includes: Combining the first intermediate table information, the second intermediate table information and the target table information to obtain comprehensive table information; Determine whether there is a second remaining element in the comprehensive table information, wherein the second remaining element is a field expression ID and a field name that does not have a field conversion relationship, a first conversion rule, and a second conversion rule; In response to the absence of the second remaining element in the comprehensive table information, the next step is to determine whether the source table information includes the intermediate table information according to the intermediate table field expression ID.

6. The method according to claim 1, characterized in that: After obtaining the field lineage information, the method further includes: The sensitive table information is updated according to the field lineage information to obtain new sensitive table information, wherein the new sensitive table information includes the sensitive field name; Matching the field lineage information with the new sensitive table information to obtain sensitive fields in the field lineage information, wherein the field lineage information includes a source table field name, an intermediate table field name, and a target table field name; The sensitive fields in the field lineage information are desensitized, wherein the desensitization includes at least one of rounding, offsetting, rearrangement, encryption, average value, invalidation, random value and data replacement.

7. The method according to claim 1 is applied to a field lineage acquisition system, wherein the system includes a message queue module, a metadata service, and a computing engine module, and is characterized in that: After obtaining the field lineage information, the method further includes: Writing the field lineage information into the message queue module; Consuming the field lineage information in the message queue module through the metadata service to write the field lineage information into the field lineage acquisition system in real time; Receiving a bloodline link location request, and determining the bloodline link according to the writing method of the field bloodline information; In response to the fact that the writing mode of the field lineage information is real-time writing, it is determined that the lineage links are, in order of writing, a computing engine module, a message queue module, and a metadata service.

8. A field lineage acquisition system, characterized in that: It includes a lineage application module, a computing engine module, a first physical machine, a second physical machine and a cloud container, wherein the first physical machine includes a message queue module and a database module, the second physical machine includes a search engine module, and the cloud container includes a front-end service, a lineage plug-in, a metadata service, a query service and a registry module. The output end of the computing engine module is connected to the first input end of the message queue module; The first input end of the metadata service is connected to the first output end of the message queue module, the second input end of the metadata service is connected to the output end of the front-end service, and the third input end of the metadata service is connected to the output end of the query service; The first output end of the metadata service is connected to the second input end of the message queue module, the second output end of the metadata service is connected to the input end of the search engine module, and the third output end of the metadata service is connected to the input end of the database module; The second output end of the message queue module is connected to the input end of the bloodline plug-in, and the third output end of the message queue module is connected to the input end of the registry module; The output end of the bloodline plug-in is connected to the input end of the bloodline application module.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images