Field blood relationship data processing method and device, computer equipment and storage medium

Abstract

The invention relates to a field blood relationship data processing method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring data table information and a to-be-analyzed instruction statement; determining a target field according to the to-be-analyzed instruction statement, and taking data table information corresponding to the target field as target analysis information; analyzing data blood relationship information of the target field according to the target analysis information; under the condition that the data blood relationship information is complete, taking the data blood relationship information as target blood relationship information; under the condition that the data consanguinity information is incomplete, metadata information of the target field is determined according to the data table information, the data consanguinity information is supplemented according to the metadata information, and target consanguinity information is obtained. By adopting the method, the accuracy of field blood relationship analysis can be improved.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a field lineage data processing method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] Data lineage is used to track the source and destination of a table, that is, to track the entire chain from table generation, conversion, filtering, to fusion with other tables, and finally providing the table to the application end to provide table query services. Through data lineage, you can analyze the production node information of the table and the downstream nodes affected by the table anomaly, and intuitively see the source and destination of the table. For complex SQL, data lineage can better assist developers in understanding SQL.

[0003] At present, in the relevant technologies, most of the processing methods for data lineage only meet the basic functions of table lineage, and the processing complexity and difficulty of field lineage are higher than that of table lineage. Therefore, in most of the relevant technologies, field lineage can only be based on static analysis, and the accuracy of field lineage analysis cannot be improved. Summary of the invention

[0004] Based on this, it is necessary to provide a field lineage data processing method, apparatus, computer equipment, computer-readable storage medium and computer program product that can improve the accuracy of field lineage analysis in response to the above-mentioned technical problems.

[0005] In a first aspect, the present application provides a method for processing field lineage data, comprising:

[0006] Get data table information and instruction statements to be parsed;

[0007] Determine the target field according to the instruction statement to be parsed, and use the data table information corresponding to the target field as the target parsing information;

[0008] Parsing the data lineage information of the target field according to the target parsing information;

[0009] When the data lineage information is complete, the data lineage information is used as the target lineage information; when the data lineage information is incomplete, the metadata information of the target field is determined according to the data table information, and the data lineage information is supplemented according to the metadata information to obtain the target lineage information.

[0010] In one of the embodiments, before determining the target field according to the instruction statement to be parsed, the method further includes:

[0011] Generate a syntax tree according to the data table information, and monitor each node on the syntax tree to obtain monitoring data information;

[0012] According to the target field and the monitoring data information, data table information corresponding to the target field is determined.

[0013] In one embodiment, the target parsing information includes source table parsing information, target table parsing information, and a data mapping relationship between the source table parsing information and the target table parsing information; and parsing the data lineage information of the target field according to the target parsing information includes:

[0014] Determine the source table lineage information of the target field according to the source table parsing information;

[0015] Determine the target table lineage information of the target field according to the target table parsing information;

[0016] According to the data mapping relationship, the source table lineage information and the target table lineage information are combined to obtain data lineage information of the target field.

[0017] In one of the embodiments, when the data lineage information is incomplete, determining metadata information of the target field according to the data table information, and supplementing the data lineage information according to the metadata information to obtain the target lineage information includes:

[0018] In the case where the source table lineage information is incomplete, querying the metadata information of the target field in the source table in the data table information;

[0019] Supplement the source table lineage information according to the metadata information;

[0020] According to the data mapping relationship, the supplemented source table lineage information and the target table lineage information are combined to obtain the target lineage information.

[0021] In one of the embodiments, when the data lineage information is incomplete, determining metadata information of the target field according to the data table information, and supplementing the data lineage information according to the metadata information to obtain the target lineage information includes:

[0022] In the case where the target table lineage information is incomplete, querying metadata information of the target field in the target table in the data table information;

[0023] Supplement the target table lineage information according to the metadata information;

[0024] According to the data mapping relationship, the target table lineage information and the source table lineage information are combined and supplemented to obtain the target lineage information.

[0025] In one embodiment, determining the metadata information of the target field according to the data table information includes:

[0026] In a case where the data table information includes a table creation statement, determining metadata information of the target field according to the table creation statement;

[0027] In the case that the data table information does not include a table creation statement, a data table associated with the target field is identified to obtain table structure information, and metadata information of the target field is determined according to the table structure information.

[0028] In a second aspect, the present application also provides a field lineage data processing device, comprising:

[0029] An information acquisition module is used to obtain data table information and instruction statements to be parsed;

[0030] A field parsing module, used to determine a target field according to the instruction statement to be parsed, and use the data table information corresponding to the target field as target parsing information;

[0031] A data parsing module, used for parsing the data lineage information of the target field according to the target parsing information;

[0032] A data processing module is used to use the data lineage information as target lineage information when the data lineage information is complete; when the data lineage information is incomplete, determine the metadata information of the target field according to the data table information, and supplement the data lineage information according to the metadata information to obtain the target lineage information.

[0033] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0034] Get data table information and instruction statements to be parsed;

[0035] Determine the target field according to the instruction statement to be parsed, and use the data table information corresponding to the target field as the target parsing information;

[0036] Parsing the data lineage information of the target field according to the target parsing information;

[0037] When the data lineage information is complete, the data lineage information is used as the target lineage information; when the data lineage information is incomplete, the metadata information of the target field is determined according to the data table information, and the data lineage information is supplemented according to the metadata information to obtain the target lineage information.

[0038] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0039] Get data table information and instruction statements to be parsed;

[0040] Determine the target field according to the instruction statement to be parsed, and use the data table information corresponding to the target field as the target parsing information;

[0041] Parsing the data lineage information of the target field according to the target parsing information;

[0042] When the data lineage information is complete, the data lineage information is used as the target lineage information; when the data lineage information is incomplete, the metadata information of the target field is determined according to the data table information, and the data lineage information is supplemented according to the metadata information to obtain the target lineage information.

[0043] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0044] Get data table information and instruction statements to be parsed;

[0045] Determine the target field according to the instruction statement to be parsed, and use the data table information corresponding to the target field as the target parsing information;

[0046] Parsing the data lineage information of the target field according to the target parsing information;

[0047] When the data lineage information is complete, the data lineage information is used as the target lineage information; when the data lineage information is incomplete, the metadata information of the target field is determined according to the data table information, and the data lineage information is supplemented according to the metadata information to obtain the target lineage information.

[0048] The above-mentioned field lineage data processing method, device, computer equipment, storage medium and computer program product obtain data table information and instruction statements to be parsed, and then determine the target field according to the instruction statements to be parsed, so as to use the data table information corresponding to the target field as the target parsing information, and parse the data lineage information of the target field according to the target parsing information. According to the target parsing information, the source and destination of the target field can be parsed, thereby forming a field lineage, that is, data lineage information. When the data lineage information is complete, the data lineage information can be used as the target lineage information, and the field lineage is often highly complex, and there may be a situation where it cannot be completely parsed directly through the target parsing information. Therefore, when the data lineage information is incomplete, the metadata information of the target field is determined according to the data table information, and the data lineage information is supplemented according to the metadata information to obtain the complete target lineage information, so that the complete field lineage can be more accurately obtained by using metadata as a supplement, thereby improving the accuracy of field lineage analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0050] FIG1 is an application environment diagram of a method for processing field lineage data in one embodiment;

[0051] FIG2 is a schematic diagram of a flow chart of a method for processing field lineage data in one embodiment;

[0052] FIG3 is a flow chart of step S208 of a method for processing field lineage data in one embodiment;

[0053] FIG4 is a block diagram of a field lineage data processing device in one embodiment;

[0054] FIG5 is a diagram showing the internal structure of a computer device in one embodiment;

[0055] FIG. 6 is a diagram showing the internal structure of a computer device in another embodiment. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0057] The field lineage data processing method provided in the embodiment of the present application can be applied in the application environment shown in Figure 1. Among them, the terminal 102 communicates with the server 104 through the network, and the terminal 102 can be used to obtain data table information, send the data table information to the server 104, and store it in the data storage system. The terminal 102 can also be used to display the target lineage information output by the server 104. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The data storage system can be used to store data table information, and can also be used to store the data lineage information obtained by parsing. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented with an independent server or a server cluster consisting of multiple servers.

[0058] In an exemplary embodiment, as shown in FIG2 , a field lineage data processing method is provided, which is described by taking the method applied to the server 104 in FIG1 as an example, and includes the following steps S202 to S206 . Among them:

[0059] Step S202, obtaining data table information and instruction statements to be parsed.

[0060] The instruction statement to be parsed may be a SQL statement, and the data table information may include information such as field names and field types in the data table.

[0061] Exemplarily, the server 104 can respond to the field lineage request of the terminal 102, obtain the corresponding data table information and the instruction statement to be parsed, generate a syntax tree based on the data table information, and monitor each node on the syntax tree to obtain the monitoring data information, and determine the data table information corresponding to the target field based on the target field and the monitoring data information.

[0062] Exemplarily, the server 104 can use the antlr4 plug-in to generate the corresponding implementation class, inherit the base Listener class generated by antlr4 through the custom listener class, implement the enter and exit method logic of the corresponding node as needed, and use the listener mode to traverse the syntax tree. During the traversal process, it will automatically enter the enter and exit methods of the corresponding node. Among them, antlr4 is a kind of grammar parsing generator, and the grammar file is written according to the grammar rules provided by antlr4. antlr4 can parse the grammar rule file to generate the corresponding syntax tree, as well as generate the corresponding java class and method.

[0063] Step S204, determining the target field according to the instruction statement to be parsed, and using the data table information corresponding to the target field as the target parsing information.

[0064] The target parsing information may include source table parsing information, target table parsing information, and a data mapping relationship between the source table parsing information and the target table parsing information.

[0065] Exemplarily, the server 104 may determine the target field according to the field included in the instruction statement to be parsed, and match the information associated with the target field in the table data information according to the target field as the target parsing information. For example, in Example 1, the instruction statement to be parsed is: create table x select name , age from x1 join x2. This statement creates table x, where the name field comes from the x1 table, and the age field comes from the x2 table. The name field and the age field are the target fields, and the information about the name field and the age field in the x1 table and the x2 table is the target table parsing information.

[0066] In another example, the instruction statement to be parsed in Example 2 is: create table x select x1.name+ x2.age from x1 join x2. This statement creates table x with a field named age, whose content comes from the result of adding the name field of x1 and the age field of x2. The name field and the age field are the target fields, and the information about the name field and the age field in the x1 table and the x2 table is the target table parsing information.

[0067] Step S206, parsing the data lineage information of the target field according to the target parsing information.

[0068] Exemplarily, the server 104 may determine the source table lineage information of the target field based on the source table parsing information; determine the target table lineage information of the target field based on the target table parsing information; and combine the source table lineage information and the target table lineage information based on the data mapping relationship to obtain the data lineage information of the target field.

[0069] According to the above example 1, the information about the name field and the age field in the x1 table and the x2 table is the target table parsing information. The name field comes from x1, and the age field comes from x2. The server 104 can record x1.name->x.name and x2.age->x.age as the data lineage information of the target field.

[0070] According to the above example 2, the server 104 parses x1.name and x2.age to derive that these two fields together generate the age field of the x table. The server 104 can record x1.name->x.age and x2.age->x.age as the data lineage information of the target field.

[0071] Exemplarily, combining the above steps S204 to S206, Example 3: When parsing the statement create table ifnot exists tab3 as select a.id + b.id as id, b.birthday as birthday from tab1a join tab0 b on a.id = b.id, the server 104 can first use the ANTLR4 tool to generate a syntax tree. By deeply traversing the syntax tree from left to right, the relationship between field mappings can be obtained, and it can be concluded that a.id is mapped to the id field of the output table tab3, b.id is also mapped to the id field in the tab3 table, and b.birthday is mapped to the birthday field of the output table tab3. At the same time, tab1 is mapped to the alias a, and tab0 is mapped to the alias b. During the traversal process, the server 104 records these mapping relationships. Then, when traversing to the from and join statements, the server 104 needs to perform equivalent replacements according to the aliases of the tables. Specifically, a.id needs to be replaced with tab1.id, so a.id->id is converted to tab1.id->tab3.id; similarly, b.id needs to be replaced with tab0.id, so b.id->id is converted to tab0.id->tab3.id. Finally, after completing the entire traversal process, server 104 obtains the complete field lineage relationship: tab1.id->tab3.id, tab0.id->tab3.id, and tab0.birthday->tab3.birthday. These mapping relationships reflect the transfer process of each field from the source table to the target table, thereby revealing the lineage relationship between the fields in the SQL query.

[0072] Step S208, when the data lineage information is complete, the data lineage information is used as the target lineage information; when the data lineage information is incomplete, the metadata information of the target field is determined according to the data table information, and the data lineage information is supplemented according to the metadata information to obtain the target lineage information.

[0073] The metadata information may include field type, field name and other information.

[0074] For example, when the data lineage information is complete, the server 104 can use the data lineage information as the target lineage information, but the field lineage is often highly complex, and there may be a situation where it cannot be completely parsed directly through the target parsing information. Therefore, when the data lineage information is incomplete, the server 104 can determine the metadata information of the target field based on the data table information. Further, when the data table information includes a table creation statement, the server 104 can determine the metadata information of the target field based on the table creation statement, and when the data table information does not include a table creation statement, identify the data table associated with the target field, obtain the table structure information, and determine the metadata information of the target field based on the table structure information.

[0075] Furthermore, the server 104 can obtain metadata information by reading table creation statements when there are table creation statements; it can also obtain metadata information actively provided by the terminal 102; it can also obtain metadata information through database connection. The server 104 can connect to the corresponding database through JDBC (Java Database Connectivity) or other database connection methods, and use a query command to obtain metadata information of the table. In addition, the server 104 can also obtain table information from the metadata system interface. If there is a dedicated metadata management system in the system (for example, a service that manages all table structures), the server 104 can directly obtain metadata through the API interface of this system.

[0076] Next, the server 104 may supplement the incomplete data lineage information according to the source data information. For example, the instruction statement to be parsed is: create table x select * from x1. The statement means to select all fields (indicated by *) from table x1 and create a new table x.

[0077] When the target parsing information parses the data lineage information of the target field, the server 104 cannot directly obtain which fields * specifically refers to, so the server 104 can obtain the metadata information in the x1 table. The server 104 can query the metadata of the x1 table. Assuming that the x1 table has the fields name and age, then * represents x1.name and x1.age. Finally, the server 104 can write down x1.name->x.name and x1.age->x.age, indicating that the name field in the x table comes from the name field of x1, and the age field comes from the age field of x1, to obtain the complete field lineage, that is, the target lineage information of the target field.

[0078] In the above-mentioned field lineage data processing method, by obtaining data table information and instruction statements to be parsed, and then determining the target field according to the instruction statements to be parsed, the data table information corresponding to the target field is used as the target parsing information, and the data lineage information of the target field is parsed according to the target parsing information. According to the target parsing information, the source and destination of the target field can be parsed, thereby forming a field lineage, that is, the data lineage information. When the data lineage information is complete, the data lineage information can be used as the target lineage information, and the field lineage is often highly complex, and there may be a situation where it cannot be fully parsed directly through the target parsing information. Therefore, when the data lineage information is incomplete, the metadata information of the target field is determined according to the data table information, and the data lineage information is supplemented according to the metadata information to obtain the complete target lineage information, so that the complete field lineage can be more accurately obtained by using metadata as a supplement, thereby improving the accuracy of field lineage analysis.

[0079] In an exemplary embodiment, as shown in FIG3 , step S208 may include steps S302 to S306 . Among them:

[0080] Step S302: when the source table lineage information is incomplete, query the metadata information of the target field in the source table in the data table information.

[0081] Exemplarily, the instruction statement to be parsed is create table x select * from x1. The statement means to select all fields (indicated by *) from table x1 and create a new table x. Since server 104 cannot directly obtain which fields * specifically refers to, the source table lineage information is incomplete. Server 104 can obtain the metadata information of the target field in the source table, that is, the metadata information in the x1 table. Accordingly, server 104 can query the metadata information of the target field in the target table in the data table information when the target table lineage information is incomplete.

[0082] Step S304: supplement the source table lineage information according to the metadata information.

[0083] For example, the server 104 can query the metadata of the x1 table. Assuming that the x1 table has fields name and age, then * represents x1.name and x1.age, thereby supplementing the source table lineage information according to the metadata information. Correspondingly, when the target table lineage information is incomplete, the server 104 can supplement the target table lineage information according to the metadata information.

[0084] Step S306, combining the supplemented source table lineage information and the target table lineage information according to the data mapping relationship to obtain the target lineage information.

[0085] For example, the server 104 can record x1.name->x.name and x1.age->x.age, indicating that the name field in the x table comes from the name field of x1, and the age field comes from the age field of x1, to obtain the complete field lineage, that is, the target lineage information of the target field. Correspondingly, when the target table lineage information is incomplete, the server 104 can combine the supplemented target table lineage information and the source table lineage information according to the data mapping relationship to obtain the target lineage information.

[0086] In an exemplary embodiment, the instruction statement to be parsed is: create table test_nest asselect a.name from (select b.name from test_nest1 b) a, which creates the table test_nest, where the name field comes from the nested query statement, and the name field in the nested statement comes from the name field of the test_nest1 table. The server 104 can parse to obtain test_nest1.name->a.name, and then a.name->test_nest.name. Next, since a is only an intermediate process, the server 104 can eliminate the intermediate relationship a and obtain the final field lineage result of test_nest1.name->test_nest.name.

[0087] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0088] Based on the same inventive concept, the embodiment of the present application also provides a field lineage data processing device for implementing the field lineage data processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more field lineage data processing device embodiments provided below can refer to the limitations of the field lineage data processing method above, and will not be repeated here.

[0089] In an exemplary embodiment, as shown in FIG4 , a field lineage data processing device is provided, comprising: an information acquisition module 402, a field analysis module 404, a data analysis module 406 and a data processing module 408, wherein:

[0090] Information acquisition module 402, used to acquire data table information and instruction statements to be parsed;

[0091] The field parsing module 404 is used to determine the target field according to the instruction statement to be parsed, and use the data table information corresponding to the target field as the target parsing information;

[0092] The data analysis module 406 is used to analyze the data lineage information of the target field according to the target analysis information;

[0093] The data processing module 408 is used to use the data lineage information as the target lineage information when the data lineage information is complete; when the data lineage information is incomplete, determine the metadata information of the target field according to the data table information, and supplement the data lineage information according to the metadata information to obtain the target lineage information.

[0094] In one embodiment, the device further comprises:

[0095] The syntax tree generation module is used to generate a syntax tree according to the data table information, and monitor each node on the syntax tree to obtain the monitoring data information;

[0096] The data monitoring unit is used to determine the data table information corresponding to the target field according to the target field and the monitored data information.

[0097] In one of the embodiments, the target resolution information includes source table resolution information, target table resolution information and a data mapping relationship between the source table resolution information and the target table resolution information; the data resolution module 406 is specifically used to: determine the source table lineage information of the target field according to the source table resolution information; determine the target table lineage information of the target field according to the target table resolution information; and combine the source table lineage information and the target table lineage information according to the data mapping relationship to obtain the data lineage information of the target field.

[0098] In one embodiment, the data processing module 408 includes: a source table data analysis unit, which is specifically used to: when the source table lineage information is incomplete, query the metadata information of the target field in the source table in the data table information; supplement the source table lineage information according to the metadata information; and combine the supplemented source table lineage information and the target table lineage information according to the data mapping relationship to obtain the target lineage information.

[0099] In one embodiment, the data processing module 408 includes: a target table data analysis unit, which is specifically used to: when the target table lineage information is incomplete, query the metadata information of the target field in the target table in the data table information; supplement the target table lineage information according to the metadata information; and combine the supplemented target table lineage information and the source table lineage information according to the data mapping relationship to obtain the target lineage information.

[0100] In one embodiment, the data processing module 408 also includes: a metadata acquisition unit, which is used to determine the metadata information of the target field according to the table creation statement when the data table information includes a table creation statement; when the data table information does not include a table creation statement, identify the data table associated with the target field, obtain the table structure information, and determine the metadata information of the target field according to the table structure information.

[0101] Each module in the above field lineage data processing device can be implemented in whole or in part by software, hardware and their combination. Each module can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0102] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG5. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data table information, and can also be used to store the data lineage information obtained by parsing. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a field lineage data processing method is implemented.

[0103] In an exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be shown in FIG6. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, the memory, and the input / output interface are connected via a system bus, and the communication interface, the display unit, and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner may be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a field lineage data processing method is implemented. The display unit of the computer device is used to form a visually visible picture, which may be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0104] Those skilled in the art will understand that the structure shown in FIG. 6 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0105] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the following steps when executing the computer program: obtaining data table information and instruction statements to be parsed; determining a target field according to the instruction statements to be parsed, and using the data table information corresponding to the target field as target parsing information; parsing data lineage information of the target field according to the target parsing information; when the data lineage information is complete, using the data lineage information as the target lineage information; when the data lineage information is incomplete, determining metadata information of the target field according to the data table information, and supplementing the data lineage information according to the metadata information to obtain the target lineage information.

[0106] In one embodiment, when the processor executes the computer program, the following steps are also implemented: generating a syntax tree according to the data table information, and monitoring each node on the syntax tree to obtain the monitored data information; determining the data table information corresponding to the target field according to the target field and the monitored data information.

[0107] In one embodiment, when the processor executes the computer program, the following steps are also implemented: determining the source table lineage information of the target field based on the source table parsing information; determining the target table lineage information of the target field based on the target table parsing information; and combining the source table lineage information and the target table lineage information based on the data mapping relationship to obtain the data lineage information of the target field.

[0108] In one embodiment, when the processor executes the computer program, the following steps are also implemented: when the source table lineage information is incomplete, query the metadata information of the target field in the source table in the data table information; supplement the source table lineage information according to the metadata information; and combine the supplemented source table lineage information and the target table lineage information according to the data mapping relationship to obtain the target lineage information.

[0109] In one embodiment, when the processor executes the computer program, the following steps are also implemented: when the target table lineage information is incomplete, querying the metadata information of the target field in the target table in the data table information; supplementing the target table lineage information according to the metadata information; and combining the supplemented target table lineage information and the source table lineage information according to the data mapping relationship to obtain the target lineage information.

[0110] In one embodiment, when the processor executes the computer program, the following steps are also implemented: when the data table information includes a table creation statement, the metadata information of the target field is determined according to the table creation statement; when the data table information does not include a table creation statement, the data table associated with the target field is identified, the table structure information is obtained, and the metadata information of the target field is determined according to the table structure information.

[0111] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: obtaining data table information and instruction statements to be parsed; determining a target field according to the instruction statements to be parsed, and using the data table information corresponding to the target field as target parsing information; parsing data lineage information of the target field according to the target parsing information; when the data lineage information is complete, using the data lineage information as the target lineage information; when the data lineage information is incomplete, determining metadata information of the target field according to the data table information, and supplementing the data lineage information according to the metadata information to obtain the target lineage information.

[0112] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: generating a syntax tree according to the data table information, and monitoring each node on the syntax tree to obtain the monitored data information; determining the data table information corresponding to the target field according to the target field and the monitored data information.

[0113] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: determining the source table lineage information of the target field based on the source table parsing information; determining the target table lineage information of the target field based on the target table parsing information; and combining the source table lineage information and the target table lineage information based on the data mapping relationship to obtain the data lineage information of the target field.

[0114] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: when the source table lineage information is incomplete, query the metadata information of the target field in the source table in the data table information; supplement the source table lineage information according to the metadata information; and combine the supplemented source table lineage information and the target table lineage information according to the data mapping relationship to obtain the target lineage information.

[0115] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: when the target table lineage information is incomplete, querying the metadata information of the target field in the target table in the data table information; supplementing the target table lineage information according to the metadata information; and combining the supplemented target table lineage information and the source table lineage information according to the data mapping relationship to obtain the target lineage information.

[0116] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: when the data table information includes a table creation statement, the metadata information of the target field is determined according to the table creation statement; when the data table information does not include a table creation statement, the data table associated with the target field is identified, the table structure information is obtained, and the metadata information of the target field is determined according to the table structure information.

[0117] In one embodiment, a computer program product is provided, including a computer program, which implements the following steps when executed by a processor: obtaining data table information and instruction statements to be parsed; determining a target field according to the instruction statements to be parsed, and using the data table information corresponding to the target field as target parsing information; parsing data lineage information of the target field according to the target parsing information; when the data lineage information is complete, using the data lineage information as target lineage information; when the data lineage information is incomplete, determining metadata information of the target field according to the data table information, and supplementing the data lineage information according to the metadata information to obtain the target lineage information.

[0118] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: generating a syntax tree according to the data table information, and monitoring each node on the syntax tree to obtain the monitored data information; determining the data table information corresponding to the target field according to the target field and the monitored data information.

[0119] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: determining the source table lineage information of the target field based on the source table parsing information; determining the target table lineage information of the target field based on the target table parsing information; and combining the source table lineage information and the target table lineage information based on the data mapping relationship to obtain the data lineage information of the target field.

[0120] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: when the source table lineage information is incomplete, query the metadata information of the target field in the source table in the data table information; supplement the source table lineage information according to the metadata information; and combine the supplemented source table lineage information and the target table lineage information according to the data mapping relationship to obtain the target lineage information.

[0121] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: when the target table lineage information is incomplete, querying the metadata information of the target field in the target table in the data table information; supplementing the target table lineage information according to the metadata information; and combining the supplemented target table lineage information and the source table lineage information according to the data mapping relationship to obtain the target lineage information.

[0122] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: when the data table information includes a table creation statement, the metadata information of the target field is determined according to the table creation statement; when the data table information does not include a table creation statement, the data table associated with the target field is identified, the table structure information is obtained, and the metadata information of the target field is determined according to the table structure information.

[0123] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0124] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0125] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0126] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for processing field lineage data, characterized in that: The method comprises: Get data table information and instruction statements to be parsed; Determine the target field according to the instruction statement to be parsed, and use the data table information corresponding to the target field as the target parsing information; Parsing the data lineage information of the target field according to the target parsing information; When the data lineage information is complete, the data lineage information is used as the target lineage information; when the data lineage information is incomplete, the metadata information of the target field is determined according to the data table information, and the data lineage information is supplemented according to the metadata information to obtain the target lineage information.

2. The method according to claim 1, characterized in that Before determining the target field according to the instruction statement to be parsed, the method further includes: Generate a syntax tree according to the data table information, and monitor each node on the syntax tree to obtain monitoring data information; According to the target field and the monitoring data information, data table information corresponding to the target field is determined.

3. The method according to claim 1, characterized in that The target parsing information includes source table parsing information, target table parsing information, and a data mapping relationship between the source table parsing information and the target table parsing information; The step of parsing the data lineage information of the target field according to the target parsing information includes: Determine the source table lineage information of the target field according to the source table parsing information; Determine the target table lineage information of the target field according to the target table parsing information; According to the data mapping relationship, the source table lineage information and the target table lineage information are combined to obtain data lineage information of the target field.

4. The method according to claim 3, characterized in that In the case where the data lineage information is incomplete, determining the metadata information of the target field according to the data table information, and supplementing the data lineage information according to the metadata information to obtain the target lineage information, including: In the case where the source table lineage information is incomplete, querying the metadata information of the target field in the source table in the data table information; Supplement the source table lineage information according to the metadata information; According to the data mapping relationship, the supplemented source table lineage information and the target table lineage information are combined to obtain the target lineage information.

5. The method according to claim 3, characterized in that: In the case where the data lineage information is incomplete, determining the metadata information of the target field according to the data table information, and supplementing the data lineage information according to the metadata information to obtain the target lineage information, including: In the case where the target table lineage information is incomplete, querying metadata information of the target field in the target table in the data table information; Supplement the target table lineage information according to the metadata information; According to the data mapping relationship, the target table lineage information and the source table lineage information are combined and supplemented to obtain the target lineage information.

6. The method according to any one of claims 1 to 5, characterized in that: The step of determining the metadata information of the target field according to the data table information includes: In a case where the data table information includes a table creation statement, determining metadata information of the target field according to the table creation statement; In the case that the data table information does not include a table creation statement, a data table associated with the target field is identified to obtain table structure information, and metadata information of the target field is determined according to the table structure information.

7. A field lineage data processing device, characterized in that: The device comprises: An information acquisition module is used to obtain data table information and instruction statements to be parsed; A field parsing module, used to determine a target field according to the instruction statement to be parsed, and use the data table information corresponding to the target field as target parsing information; A data parsing module, used for parsing the data lineage information of the target field according to the target parsing information; A data processing module is used to use the data lineage information as target lineage information when the data lineage information is complete; when the data lineage information is incomplete, determine the metadata information of the target field according to the data table information, and supplement the data lineage information according to the metadata information to obtain the target lineage information.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images