Data blood relationship construction method and device, computer equipment and storage medium

Abstract

The invention provides a data blood relationship construction method and device, computer equipment and a storage medium, and belongs to the technical field of big data processing. The method comprises the steps of obtaining target business data of a to-be-constructed data blood relationship; performing structured processing on reference table data in a plurality of data rows of the target business data to obtain a first dependency relationship among the reference table data; performing structured processing on the respective field data of the multiple pieces of reference table data to obtain a second dependency relationship among the respective field data of the multiple pieces of reference table data; and constructing a data blood relationship of the target business data based on the first dependency relationship and the second dependency relationship. By adopting the method, the working efficiency of constructing the big data real blood relationship can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of big data processing, and in particular to a method, device, computer equipment and storage medium for constructing data lineage relationships. Background Art

[0002] In the related technologies, the construction of the real blood relationship of big data relies on manual maintenance of the dependency relationship of big data task scheduling. As a result, with the increasing complexity of big data business scenarios, the workload of manually maintaining the dependency relationship of big data task scheduling has gradually become complicated, making it difficult to improve the overall work efficiency of building the real blood relationship of big data. Therefore, how to improve the work efficiency of building the real blood relationship of big data has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a method, device, computer equipment and storage medium for constructing data lineage relationships, aiming to improve the work efficiency of constructing real lineage relationships of big data.

[0004] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a method for constructing a data lineage relationship, the method comprising:

[0005] Obtain the target business data of the data lineage relationship to be constructed;

[0006] Performing structural processing on the referenced table data in the multiple data rows of the target business data to obtain a first dependency relationship between the multiple referenced table data;

[0007] Performing structural processing on the field data of each of the plurality of referenced table data to obtain a second dependency relationship between the field data of each of the plurality of referenced table data;

[0008] A data lineage relationship of the target business data is constructed based on the first dependency relationship and the second dependency relationship.

[0009] In some embodiments, obtaining target business data for which data lineage relationship is to be established includes:

[0010] Traversing the business data for which data lineage relationships are to be constructed to confirm invalid data in the business data that is not related to the data processing task;

[0011] The invalid data is cleaned to obtain target business data for which data lineage relationship is to be constructed in the business data.

[0012] In some embodiments, traversing the business data for which data lineage relationships are to be constructed to confirm invalid data in the business data that is not related to the data processing task includes:

[0013] Traversing the business data for which data lineage relationships are to be constructed to confirm operation identification characters and statement separation characters in the business data;

[0014] Confirming that the business data is located in a first data row before the operation identification character, and confirming that the business data is located in a second data row after the statement separation character;

[0015] The first data row and the second data row are marked as invalid data in the business data that is irrelevant to the data processing task.

[0016] In some embodiments, after marking the first data row and the second data row as invalid data in the business data that is not related to the data processing task, the method further includes:

[0017] In the plurality of third data rows where the business data is located between the operation identification character and the statement separation character, confirming that a target data row with a comment identification character exists;

[0018] In the case where the comment identification character is located at the beginning of the target data row, marking the target data row as invalid data in the business data that is irrelevant to the data processing task;

[0019] When the comment identification character is located in the middle of the target data row, the comment identification character and the data after the comment identification character in the target data row are marked as invalid data in the business data that is irrelevant to the data processing task.

[0020] In some embodiments, the structural processing of the reference table data in the multiple data rows of the target business data includes:

[0021] Identify a first target data row containing a table keyword among the multiple data rows of the target business data;

[0022] Filtering the data before the table keyword and the data after the preset partition keyword in the first target data row to structure the first reference table data between the table keyword and the preset partition keyword;

[0023] Identifying a second target data row containing a reference keyword pair among the multiple data rows of the target business data;

[0024] In the second target data row, structuring the second reference table data between the reference keyword pairs;

[0025] The reference table data includes the first reference table data and the second reference table data.

[0026] In some embodiments, the structural processing of the field data of each of the plurality of reference table data includes:

[0027] identifying a first first search keyword pair after the first target data row, and identifying a first second search keyword pair after the second target data row;

[0028] Structuring the first field data between the first search keyword pair and the generation logic of the first field data; the first field data is the field data of the first reference table data;

[0029] The second field data between the second search keyword pairs is structured; the second field data is the field data of the second reference table data.

[0030] In some embodiments, the step of constructing a data lineage relationship of the target business data based on the first dependency relationship and the second dependency relationship includes:

[0031] Output and display the first dependency relationship and the second dependency relationship, and receive a sampling request for a target dependency relationship in the first dependency relationship and the second dependency relationship;

[0032] In response to the sampling request, highlighting the target dependency;

[0033] When a confirmation indication for the target dependency is received, the first dependency and the second dependency are used as real data lineages to construct a data lineage relationship for the target business data.

[0034] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes a device for constructing a data lineage relationship, the device comprising:

[0035] An acquisition module is used to acquire target business data of the data lineage relationship to be constructed;

[0036] A structured processing module, configured to perform structured processing on the reference table data in the multiple data rows of the target business data to obtain a first dependency relationship between the multiple reference table data; and perform structured processing on the field data of each of the multiple reference table data to obtain a second dependency relationship between the field data of each of the multiple reference table data;

[0037] A data lineage construction module is used to construct the data lineage relationship of the target business data based on the first dependency relationship and the second dependency relationship.

[0038] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes a computer device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the method described in the first aspect when executing the computer program.

[0039] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0040] In addition, to achieve the above-mentioned purpose, the fifth aspect of an embodiment of the present application also proposes a computer program product, which includes a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0041] The data lineage construction method, apparatus, computer equipment, computer-readable storage medium and computer program product proposed in the embodiments of the present application obtain target business data for which data lineage is to be constructed; perform structural processing on referenced table data in multiple data rows of the target business data to obtain a first dependency relationship between the multiple referenced table data; perform structural processing on the field data of each of the multiple referenced table data to obtain a second dependency relationship between the field data of each of the multiple referenced table data; and construct the data lineage of the target business data based on the first dependency relationship and the second dependency relationship.

[0042] The embodiment of the present application obtains the target business data, and traverses the target business data to first perform structural processing on the referenced table data in multiple data rows of the target business data, thereby obtaining a first dependency relationship between the multiple referenced table data, and also performs structural processing on the field data of each of the multiple referenced table data, thereby obtaining a second dependency relationship between the field data of each of the multiple referenced table data, and then constructs the data lineage relationship of the target business data based on the first dependency relationship and the second dependency relationship. That is, compared with the method of constructing the real lineage of data based on the dependency relationship of manually maintaining big data task scheduling, the embodiment of the present application parses the target business data to construct the real lineage relationship of the table data in the business data depending on the table data, which can effectively improve the overall work efficiency of constructing the real lineage relationship of big data. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A schematic diagram of the steps of the method for constructing data kinship relationships provided in some embodiments of the present application;

[0044] Figure 2 for Figure 1 Schematic diagram of detailed process flow of step S101;

[0045] Figure 3 for Figure 2 Schematic diagram of detailed process flow of step S201;

[0046] Figure 4 A schematic diagram of the steps of the method for constructing data kinship relationships provided in the embodiments of the present application in other embodiments;

[0047] Figure 5 for Figure 1 Schematic diagram of detailed process flow of step S102;

[0048] Figure 6 for Figure 1 Schematic diagram of detailed process flow of step S103;

[0049] Figure 7 for Figure 1 Schematic diagram of detailed process flow of step S104;

[0050] Figure 8 It is a schematic diagram of the structure of a device for constructing data blood relationship provided in an embodiment of the present application;

[0051] Fig. 9 It is a schematic diagram of the hardware structure of the computer device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0053] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0055] First, the overall concept of the embodiments of the present application is described.

[0056] The true blood relationship of data refers to the relationship between business data in the process of generation, processing, circulation and extinction, which is similar to the blood relationship in human society. Data blood relationship is an important area of ​​big data governance. Good data blood relationship is an important basis for evaluating the professionalism of a financial technology company. Data blood relationship mainly provides data managers with a better positioning for executing governance strategies in the fields of intelligent operation and maintenance, event root cause analysis, unified business terminology, correlation impact analysis, performance evaluation, security compliance, etc., to help company development.

[0057] In the related art, the construction of the real blood relationship of big data relies on manual maintenance of the dependency relationship of big data task scheduling. In this way, with the increasing complexity of big data business scenarios, the workload of manually maintaining the dependency relationship of big data task scheduling has gradually become complicated. In addition, for complex business scenarios (such as daily update tasks relying on monthly update and quarterly update tasks, tasks of platform A relying on platform B, etc.), the manually maintained dependency relationship of big data task scheduling often leads to the loss of the real blood relationship of big data. In addition, in the related art, there is also a lack of easy-to-understand, concise, intuitive, safe and efficient display methods for the constructed real blood relationship of big data. All of the above make it difficult to improve the overall work efficiency of constructing the real blood relationship of big data. Therefore, how to improve the work efficiency of constructing the real blood relationship of big data has become a technical problem that needs to be solved urgently.

[0058] Based on this, the embodiments of the present application provide a method, device, computer equipment, computer-readable storage medium, and computer program product for constructing data lineage relationships, aiming to improve the work efficiency of constructing real lineage relationships of big data. The embodiments of the present application obtain the target business data for which the data lineage relationship is to be constructed; perform structural processing on the referenced table data in multiple data rows of the target business data to obtain a first dependency relationship between the multiple referenced table data; perform structural processing on the field data of each of the multiple referenced table data to obtain a second dependency relationship between the field data of each of the multiple referenced table data; and construct the data lineage relationship of the target business data based on the first dependency relationship and the second dependency relationship.

[0059] That is, the embodiment of the present application obtains the target business data, and traverses the target business data to first perform structural processing on the referenced table data in multiple data rows of the target business data to obtain a first dependency relationship between the multiple referenced table data, and also performs structural processing on the field data of each of the multiple referenced table data to obtain a second dependency relationship between the field data of each of the multiple referenced table data, and then constructs the data lineage relationship of the target business data based on the first dependency relationship and the second dependency relationship. In this way, compared with the method of constructing the real lineage of data based on the dependency relationship of manually maintaining big data task scheduling, the embodiment of the present application parses the target business data to construct the real lineage relationship of the table data in the business data depending on the table data, which can effectively improve the overall work efficiency of constructing the real lineage relationship of big data.

[0060] Based on the overall concept of the above-mentioned embodiments of the present application, specific embodiments of the method, device, computer equipment, computer readable storage medium and computer program product for constructing data lineage relationships provided by the embodiments of the present application are proposed. First, each specific embodiment of the method for constructing data lineage relationships in the embodiments of the present application is described in detail.

[0061] It should be noted that in each specific implementation of the method for constructing data kinship relationships provided in the embodiments of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiments of the present application need to obtain sensitive personal information of the user, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiments of the present application will be obtained.

[0062] In addition, the method for constructing data lineage relationships provided in the embodiments of the present application relates to the field of big data processing technology. The method for constructing data lineage relationships provided in the embodiments of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, or can be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the method for constructing data lineage relationships, etc., but is not limited to the above forms.

[0063] Alternatively, the method for constructing data kinship relationships provided by the embodiments of the present application can also be used in numerous general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer computer devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0064] For ease of understanding and explanation, the following text will take the method for constructing a data lineage relationship provided by the terminal device application embodiment of the present application as an example for detailed description. The implementation of the method for constructing a data lineage relationship provided by the embodiment of the present application in any of the above-mentioned forms of subject application can refer to the process of the method for constructing a data lineage relationship in the terminal device application described below.

[0065] Please refer to Figure 1 , Figure 1 The following is a flow chart of the steps of the method for constructing data kinship relationships in some embodiments of the present application. Figure 1 The execution order of some method steps is shown in the figure, but based on different design requirements of actual applications, the data kinship construction method provided in the embodiment of the present application can certainly adopt an execution order different from the method steps shown in the figure. That is, Figure 1The order of the steps in the method shown does not constitute a limitation on the execution logic order of the method for constructing the data kinship relationship provided in the embodiment of the present application. Figure 1 Reasonable changes in the order of the method steps shown should be included in the scope of protection of the method for constructing data kinship relationships provided in the embodiments of the present application.

[0066] like Figure 1 As shown, in some embodiments, the method for constructing data lineage relationships provided in the embodiments of the present application may include but is not limited to steps S101 to S104.

[0067] Step S101: Acquire target business data for which data lineage relationship is to be constructed.

[0068] The terminal device can obtain the business data generated by the big data business platform in real time by connecting with the big data business platform, and further pre-process the business data to obtain the target business data for which data lineage relationship needs to be established in the business data.

[0069] It should be noted that the big data business platform can be a business system that uses a distributed data storage method to store business data. Business data is script data generated in real time based on business needs and stored on the big data business platform. The script data is used to instruct the big data business platform or terminal devices related to the big data business platform to perform data scheduling for query or analysis and other processing. For example, the big data business platform is specifically a system used by financial staff to manage financial business. The terminal device communicates with the system to obtain the latest Hsql script generated by the system in real time as the business data of the data blood relationship to be constructed. Specifically, the Hsql script can be a "xxx.sh" file generated in real time by the system in response to business management needs. It uses the shll language and the content is sql in hive syntax. The terminal device can obtain the Hsql script in real time by monitoring the fixed directory on the system production hadoop cluster.

[0070] In addition, the terminal device may perform preprocessing on the acquired business data by: performing unified processing on data with different data structures and data formats, etc. It should be understood that, based on different design requirements of actual applications, the terminal device may also perform other preprocessing operations on the acquired business data, and then use the business data after the preprocessing operations as the target business data for which the data lineage relationship is to be constructed, that is, the data lineage relationship construction method provided in the embodiment of the present application does not limit the specific types of preprocessing of business data by the terminal device.

[0071] In some embodiments, the terminal device can also receive and obtain business data that requires data lineage relationship building and is actively uploaded by the staff through a data transmission interface provided for the staff, and then pre-process the business data to obtain target business data.

[0072] Step S102: Structuring the referenced table data in the multiple data rows of the target business data to obtain a first dependency relationship between the multiple referenced table data.

[0073] It should be noted that the referenced table data may be data such as "Table A" or "Table B" in a row of business code data (data row) of the target business data.

[0074] After the terminal device obtains the target business data for which data lineage relationship needs to be established, it parses the target business data to perform structural processing on the referenced table data in multiple data rows of the target business data, thereby obtaining a first dependency relationship between the multiple referenced table data.

[0075] In some embodiments, the terminal device can traverse the target business data to first confirm that there is a target data row that references table data among multiple data rows of the target business data, and then parse the target data row and structure the referenced table data in each target data row to obtain a first dependency relationship between the referenced table data.

[0076] Step S103: Structuring the field data of each of the plurality of referenced table data to obtain a second dependency relationship between the field data of each of the plurality of referenced table data.

[0077] In the process of parsing the target business data to structure the referenced table data, the terminal device further structures the field data of each referenced table data in the target business data, thereby obtaining the second dependency relationship between the field data of each referenced table data.

[0078] In some embodiments, the terminal device may choose to traverse the data rows of the target business data, and after each target data row containing referenced table data is determined, the terminal device may immediately parse the target data row to structure the field data of the referenced table data located in the target data row and its formation logic, thereby obtaining the second dependency relationship between the field data of each referenced table data in the target business data.

[0079] In other embodiments, the terminal device may also choose to traverse all data rows of the target business data to determine all target data rows containing referenced table data, and then uniformly parse all target data rows one by one to structure the field data of the referenced table data contained therein and its formation logic, thereby obtaining the second dependency relationship between the field data of each referenced table data in the target business data.

[0080] Step S104: constructing a data lineage relationship of the target business data based on the first dependency relationship and the second dependency relationship.

[0081] After the terminal device parses the target business data to obtain the first dependency relationship by structured reference table data, and obtains the second dependency relationship by structured reference table data field data, the terminal device further performs intelligent verification on the first dependency relationship and the second dependency relationship, so as to directly represent the real lineage of the target business data by the first dependency relationship and the second dependency relationship when confirming that the first dependency relationship and the second dependency relationship pass the verification, so as to construct the data lineage relationship of the target business data.

[0082] It should be noted that the intelligent verification of the first dependency and the second dependency by the terminal device can be to display the first dependency and the second dependency, so as to accept the manual sampling of the first dependency and the second dependency by the staff. When the staff completes the sampling of the first dependency and the second dependency and inputs an instruction to confirm that the verification is correct to the terminal device, the terminal device can confirm that the first dependency and the second dependency have passed the verification.

[0083] In some embodiments, the terminal device performs intelligent verification on the first dependency and the second dependency separately, and when it is confirmed that any one of them has passed the verification, the data lineage relationship of the target business data is first constructed based on the dependency that has passed the verification. For example, after the terminal device parses the target business data to obtain the first dependency by structuring the referenced table data, it immediately performs intelligent verification on the first dependency, and when it is confirmed that the first dependency has passed the verification, the first dependency is directly used to represent the real lineage between the referenced table data in the target business data, thereby constructing the data lineage relationship between the tables in the target business data. Afterwards, after the terminal device further structures the field data of the referenced table data to obtain the second dependency, it also immediately performs intelligent verification on the second dependency, and when it is confirmed that the second dependency has passed the verification, the second dependency is directly used to represent the real lineage between the field data of the referenced table data in the target business data, thereby constructing the data lineage relationship between the table field data in the target business data.

[0084] In the embodiment of the present application, the business data generated in real time by the big data business platform is obtained by docking the terminal device with the big data business platform, and the target business data that needs to build a data lineage relationship in the business data is obtained by further preprocessing the business data. Afterwards, the target business data is parsed by the terminal device to structure the reference table data in multiple data rows of the target business data, thereby obtaining the first dependency relationship between multiple reference table data. In the process of parsing the target business data to structure the reference table data by the terminal device, for the reference table data in the target business data, the terminal device is further used to structure the field data of each reference table data, thereby obtaining the second dependency relationship between the field data of each reference table data. Finally, the first dependency and the second dependency are intelligently verified by the terminal device, so that when the first dependency and the second dependency are confirmed to pass the verification, the first dependency and the second dependency are directly represented by the real lineage of the target business data to construct the data lineage relationship of the target business data.

[0085] In this way, compared with the method of constructing the true lineage of data based on the dependency relationship of manually maintaining big data task scheduling, the embodiment of the present application parses the target business data to construct the true lineage relationship of the table data in the business data and the table data in the business data, which can effectively improve the overall work efficiency of constructing the true lineage relationship of big data.

[0086] When the terminal device obtains the target business data, it can filter the received business data to screen out all the valid data related to the data scheduling task, and then use the valid data as the target business data to parse it to build the data lineage relationship. In this way, it can avoid parsing invalid data in the business data that is not related to the data scheduling task to waste the computing resources of the terminal device, thereby further improving the overall work efficiency of building data lineage relationships.

[0087] Please refer to Figure 2 , Figure 2 for Figure 1 Schematic diagram of the detailed process flow of step S101.

[0088] like Figure 2 As shown, in some embodiments, the above-mentioned step S101: obtaining the target business data for which the data lineage relationship is to be constructed may include the following steps S201 and S202.

[0089] Step S201: traverse the business data for which data lineage relationships are to be constructed to confirm invalid data in the business data that is not related to the data processing task.

[0090] After obtaining the business data for which data lineage relationships need to be established, the terminal device traverses the business data to confirm invalid data such as Chinese annotations in the business data that are irrelevant to data processing tasks (scheduling, query, analysis, etc.).

[0091] Step S202: Clean the invalid data to obtain target business data for which data lineage relationship is to be constructed in the business data.

[0092] After the terminal device traverses the business data to confirm the invalid data in the business data that is not related to the data processing task, it filters and cleans the invalid data, and uses the valid data related to the data processing task remaining in the business data after cleaning as the target business data for the data blood relationship to be established.

[0093] Exemplarily, when the business data acquired by the terminal device and requiring the construction of data lineage relationships is an Hsql script, the terminal device can specifically use the computer programming language python to traverse each line of data in the script, that is, by setting the python encoding to ISO-8859-1, the interference of the Chinese comments in the Hsql script in parsing the script code to construct data lineage relationships is resolved.

[0094] In this embodiment, since the business data received by the terminal device usually contains a large amount of redundant data that is irrelevant to the specific data processing task, and the terminal device often does not need to consider this part of redundant data when building data lineage relationships, the terminal device first cleans this part of redundant data before formally parsing the data to build data lineage relationships, and only retains the valid data (target business data) needed to build data lineage relationships in the business data for parsing. In this way, compared with code analysis of all business data to build data lineage relationships, only analyzing part of the valid data in all business data can further improve the overall work efficiency of building data lineage relationships.

[0095] When the terminal device cleans the invalid data in the business data that is not related to the data processing task to obtain the target business data for the data blood relationship to be constructed, it can specifically identify the key characters in the business data to clarify which data rows in the business data need to participate in the parsing of valid data and which data rows are invalid data that do not need to participate in the parsing. Among them, the content in the business data that only serves as a code description and annotation is invalid data that does not need to be parsed, so it is necessary to filter and clean the invalid data. For example, when the business data is an Hsql script, there will be English, Chinese, numbers and other content in the Hsql script and the position of the content is random. Since some of the content in the Hsql script is irrelevant to the sql logic and only serves as a code description and annotation, this part of the content is the invalid data that the terminal device needs to clean first.

[0096] Please refer to Figure 3 , Figure 3 for Figure 2 Schematic diagram of the detailed step flow of step S201.

[0097] like Figure 3 As shown, in some embodiments, the above-mentioned step S201: traversing the business data to be constructed with data lineage relationships to confirm invalid data in the business data that is not related to the data processing task may include steps S301 to S303 as shown below.

[0098] Step S301: traverse the business data for which data lineage relationships are to be constructed to confirm operation identification characters and statement separator characters in the business data.

[0099] When traversing the business data for which data lineage relationships need to be established, the terminal device traverses each data row of the business data one by one to confirm the operation identification characters and statement separation characters in the business data.

[0100] It should be noted that, when the business data is an Hsql script, the operation identification character in the business data can be a keyword character (hive-ve”), and the statement separator character in the business data can be a keyword character (;”). The terminal device traverses each data row of the business data to identify keyword character 1: “hive-ve” and keyword character 2: “;”, and confirms that the code line (data line) between keyword character 1 and keyword character 2 is the Hsql syntax that needs to be parsed when building data lineage relationships.

[0101] Step S302: confirm that the business data is located in the first data row before the operation identification character, and confirm that the business data is located in the second data row after the statement separation character.

[0102] After identifying and confirming the operation identification character and statement separator character in the business data, the terminal device confirms all data rows in the business data before the operation identification character as the first data row, and confirms all data rows in the business data after the statement separator character as the second data row.

[0103] Step S303: Mark the first data row and the second data row as invalid data in the business data that is irrelevant to the data processing task.

[0104] Since the code behaviors between the operation identification characters and the statement separator characters in the business data need to be parsed for grammatical content related to the data processing task, after the terminal device confirms the first data row and the second data row in the business data, it can further mark the first data row and the second data row as invalid data in the business data that are not related to the data processing task, and then filter and clean the invalid data.

[0105] When the terminal device cleans invalid data in the business data that is not related to the data processing task to obtain the target business data for the data lineage relationship to be constructed, the terminal device can further identify the comment content that does not affect the code logic for the code lines located between the operation identification characters and the statement separator characters, so as to mark these comment contents as invalid data that are not related to the data processing task for filtering and cleaning.

[0106] Please refer to Figure 4 , Figure 4 A schematic flowchart of the steps of the method for constructing data kinship relationships provided in the embodiment of the present application in other embodiments.

[0107] like Figure 4As shown, in some embodiments, the method for constructing data lineage relationships provided in the embodiments of the present application, after the above-mentioned step S303: marking the first data row and the second data row as invalid data in the business data that are irrelevant to the data processing task, may also include steps S401 to S403 as shown below.

[0108] Step S401: confirming that there is a target data row where a comment identification character exists among a plurality of third data rows where the business data is located between the operation identification character and the statement separation character.

[0109] It should be noted that, when the business data is an Hsql script, the comment identification character in the business data can be: --. In sql syntax, the character "--" represents a comment, and the content after it does not affect the code logic.

[0110] After marking the first data row and the second data row as invalid data in the business data that is not related to the data processing task, the terminal device marks multiple data rows between the operation identification character and the statement separation character in the business data as third data rows, and then traverses each third data row one by one to determine one or more target data rows in the multiple third data rows where the comment identification character exists.

[0111] Step S402: when the comment identification character is located at the beginning of the target data row, the target data row is marked as invalid data in the business data that is irrelevant to the data processing task.

[0112] When the terminal device identifies and determines that a target data row has a comment identification character in a plurality of third data rows, it can further identify whether the comment identification character is located at the beginning of the target data row or at the middle of the target data row. When the terminal device identifies that the comment identification character contained in a certain target data row is located at the beginning of the target data row, it confirms that the contents of the target data row are all comment contents that do not affect the code logic, thereby marking the entire target data row as invalid data in the business data that is irrelevant to the data processing task.

[0113] Step S403: when the comment identification character is located in the middle of the target data row, the comment identification character and the data after the comment identification character in the target data row are marked as invalid data in the business data that is irrelevant to the data processing task.

[0114] When the terminal device recognizes that the comment identification character contained in a target data row is located in the middle of the target data row, it confirms that the content in the target data row starting from and after the comment identification character is comment content that does not affect the code logic, and thus marks the comment identification character in the target data row and the data in the target data row that is located after the comment identification character as invalid data in the business data that is not related to the data processing task.

[0115] After filtering and cleaning invalid data in the business data that is not related to the data processing task, the terminal device can obtain the target business data that only contains valid data (such as a complete SQL statement in an Hsql script). In this way, the terminal device performs structured processing on the referenced table data and the field data of the referenced table data in the target business data based on the code logic of the data processing task (such as the characteristics of the SQL syntax), thereby parsing the relationship between the referenced table data and the field data of the referenced table data.

[0116] Please refer to Figure 5 , Figure 5 for Figure 1 Schematic diagram of the detailed step flow of step S102.

[0117] like Figure 5 As shown, in some embodiments, the step of "structuring the referenced table data in the multiple data rows of the target business data" in the above step S103 may include the following steps S501 to S504.

[0118] Step S501: Identify a first target data row containing a table keyword among multiple data rows of the target business data.

[0119] It should be noted that the table keyword may be the table keyword after the insert keyword in the code line.

[0120] When the terminal device parses the target business data to structure the referenced table data in the target business data, it first identifies whether multiple data rows of the target business data include table keywords one by one, and then, if it is identified that a certain data row contains the table keyword, confirms the data row as the first target data row.

[0121] Step S502: filtering the data before the table keyword and the data after the preset partition keyword in the first target data row to structure the first reference table data between the table keyword and the preset partition keyword.

[0122] It should be noted that the reference table data in the target business data includes the first reference table data. In addition, the preset partition keyword may be a partition keyword in the code line.

[0123] After the terminal device identifies the first target data row among multiple data rows of target business data, the terminal device filters the data before the table keyword and the data after the preset partition keyword in the first target data row, thereby structuring the first referenced table data between the table keyword and the preset partition keyword.

[0124] Step S503: Identify a second target data row containing a reference keyword pair among the multiple data rows of the target business data.

[0125] It should be noted that the reference keyword pairs include: from keyword + where keyword, and join keyword + on keyword.

[0126] When the terminal device parses the target business data to structure the reference table data in the target business data, it can also identify whether multiple data rows of the target business data include reference keyword pairs one by one, and then, if a data row is identified as containing the reference keyword pair, confirm the data row as the second target data row.

[0127] Step S504: In the second target data row, second reference table data between the reference keyword pairs is structured.

[0128] It should be noted that the reference table data in the target business data also includes second reference table data.

[0129] After the terminal device identifies the second target data row among multiple data rows of target business data, the terminal device filters the data before and after the reference keyword pair in the second target data row, thereby only structuring the second reference table data between the reference keyword pair.

[0130] Exemplarily, the terminal device first identifies the data rows with the insert keyword and the table keyword in the multiple data rows of the target business data, thereby obtaining the first target data row: Insert overwrite table table a partition(). Afterwards, the terminal device filters the content before the table keyword in the first target data row, and also filters the content after the partition keyword, thereby obtaining the first reference table data: table a.

[0131] In addition, the terminal device also identifies data rows with from keyword + where keyword in multiple data rows of the target business data, thereby obtaining a second target data row: from table b where. Afterwards, the terminal device filters the content before the from keyword in the second target data row, and filters the content after the where keyword, thereby obtaining the second reference table data: table b. Furthermore, the terminal device also identifies data rows with join keyword + on keyword in multiple data rows of the target business data, thereby obtaining a second target data row: join table c on. Afterwards, the terminal device filters the content before the join keyword in the second target data row, and filters the content after the on keyword, thereby obtaining the second reference table data: table c.

[0132] In this way, the terminal device can identify the referenced table data in the target business data through the above-mentioned steps of structuring table a in the first target data row, structuring table b in the second target data row, and structuring table c in the second target data row: the relationship between table a and table b, table c (first dependency relationship) is: table a<-table b, table a<-table c (the upstream lineage of table a is table b, table c).

[0133] The terminal device may further perform structural processing on the field data of the referenced table data based on the structural processing of the referenced table data, thereby obtaining a second dependency relationship between the field data of the plurality of referenced table data.

[0134] Please refer to Figure 6 , Figure 6 for Figure 1 Schematic diagram of the detailed process flow of step S103.

[0135] like Figure 6 As shown, in some embodiments, the step of "structuring the field data of each of the multiple reference table data" in the above step S103 may include steps S601 to S603 as shown below.

[0136] Step S601: Identify the first first search keyword pair after the first target data row, and identify the first second search keyword pair after the second target data row.

[0137] It should be noted that the first search keyword pair may be the select keyword + from keyword in the code line. The second search keyword pair may be the same as the first search keyword pair, except that the first search keyword pair is located after the first target data line, while the second search keyword pair is located after the second target data line.

[0138] After the first reference table data located between the table keyword and the preset partition keyword in the structured first target data row, the terminal device may further identify the data row after the first target data row to identify the first first search keyword pair after the first target data row. In addition, after the second reference table data located between the reference keyword pair in the structured second target data row, the terminal device may further identify the data row after the second target data row to identify the first second search keyword pair after the second target data row.

[0139] Step S602: Structuring the first field data between the first search keyword pair and the generation logic of the first field data; the first field data is the field data of the first reference table data.

[0140] After identifying the first search keyword pair after the first target data row, the terminal device identifies the first field data located between the first search keyword pair and the generation logic of the first field data, thereby achieving structured processing of the first field data of the first referenced table data in the first target data row and its generation logic.

[0141] Step S603: Structuring the second field data between the second search keyword pair; the second field data is the field data of the second reference table data.

[0142] After identifying the second search keyword pair after the second target data row, the terminal device also identifies the second field data located between the second search keyword pair, thereby implementing structured processing of the second field data of the second reference table data in the second target data row.

[0143] Exemplarily, after the terminal device identifies the first target data row: Insert overwrite table table apartition(), and performs structural processing on the first reference table data: table a in the first target data row, it further identifies the first first search keyword pair after the first target data row: Select keyword + From keyword, thereby obtaining the following data row:

[0144] Select

[0145] ,b1 / c1 as a1

[0146] ,a2

[0147] From

[0148] Then, the terminal device recognizes the first field data a1 after the keyword as between the select keyword and the from keyword, which is the field data of the first reference table data table a. Moreover, the terminal device also recognizes the content before the keyword as and between the characters (,) as the formation logic of the first field data a1 (specifically: 1. addition, subtraction, multiplication and division nvl(b1,0) / nvl(b1,0), b1+c1, b1-c1, b1*c1, 2. aggregation operation: sum(nvl(b1,0)), AVG(c1) 3. deduplication operation: count(distinct b1), 4. logical operation: if(nvl(b1,0)>0,c1,b1), 5. logical operation: Case when b1>0thenc1 else b1 end).

[0149] In addition, based on the same principle as identifying the first table field data a1 between the select keyword and the from keyword, the terminal device first identifies the first second search keyword pair after the second target data row: Select keyword + From keyword, and then identifies the second field data between the select keyword and the from keyword: b1, and the second field data c1, thereby confirming that b1 comes from table b and c1 comes from table c.

[0150] After the terminal device identifies the formation logic of the first field data a1, it only needs to check whether b1 and c1 exist in the formation logic to determine whether the calculation factors of a1 include b1 and c1, thereby deriving that the upstream lineage of a1 is b1 and c1. In this way, the terminal device can obtain the relationship between the table level and the field level by parsing the Hsql script, wherein the first dependency at the table level is: the upstream lineage of table a is table b and table c, and the second dependency at the field level is: the upstream lineage of field a1 of table a is field b1 of table b and field c1 of table c.

[0151] Please refer to Figure 7 , Figure 7 for Figure 1 Schematic diagram of the detailed step flow of step S104.

[0152] like Figure 7 As shown, in some embodiments, the above-mentioned step S104: constructing the data lineage relationship of the target business data based on the first dependency relationship and the second dependency relationship may include steps S701 to S703 as shown below.

[0153] Step S701: output and display the first dependency relationship and the second dependency relationship, and receive a sampling request for a target dependency relationship in the first dependency relationship and the second dependency relationship.

[0154] After obtaining the first dependency and the second dependency, the terminal device can output and display the first dependency and the second dependency through a visual human-computer interaction interface provided to the staff. In this way, the staff can input a sampling request for the target dependency in the first dependency and the second dependency through the visual human-computer interaction interface, and the terminal device can receive the sampling request.

[0155] Step S702: In response to the sampling request, the target dependency relationship is highlighted.

[0156] After receiving the sampling request input by the staff, the terminal device immediately responds to the sampling request, thereby highlighting the target dependency relationship pointed to by the sampling request through the visual human-computer interaction interface. Alternatively, the terminal device can also highlight the target dependency relationship and the data row in the corresponding target business data through the visual human-computer interaction interface.

[0157] Step S703: When a confirmation indication for the target dependency is received, the first dependency and the second dependency are used as real data lineages to construct the data lineage of the target business data.

[0158] After the terminal device highlights the target dependency in response to the sampling request, the staff can further check the reference table data and / or the field data of the reference table data corresponding to the structured target dependency of the terminal device, so as to confirm whether the target dependency obtained based on the reference table data and / or the reference table data is accurate, and if confirmed to be correct, input a confirmation indication of the target dependency through the visual human-computer interaction interface. After receiving the confirmation indication for the target dependency, the terminal device can confirm that the sampling inspection conducted by the staff on the first dependency and the second dependency has been confirmed to be correct, so as to use the first dependency and the first dependency as the real data lineage to construct the data lineage relationship of the target business data.

[0159] In this embodiment, after obtaining the first dependency relationship through the reference table data in the structured target business data of the terminal device, and obtaining the second dependency relationship through the field data of the structured reference table data, the first dependency relationship and the second dependency relationship are output and displayed, so that the staff can perform random inspections on the displayed first dependency relationship and the second dependency relationship, and when the staff completes the random inspection and enters the confirmation instruction, the first dependency relationship and the second dependency relationship are used as the real lineage of the target business data, thereby constructing the data lineage relationship of the target business data. In this way, by adding the link of manual verification based on the dependency relationship to construct the data lineage relationship, double insurance for accurately constructing the data lineage relationship is achieved, ensuring the authenticity and completeness of the data lineage relationship. In addition, in this embodiment, by visually displaying the data relationship, the staff can more intuitively understand the real lineage relationship of the business data.

[0160] See also Figure 8 The embodiment of the present application also provides a device for constructing a data lineage relationship, which can implement the above-mentioned method for constructing a data lineage relationship, and the device includes:

[0161] An acquisition module is used to acquire target business data of the data lineage relationship to be constructed;

[0162] A structured processing module, configured to perform structured processing on the reference table data in the multiple data rows of the target business data to obtain a first dependency relationship between the multiple reference table data; and perform structured processing on the field data of each of the multiple reference table data to obtain a second dependency relationship between the field data of each of the multiple reference table data;

[0163] A data lineage construction module is used to construct the data lineage relationship of the target business data based on the first dependency relationship and the second dependency relationship.

[0164] In some embodiments, the acquisition module is also used to traverse the business data for which data lineage relationships are to be constructed to confirm invalid data in the business data that is not related to the data processing task; and to clean the invalid data to obtain the target business data for which data lineage relationships are to be constructed in the business data.

[0165] In some embodiments, the acquisition module is also used to traverse the business data for which data lineage relationships are to be constructed to confirm operation identification characters and statement separator characters in the business data; confirm that the business data is located in the first data row before the operation identification characters, and confirm that the business data is located in the second data row after the statement separator characters; and mark the first data row and the second data row as invalid data in the business data that is not related to the data processing task.

[0166] In some embodiments, the acquisition module is also used to confirm that there is a target data row with a comment identifier character in a plurality of third data rows where the business data is located between the operation identifier character and the statement separator character; when the comment identifier character is located at the starting position of the target data row, marking the target data row as invalid data in the business data that is not related to the data processing task; and, when the comment identifier character is located in the middle position of the target data row, marking the comment identifier character and the data in the target data row that is located after the comment identifier character as invalid data in the business data that is not related to the data processing task.

[0167] In some embodiments, the structured processing module is also used to identify a first target data row containing a table keyword among multiple data rows of the target business data; filter the data located before the table keyword and the data located after the preset partition keyword in the first target data row to structure the first reference table data between the table keyword and the preset partition keyword; identify a second target data row containing a reference keyword pair among multiple data rows of the target business data; and, in the second target data row, structure the second reference table data between the reference keyword pair; the reference table data includes the first reference table data and the second reference table data.

[0168] In some embodiments, the structured processing module is also used to identify the first first search keyword pair after the first target data row, and to identify the first second search keyword pair after the second target data row; to structure the first field data between the first search keyword pair and the generation logic of the first field data; the first field data is the field data of the first reference table data; and to structure the second field data between the second search keyword pair; the second field data is the field data of the second reference table data.

[0169] In some embodiments, the data lineage construction module is also used to output and display the first dependency and the second dependency, and receive a sampling request for a target dependency in the first dependency and the second dependency; highlight the target dependency in response to the sampling request; and, upon receiving a confirmation indication for the target dependency, use the first dependency and the second dependency as the true lineage of the data to construct the data lineage of the target business data.

[0170] It should be noted that the specific implementation of the data lineage relationship construction device is basically the same as the specific implementation example of the above-mentioned data lineage relationship construction method, and will not be repeated here.

[0171] The embodiment of the present application also provides a computer device, the computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned data kinship construction method when executing the computer program. The computer device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0172] See also Fig. 9 , Fig. 9 The hardware structure of a computer device according to another embodiment is shown, and the computer device includes:

[0173] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0174] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the method for constructing the data kinship relationship in the embodiment of this application;

[0175] Input / output interface 903, used to implement information input and output;

[0176] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0177] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0178] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0179] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, it implements the above-mentioned method for constructing data blood relationship.

[0180] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0181] An embodiment of the present application also provides a computer program product, including a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the method for constructing the above-mentioned data blood relationship is implemented.

[0182] The specific implementation methods of the computer device, computer-readable storage medium, and computer program product provided in the embodiments of the present application are basically the same as the specific implementation methods of the above-mentioned data lineage relationship construction method, and will not be repeated here.

[0183] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0184] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0185] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0186] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0187] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0188] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0189] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0190] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0191] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0192] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0193] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for constructing data kinship relationships, characterized in that: The method comprises: Obtain the target business data of the data lineage relationship to be constructed; Performing structural processing on the referenced table data in the multiple data rows of the target business data to obtain a first dependency relationship between the multiple referenced table data; Performing structural processing on the field data of each of the plurality of referenced table data to obtain a second dependency relationship between the field data of each of the plurality of referenced table data; A data lineage relationship of the target business data is constructed based on the first dependency relationship and the second dependency relationship.

2. The method according to claim 1, characterized in that The step of obtaining target business data of the data blood relationship to be constructed includes: Traversing the business data for which data lineage relationships are to be constructed to confirm invalid data in the business data that is not related to the data processing task; The invalid data is cleaned to obtain target business data for which data lineage relationship is to be constructed in the business data.

3. The method according to claim 2, characterized in that The traversing of the business data to be constructed with data lineage relationships to confirm invalid data in the business data that is not related to the data processing task includes: Traversing the business data for which data lineage relationships are to be constructed to confirm operation identification characters and statement separation characters in the business data; Confirming that the business data is located in a first data row before the operation identification character, and confirming that the business data is located in a second data row after the statement separation character; The first data row and the second data row are marked as invalid data in the business data that is irrelevant to the data processing task.

4. The method according to claim 3, characterized in that After marking the first data row and the second data row as invalid data in the business data that is irrelevant to the data processing task, the method further includes: In the plurality of third data rows where the business data is located between the operation identification character and the statement separation character, confirming that a target data row with a comment identification character exists; In the case where the comment identification character is located at the beginning of the target data row, marking the target data row as invalid data in the business data that is irrelevant to the data processing task; When the comment identification character is located in the middle of the target data row, the comment identification character and the data after the comment identification character in the target data row are marked as invalid data in the business data that is irrelevant to the data processing task.

5. The method according to claim 1, characterized in that: The structural processing of the reference table data in the multiple data rows of the target business data includes: Identify a first target data row containing a table keyword among the multiple data rows of the target business data; Filtering the data before the table keyword and the data after the preset partition keyword in the first target data row to structure the first reference table data between the table keyword and the preset partition keyword; Identifying a second target data row containing a reference keyword pair among the multiple data rows of the target business data; In the second target data row, structuring the second reference table data between the reference keyword pairs; The reference table data includes the first reference table data and the second reference table data.

6. The method according to claim 5, characterized in that The structural processing of the field data of each of the plurality of reference table data includes: identifying a first first search keyword pair after the first target data row, and identifying a first second search keyword pair after the second target data row; Structuring the first field data between the first search keyword pair and the generation logic of the first field data; the first field data is the field data of the first reference table data; The second field data between the second search keyword pairs is structured; the second field data is the field data of the second reference table data.

7. The method according to any one of claims 1 to 6, characterized in that The step of constructing a data lineage relationship of the target business data based on the first dependency relationship and the second dependency relationship includes: Output and display the first dependency relationship and the second dependency relationship, and receive a sampling request for a target dependency relationship in the first dependency relationship and the second dependency relationship; In response to the sampling request, highlighting the target dependency; When a confirmation indication for the target dependency is received, the first dependency and the second dependency are used as real data lineages to construct a data lineage relationship for the target business data.

8. A device for constructing data blood relationship, characterized in that: The device comprises: An acquisition module is used to acquire target business data of the data lineage relationship to be constructed; A structured processing module, configured to perform structured processing on the reference table data in the multiple data rows of the target business data to obtain a first dependency relationship between the multiple reference table data; and perform structured processing on the field data of each of the multiple reference table data to obtain a second dependency relationship between the field data of each of the multiple reference table data; A data lineage construction module is used to construct the data lineage relationship of the target business data based on the first dependency relationship and the second dependency relationship.

9. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the method for constructing data kinship relationships as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for constructing data kinship relationships described in any one of claims 1 to 7 is implemented.

Images