Data lineage analysis method, device, computer equipment and storage medium

Abstract

The embodiments of the present invention disclose a data lineage analysis method, apparatus, computer equipment and storage medium. The method includes: obtaining a data source to be analyzed; configuring relevant information of the data processing process according to the data source to be analyzed; creating a job-level thread and a field-level thread; using the job-level thread in combination with the relevant information to analyze the lineage message to obtain the field lineage message, and storing the field lineage message in the message queue; using the field-level thread to obtain the field lineage message from the message queue; analyzing the field lineage and field processing process according to the field lineage message to obtain the analysis result; storing the analysis result and the relevant information in the database. By implementing the method of the embodiment of the present invention, the problems of being unable to uniformly parse the lineage of different engines, the low accuracy of parsing the lineage relationship and the high complexity can be solved, the data processing process can be analyzed, and the analysis results can be obtained in real time.

Description

Technical Field

[0001] The present invention relates to a data analysis method, and more specifically to a data lineage analysis method, apparatus, computer equipment, and storage medium. Background Art

[0002] Since the 1990s, digital technology has developed rapidly. Modern information technology represented by computers has provided methods and means for data processing. Digital technology has transformed from a tool to help economic development to the core to lead economic development, and the "digital economy" was born.

[0003] In response to the urgent needs of enterprises and institutions, after years of development, the industry now offers a number of mature data middle-office solutions to help enterprises transform towards digitalization. One of the core functions of the data middle-office is data processing, which mainly involves writing SQL, code, JAR packages, etc. supported by HIVE, SPARK, and FLINK engines. Data lineage analysis based on this requires reliance on specific SQL parsers or engines, which raises the technical threshold, and the accuracy of the analysis is difficult to achieve commercial levels, especially for field-level lineage. The data processing process links are relatively long, the processing methods are diverse, and the SQL is relatively complex. On this basis, it is very difficult to analyze the processing links and processes of a specific field, especially if there are multiple engines. The parsing methods of each engine vary greatly, and the complexity increases linearly with the number of engines. Moreover, the performance indicators and accuracy of the analysis cannot be guaranteed.

[0004] Therefore, it is necessary to design a new method to solve the problem that different engines cannot uniformly analyze the blood relationship, the accuracy of analyzing the blood relationship is low and the complexity is high, and the data processing process can be analyzed to obtain the analysis results in real time. Summary of the Invention

[0005] The purpose of the present invention is to overcome the defects of the prior art and provide a data lineage analysis method, device, computer equipment and storage medium.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a data lineage analysis method, comprising:

[0007] Obtain the data source to be analyzed;

[0008] Configuring relevant information of the data processing process according to the data source to be analyzed;

[0009] Create job-level threads and field-level threads;

[0010] Utilizing the job-level thread in combination with the relevant information to analyze the lineage message to obtain field lineage message, and storing the field lineage message in a message queue;

[0011] Obtaining the field lineage message from the message queue using the field-level thread;

[0012] Analyzing the field lineage and field processing process according to the field lineage information to obtain an analysis result;

[0013] Storing the analysis results and the relevant information in a database;

[0014] The analyzing the lineage message by using the job-level thread in combination with the relevant information to obtain field lineage message, and storing the field lineage message in a message queue includes:

[0015] Filtering unchanged table-level configuration information according to the relevant information to obtain modified table-level configuration information;

[0016] Assemble the modified table-level configuration information to obtain field lineage information;

[0017] Storing the field lineage information in the message queue;

[0018] Analyze the lineage of data sources and modified data tables;

[0019] Get the cache data stored by the field-level thread;

[0020] Polling to determine whether the task corresponding to the field-level thread is completed;

[0021] If it is determined that the task corresponding to the field-level thread is completed, responding to the analysis result of the field-level thread;

[0022] If it is determined that the task corresponding to the field-level thread is not completed, the polling is performed to determine whether the task corresponding to the field-level thread is completed.

[0023] A further technical solution is: the relevant information includes data source information, table information, information on whether table-level configuration has been changed, field-level data processing information, and field-level output field mapping information.

[0024] A further technical solution is as follows: creating a job-level thread and a field-level thread includes:

[0025] Use the main thread to start multiple job-level threads and field-level threads based on configuration.

[0026] A further technical solution is: the polling to determine whether the task corresponding to the field-level thread is completed includes:

[0027] Determine whether the number of completed field-level tasks in the cached data is equal to the size of the message queue;

[0028] If the number of completed field-level tasks in the cached data is equal to the size of the message queue, determining that the task corresponding to the field-level thread is completed;

[0029] If the number of completed field-level tasks in the cache data is not equal to the size of the message queue, it is determined that the task corresponding to the field-level thread is not completed.

[0030] A further technical solution is: analyzing the field lineage and field processing process according to the field lineage message to obtain an analysis result, including:

[0031] Analyzing the lineage and field processing of the output fields in the lineage message to obtain lineage analysis results and field processing analysis results;

[0032] The lineage analysis result and the analysis result of the field processing process are used to verify whether they are correct. When the verification is correct, the lineage analysis result and the analysis result of the field processing process are stored in the cache to obtain the analysis result, wherein the analysis result includes the cache data.

[0033] A further technical solution is that the storing of the analysis results and the related information in a database includes:

[0034] The analysis results, data source lineage, modified data table lineage, and the related information are stored in a database.

[0035] The present invention also provides a data lineage analysis device, which is characterized by comprising:

[0036] A data source acquisition unit, used to acquire the data source to be analyzed;

[0037] a configuration unit, configured to configure relevant information of a data processing process according to the data source to be analyzed;

[0038] Creation unit, used to create job-level threads and field-level threads;

[0039] a job-level analysis unit, configured to analyze the lineage message using the job-level thread in combination with the relevant information to obtain field lineage message, and store the field lineage message in a message queue;

[0040] A message acquisition unit, configured to acquire the field lineage message from the message queue using the field-level thread;

[0041] A field-level analysis unit, configured to analyze field lineage and field processing according to the field lineage information to obtain an analysis result;

[0042] A storage unit, configured to store the analysis results and the related information in a database;

[0043] The job-level analysis unit includes a filtering subunit, an assembling subunit, a storing subunit, a lineage analysis subunit, a cache data acquisition subunit, a task judgment subunit, and a response subunit;

[0044] A filtering subunit is used to filter the unchanged table-level configuration information according to the relevant information to obtain the modified table-level configuration information; an assembling subunit is used to assemble the modified table-level configuration information to obtain the field lineage message; a storage subunit is used to store the field lineage message in the message queue; a lineage analysis subunit is used to analyze the data source lineage and the modified data table lineage; a cache data acquisition subunit is used to obtain the cache data stored by the field-level thread; a task judgment subunit is used to poll and judge whether the task corresponding to the field-level thread is completed; if it is determined that the task corresponding to the field-level thread is not completed, then the polling is performed to judge whether the task corresponding to the field-level thread is completed; a response subunit is used to respond to the analysis result of the field-level thread if it is determined that the task corresponding to the field-level thread is completed.

[0045] The present invention further provides a computer device, comprising a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above method when executing the computer program.

[0046] The present invention also provides a storage medium storing a computer program, which can implement the above method when executed by a processor.

[0047] The beneficial effects of the present invention compared with the prior art are: the present invention constructs job-level threads and field-level threads by configuring relevant information of the data processing process, uses job-level threads to analyze bloodline messages, uses the field-level threads to analyze field bloodline and field processing processes, and stores the results in a database to solve the problem of being unable to uniformly parse the bloodline relationships of different engines, the low accuracy and high complexity of parsing the bloodline relationships, and can analyze the data processing process and obtain analysis results in real time.

[0048] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0050] Figure 1 A schematic diagram of an application scenario of the data lineage analysis method provided by an embodiment of the present invention;

[0051] Figure 2 A schematic diagram of the process of the data lineage analysis method provided by an embodiment of the present invention;

[0052] Figure 3 A schematic diagram of a sub-process of a data lineage analysis method provided by an embodiment of the present invention;

[0053] Figure 4 A schematic diagram of a sub-process of a data lineage analysis method provided by an embodiment of the present invention;

[0054] Figure 5 A schematic diagram of a sub-process of a data lineage analysis method provided by an embodiment of the present invention;

[0055] Figure 6 A schematic block diagram of a data lineage analysis device provided by an embodiment of the present invention;

[0056] Figure 7 A schematic block diagram of a job-level analysis unit of a data lineage analysis device provided in an embodiment of the present invention;

[0057] Figure 8 A schematic block diagram of a task determination subunit of a data lineage analysis device provided by an embodiment of the present invention;

[0058] Figure 9 A schematic block diagram of a field-level analysis unit of a data lineage analysis device provided in an embodiment of the present invention;

[0059] Figure 10 A schematic block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0061] It will be understood that when used in this specification and the appended claims, the terms “comprises” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0062] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the present invention. As used in the specification and appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0063] It should be further understood that the term "and / or" used in the present description and appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0064] See also Figure 1 and Figure 2 , Figure 1 Schematic diagram of an application scenario of the data lineage analysis method provided in an embodiment of the present invention. Figure 2 This is a schematic flow chart of a data lineage analysis method provided in an embodiment of the present invention. This data lineage analysis method is applied to a server. The server interacts with a terminal, obtains the data source to be analyzed from the terminal, and processes the data lineage by configuring information, performing data lineage analysis, and storing the configuration data and analysis results in a database.

[0065] Figure 2 FIG. 1 is a flow chart of the data lineage analysis method provided by an embodiment of the present invention. Figure 2 As shown, the method includes the following steps S110 to S170.

[0066] S110: Obtain the data source to be analyzed.

[0067] In this embodiment, the data source to be analyzed refers to data input from a terminal that requires data lineage analysis.

[0068] S120: Configure relevant information of the data processing process according to the data source to be analyzed.

[0069] In this embodiment, the relevant information includes data source information, table information, information on whether table-level configuration has been changed, field-level data processing information, and field-level output field mapping information.

[0070] The data source information includes the data source type, IP address, port number, instance name, database name, user name, and password of the input and output data sources; the table information includes the input table name and the output table name; the table-level configuration change information refers to whether the table-level configuration information has been changed, and can be selected as yes or no; the field-level data processing information includes the field processing process and the field lineage information during the processing process, which is used to quickly, accurately, and uniformly analyze the field processing process and lineage information, the input field name during the processing process, multiple separated by commas, and the output field name during the processing process; the field-level output field mapping information includes the mapped input field name and the mapped output field name. This information, combined with the field-level processing process information, can be used to quickly, accurately, and uniformly analyze the field lineage information; wherein the above-mentioned input is the data source end; the above-mentioned output is the data target end. The table-level configuration change information is used for incremental analysis of data lineage and processing process information.

[0071] S130: Create a job-level thread and a field-level thread.

[0072] In this embodiment, a main thread is used to launch multiple job-level and field-level threads on a configuration-by-configuration basis. These threads perform analysis in parallel, enabling parallel analysis of data sources, table lineages, and field lineages. Concurrent analysis of job-level threads involves setting the number of job threads to a configurable value, analyzing multiple configurations simultaneously and adjusting dynamically based on server performance. Similarly, setting the number of field threads to a configurable value, analyzing multiple fields simultaneously and adjusting dynamically based on server performance, allows for concurrent analysis of multiple fields.

[0073] S140. Analyze the lineage message using the job-level thread in combination with the relevant information to obtain field lineage message, and store the field lineage message in a message queue.

[0074] In this embodiment, the field lineage message refers to the job-level thread traversing the output field set, and each field extracts the lineage and processing process information related to the current field from the field-level data processing information and the field-level output field mapping information, and the content is composed of the lineage and processing process information related to the current field.

[0075] In one embodiment, see Figure 3 , the above-mentioned step S140 may include steps S141~S147.

[0076] S141. Filter unchanged table-level configuration information according to the relevant information to obtain modified table-level configuration information.

[0077] S142. Assemble the modified table-level configuration information to obtain field lineage information;

[0078] S143. Store the field lineage information into a message queue.

[0079] In this embodiment, each piece of combined field lineage and processing information is put into a collection in memory to achieve the purpose of storing it in the message queue.

[0080] S144. Analyze the lineage of the data source and the lineage of the modified data tables.

[0081] Specifically, analyzing the lineage of data sources means traversing the set of output data sources, and for each output data source, analyzing the set of input data sources that are related to it in the data processing process; analyzing the lineage of modified data tables means traversing the set of output data tables, and for each output data table, first determining whether the configuration has been changed, and if there has been a change, analyzing the set of input data tables that are related to it in the data processing process.

[0082] S145. Obtain cache data stored by the field-level thread.

[0083] In this embodiment, the cached data refers to the results of lineage analysis and the analysis results of the field processing process.

[0084] S146: Poll to determine whether the task corresponding to the field-level thread is completed.

[0085] In one embodiment, see Figure 4 , the above-mentioned step S146 may include steps S1461~S1463.

[0086] S1461. Determine whether the number of completed field-level tasks in the cached data is equal to the size of the message queue;

[0087] S1462: If the number of completed field-level tasks in the cached data is equal to the size of the message queue, determine that the task corresponding to the field-level thread is completed;

[0088] S1463: If the number of completed field-level tasks in the cached data is not equal to the size of the message queue, it is determined that the task corresponding to the field-level thread is not completed.

[0089] The completed field data is stored in the cache and used for comparison with the size of the field set to be analyzed during job-level thread polling to determine whether the field-level task is completed. The size of the field set to be analyzed is the size of the message queue.

[0090] S147: If it is determined that the task corresponding to the field-level thread is completed, respond to the analysis result of the field-level thread.

[0091] In this embodiment, responding to the analysis result in real time means responding the analysis result to the requesting party, ie, the terminal, in real time.

[0092] If it is determined that the task corresponding to the field-level thread is not completed, step S146 is executed.

[0093] In another embodiment, after three pollings to determine whether the task corresponding to the field-level thread is completed, the interval time can be configured according to the actual situation, that is, the polling interval can be configured. If the field-level thread is not completed, start a polling to check whether there is an error in the field-level thread analysis. Determine the size of the error_queue[T] set. If it is greater than zero, it indicates a configuration error. Then respond to the message content in the error_queue and prompt the user to reconfigure the lineage and processing information of this output field. If the set size is empty, it indicates no configuration error. Continue to poll to determine whether the field-level thread task is completed. If the field-level thread has been completed, respond to the analysis results of the field-level thread. Polling to check whether there are errors in the field-level thread analysis improves accuracy, the job fails quickly, and prompts the user to configure the wrong field and the reason.

[0094] S150, using the field-level thread to obtain the field lineage message from the message queue;

[0095] S160: Analyze the field lineage and field processing process according to the field lineage information to obtain an analysis result.

[0096] In this embodiment, the analysis result refers to the content formed by storing the blood relationship analysis result and the analysis result of the field processing process in the cache when the verification is correct.

[0097] In one embodiment, see Figure 5 , the above-mentioned step S160 may include steps S161~S162.

[0098] S161. Analyze the lineage and field processing of the fields output in the lineage message to obtain lineage analysis results and field processing analysis results;

[0099] S162. Use the blood relationship analysis result and the analysis result of the field processing process to verify whether they are correct. When the verification is correct, store the blood relationship analysis result and the analysis result of the field processing process in the cache to obtain the analysis result, wherein the analysis result includes the cache data.

[0100] Analyzing field lineages involves analyzing the set of input fields that are related to the output fields during data processing; analyzing field processing procedures involves analyzing the processing procedures for the output fields. Verifying field lineages and field processing procedures involves verifying the lineage analysis results and the field processing results. This verifies whether the analyzed field lineage input fields are identical to the input fields used in the field processing procedures. This helps determine the correctness of the analysis results and improves their accuracy.

[0101] S170: Store the analysis results and the related information in a database.

[0102] In this embodiment, the analysis results, data source lineage, modified data table lineage, and the related information are stored in a database.

[0103] The information stored in the database includes data source lineage, modified data table lineage, field lineage, field processing process, and related information.

[0104] For example, under the data source source_input, there is a student score table (ods_student_score) with the fields id (the primary key of the student score table), student_id (the primary key of the student), student_name (the student's name), student_age (the student's age), course_id (the primary key of the course), and score (the score). From the student score table (ods_student_score), a student score summary table (dwd_student_score) needs to be generated under the data source source_output. The table contains the fields id (the primary key of the student), name (the name), age (the age), and total (the total score of all courses).

[0105] First, configure the data processing process according to the above steps: configure table_config[K] by table as the dimension, and for the output table dwd_student_score, configure the input data source information source_input and the output data source source_output information including the data source type, IP address, port number, instance name, database name, user name and password; configure table information including the input table name ods_student_score and the output table name dwd_student_score; configure table-level configuration whether to change information table_config_edit and select yes; configure field-level data processing information including the field processing process sum(score), the input field name score during the processing, and the output field name total during the processing; configure field-level output field mapping information, mapping input field name student_id, mapping output field name id, mapping input field name student_name, mapping output field name name, mapping input field name student_age, mapping output field name age, mapping input field name total, and mapping output field name total; if there are multiple tables, put them all into one configuration information job_config after configuration.

[0106] The main thread creates N job-level threads job1-N and M field-level threads task1-M, and puts them into the corresponding thread pool. The job-level threads can process up to 5 concurrently, and the field-level threads can process up to 10 concurrently. The job thread and task thread start analysis in parallel. The job thread obtains the configuration information table_config[K] of the table dimension under the job_config configuration, first filters out the tables where table_config_edit is no, and the remaining L configuration information table_config[L]; the field processing information and output field mapping information in the table are packaged into a lineage message according to the output field. The message needs to include the output data source information, output table name, output field name, input data source name, input table name, input field name, processing process, processing process input field name, processing process output field name; for messages without processing process, taking the output field id as an example, the parsed message is the output data source name source_output, the output table name dwd_student_score, the output field name id, and the input data source name source _input, input table name ods_student_score, input field name student_id, processing is empty, processing input field name is empty, processing output field name is empty; the message with processing takes the output field total as an example, the parsed message is output data source name source_output, output table name dwd_student_score, output field name total, input data source name source_input, input table name ods_student_score, input field name total, processing is sum(score), processing input field name is score, processing output field name is total, and then put the lineage message into the message queue message_queue[P]. The job thread analyzes the modified table-level configuration table_config[L], analyzes the data source lineage, inputs the analysis results into the data source source_input, outputs the data source source_output, and stores them in the memory collection source_blood[R]. It also analyzes the table-level lineage, inputs the analysis results into the data source source_input, inputs the data table ods_student_score, outputs the data source source_output, outputs the data table dwd_student_score, and stores them in the memory collection table_blood[S].The job thread then polls to obtain the cache data field_blood[Q] put into the task thread, and determines whether the number of completed field-level tasks in the cache is equal to the size of the message queue message_queue[P]. If they are equal, it means completion, and after completion, the analysis results are responded to in real time.

[0107] The task thread concurrently retrieves lineage information from the message queue message_queue[P] and analyzes field lineage. If a processing step is required, the field processing step is analyzed. For example, for the output field total, the analyzed lineage information includes the output data source name source_output, the output table name dwd_student_score, the output field name total, the input data source name source_input, the input table name ods_student_score, the data processing step sum(score), and the input field name score. The task thread also verifies that the data processing step information contains the input field name. If not, the analysis information is incorrect. The field lineage information and the error description (i.e., the data processing step information does not contain the input field name) are placed in the error set error_queue[T], prompting the user to reconfigure the lineage and processing step information for this output field. At this point, all field-level processing threads in this job are stopped.

[0108] Finally, the configuration information job_config, which includes whether the table-level records have been changed, is stored in the database for incremental analysis; the analysis data source_outputlood[R], table_blood[S], and field_blood[Q] are stored in the database.

[0109] The above-mentioned data lineage analysis method can quickly, accurately and in real time analyze data lineage relationships and processing processes, especially on big data platforms, where compatibility with multiple computing engines is required, and unified analysis can be achieved. At the same time, it is easy to implement, saving team R&D costs and avoiding major technical risks. It not only provides multi-dimensional lineage relationships, but also provides accurate data processing processes.

[0110] The above-mentioned data lineage analysis method configures relevant information of the data processing process, constructs job-level threads and field-level threads, uses job-level threads to analyze lineage messages, uses the field-level threads to analyze field lineage and field processing processes, and stores the results in the database to solve the problem of being unable to uniformly parse the lineage relationships of different engines, the low accuracy and high complexity of parsing lineage relationships, and can analyze the data processing process and obtain analysis results in real time.

[0111] Figure 6FIG is a schematic block diagram of a data lineage analysis device 300 provided by an embodiment of the present invention. Figure 6 As shown, corresponding to the above data lineage analysis method, the present invention also provides a data lineage analysis device 300. The data lineage analysis device 300 includes a unit for executing the above data lineage analysis method, and the device can be configured in a server. Figure 6 The data lineage analysis device 300 includes a data source acquisition unit 301, a configuration unit 302, a creation unit 303, a job-level analysis unit 304, a message acquisition unit 305, a field-level analysis unit 306, and a storage unit 307.

[0112] The data source acquisition unit 301 is used to acquire the data source to be analyzed; the configuration unit 302 is used to configure relevant information of the data processing process according to the data source to be analyzed; the creation unit 303 is used to create a job-level thread and a field-level thread; the job-level analysis unit 304 is used to use the job-level thread in combination with the relevant information to analyze the lineage message to obtain the field lineage message, and store the field lineage message in the message queue; the message acquisition unit 305 is used to use the field-level thread to obtain the field lineage message from the message queue; the field-level analysis unit 306 is used to analyze the field lineage and field processing process according to the field lineage message to obtain the analysis result; the storage unit 307 is used to store the analysis result and the relevant information in the database.

[0113] In one embodiment, the creation unit 303 is configured to use the main thread to start multiple job-level threads and field-level threads in configuration units.

[0114] In one embodiment, if Figure 7 As shown, the job-level analysis unit 304 includes a filtering subunit 3041 , an assembling subunit 3042 , a storage subunit 3043 , a lineage analysis subunit 3044 , a cache data acquisition subunit 3045 , a task judgment subunit 3046 and a response subunit 3047 .

[0115] The filtering subunit 3041 is used to filter the unchanged table-level configuration information according to the relevant information to obtain the modified table-level configuration information; the assembling subunit 3042 is used to assemble the modified table-level configuration information to obtain the field lineage message; the storage subunit 3043 is used to store the field lineage message in the message queue; the lineage analysis subunit 3044 is used to analyze the data source lineage and the modified data table lineage; the cache data acquisition subunit 3045 is used to obtain the cache data stored by the field-level thread; the task judgment subunit 3046 is used to poll and determine whether the task corresponding to the field-level thread is completed; if it is determined that the task corresponding to the field-level thread is completed, the polling is executed to determine whether the task corresponding to the field-level thread is completed; the response subunit 3047 is used to respond to the analysis result of the field-level thread if it is determined that the task corresponding to the field-level thread is not completed.

[0116] In one embodiment, if Figure 8 As shown, the task judgment subunit 3046 includes a quantity judgment module 30461 , a first determination module 30462 and a second determination module 30463 .

[0117] The quantity judgment module 30461 is used to judge whether the number of completed field-level tasks in the cache data is equal to the size of the message queue; the first determination module 30462 is used to determine that the task corresponding to the field-level thread is completed if the number of completed field-level tasks in the cache data is equal to the size of the message queue; the second determination module 30463 is used to determine that the task corresponding to the field-level thread is not completed if the number of completed field-level tasks in the cache data is not equal to the size of the message queue.

[0118] In one embodiment, if Figure 9 As shown, the field-level analysis unit 306 includes a process analysis subunit 3061 and a verification subunit 3062 .

[0119] The process analysis sub-unit 3061 is used to analyze the lineage and field processing process of the fields output in the lineage message to obtain the lineage analysis results and the analysis results of the field processing process; the verification sub-unit 3062 is used to use the lineage analysis results and the analysis results of the field processing process to verify whether they are correct. When the verification is correct, the lineage analysis results and the analysis results of the field processing process are stored in the cache to obtain the analysis results, wherein the analysis results include the cache data.

[0120] In one embodiment, the storage unit 307 is used to store the analysis results, data source lineage, modified data table lineage, and related information in a database.

[0121] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned data lineage analysis device 300 and each unit can refer to the corresponding description in the aforementioned method embodiment. For the convenience and brevity of the description, it will not be repeated here.

[0122] The data lineage analysis device 300 can be implemented as a computer program. The computer program can be used in Figure 10 Runs on the computer equipment shown.

[0123] See also Figure 10 , Figure 10 1 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device 50 may be a server, wherein the server may be an independent server or a server cluster composed of multiple servers.

[0124] See Figure 10 The computer device 500 includes a processor 502 , a memory, and a network interface 505 connected via a system bus 501 , wherein the memory may include a non-volatile storage medium 503 and an internal memory 504 .

[0125] The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions, which, when executed, can enable the processor 502 to perform a data lineage analysis method.

[0126] The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500.

[0127] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a data lineage analysis method.

[0128] The network interface 505 is used to communicate with other devices through the network. Figure 10 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device 500 to which the solution of the present application is applied. The specific computer device 500 may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0129] The processor 502 is configured to execute a computer program 5032 stored in the memory to implement the following steps:

[0130] Obtain the data source to be analyzed; configure relevant information of the data processing process according to the data source to be analyzed; create a job-level thread and a field-level thread; use the job-level thread in combination with the relevant information to analyze the lineage message to obtain the field lineage message, and store the field lineage message in the message queue; use the field-level thread to obtain the field lineage message from the message queue; analyze the field lineage and field processing process according to the field lineage message to obtain the analysis result; store the analysis result and the relevant information in the database.

[0131] The relevant information includes data source information, table information, information on whether table-level configuration has been changed, field-level data processing information, and field-level output field mapping information.

[0132] In one embodiment, the processor 502 implements the following steps when implementing the steps of creating a job-level thread and a field-level thread:

[0133] Use the main thread to start multiple job-level threads and field-level threads based on configuration.

[0134] In one embodiment, when implementing the steps of analyzing the lineage message using the job-level thread in combination with the relevant information to obtain field lineage message and storing the field lineage message in the message queue, the processor 502 specifically implements the following steps:

[0135] Filter the unchanged table-level configuration information according to the relevant information to obtain the modified table-level configuration information; assemble the modified table-level configuration information to obtain field lineage information; store the field lineage information in a message queue; analyze the data source lineage and the modified data table lineage; obtain the cached data stored by the field-level thread; poll to determine whether the task corresponding to the field-level thread is completed; if it is determined that the task corresponding to the field-level thread is completed, respond to the analysis result of the field-level thread; if it is determined that the task corresponding to the field-level thread is not completed, execute the polling to determine whether the task corresponding to the field-level thread is completed.

[0136] In one embodiment, when implementing the polling step to determine whether the task corresponding to the field-level thread is completed, the processor 502 specifically implements the following steps:

[0137] Determine whether the number of completed field-level tasks in the cache data is equal to the size of the message queue; if the number of completed field-level tasks in the cache data is equal to the size of the message queue, determine that the task corresponding to the field-level thread is completed; if the number of completed field-level tasks in the cache data is not equal to the size of the message queue, determine that the task corresponding to the field-level thread is not completed.

[0138] In one embodiment, when the processor 502 performs the step of analyzing the field lineage and the field processing process according to the field lineage information to obtain the analysis result, the processor 502 specifically performs the following steps:

[0139] The lineage and field processing process of the fields output in the lineage message are analyzed to obtain the lineage analysis results and the analysis results of the field processing process; the lineage analysis results and the analysis results of the field processing process are used to verify whether they are correct. When the verification is correct, the lineage analysis results and the analysis results of the field processing process are stored in the cache to obtain the analysis results, wherein the analysis results include the cache data.

[0140] In one embodiment, when implementing the step of storing the analysis results and the related information in the database, the processor 502 specifically implements the following steps:

[0141] The analysis results, data source lineage, modified data table lineage, and the related information are stored in a database.

[0142] It should be understood that in the embodiment of the present application, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0143] Those skilled in the art will appreciate that all or part of the steps in the method of the above-described embodiment can be implemented by instructing the relevant hardware through a computer program. The computer program includes program instructions, which can be stored in a storage medium that is computer-readable. The program instructions are executed by at least one processor in the computer system to implement the steps in the method of the above-described embodiment.

[0144] Therefore, the present invention also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program, wherein when the computer program is executed by a processor, the processor performs the following steps:

[0145] Obtain the data source to be analyzed; configure relevant information of the data processing process according to the data source to be analyzed; create a job-level thread and a field-level thread; use the job-level thread in combination with the relevant information to analyze the lineage message to obtain the field lineage message, and store the field lineage message in the message queue; use the field-level thread to obtain the field lineage message from the message queue; analyze the field lineage and field processing process according to the field lineage message to obtain the analysis result; store the analysis result and the relevant information in the database.

[0146] The relevant information includes data source information, table information, information on whether table-level configuration has been changed, field-level data processing information, and field-level output field mapping information.

[0147] In one embodiment, when the processor executes the computer program to implement the steps of creating a job-level thread and a field-level thread, the processor specifically implements the following steps:

[0148] Use the main thread to start multiple job-level threads and field-level threads based on configuration.

[0149] In one embodiment, when the processor executes the computer program to implement the steps of analyzing the lineage message using the job-level thread in combination with the relevant information to obtain field lineage message and storing the field lineage message in a message queue, the processor specifically implements the following steps:

[0150] Filtering unchanged table-level configuration information according to the relevant information to obtain modified table-level configuration information;

[0151] Assemble the modified table-level configuration information to obtain field lineage messages; store the field lineage messages in a message queue; analyze the data source lineage and the modified data table lineage; obtain the cache data stored by the field-level thread; poll to determine whether the task corresponding to the field-level thread is completed; if it is determined that the task corresponding to the field-level thread is completed, respond to the analysis result of the field-level thread; if it is determined that the task corresponding to the field-level thread is not completed, execute the polling to determine whether the task corresponding to the field-level thread is completed.

[0152] In one embodiment, when the processor executes the computer program to implement the step of polling to determine whether the task corresponding to the field-level thread is completed, the processor specifically implements the following steps:

[0153] Determine whether the number of completed field-level tasks in the cache data is equal to the size of the message queue; if the number of completed field-level tasks in the cache data is equal to the size of the message queue, determine that the task corresponding to the field-level thread is completed; if the number of completed field-level tasks in the cache data is not equal to the size of the message queue, determine that the task corresponding to the field-level thread is not completed.

[0154] In one embodiment, when the processor executes the computer program to implement the step of analyzing the field lineage and field processing according to the field lineage message to obtain an analysis result, the processor specifically implements the following steps:

[0155] The lineage and field processing process of the fields output in the lineage message are analyzed to obtain the lineage analysis results and the analysis results of the field processing process; the lineage analysis results and the analysis results of the field processing process are used to verify whether they are correct. When the verification is correct, the lineage analysis results and the analysis results of the field processing process are stored in the cache to obtain the analysis results, wherein the analysis results include the cache data.

[0156] In one embodiment, when the processor executes the computer program to implement the step of storing the analysis results and the related information in the database, the processor specifically implements the following steps:

[0157] The analysis results, data source lineage, modified data table lineage, and the related information are stored in a database.

[0158] The storage medium may be any computer-readable storage medium that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk.

[0159] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the composition and steps of each example according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0160] In the several embodiments provided herein, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the various units is merely a logical functional division, and actual implementation may employ other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be omitted or not implemented.

[0161] The steps in the methods of the embodiments of the present invention may be adjusted in order, combined, or deleted as needed. The units in the devices of the embodiments of the present invention may be combined, divided, or deleted as needed. Furthermore, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit.

[0162] If this integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (such as a personal computer, terminal, or network device) to execute all or part of the steps of the method described in various embodiments of the present invention.

[0163] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A data lineage analysis method, characterized in that: include: Obtain the data source to be analyzed; Configuring relevant information of the data processing process according to the data source to be analyzed; Create job-level threads and field-level threads; Utilizing the job-level thread in combination with the relevant information to analyze the lineage message to obtain field lineage message, and storing the field lineage message in a message queue; Obtaining the field lineage message from the message queue using the field-level thread; Analyzing the field lineage and field processing process according to the field lineage information to obtain an analysis result; Storing the analysis results and the relevant information in a database; The analyzing the lineage message by using the job-level thread in combination with the relevant information to obtain field lineage message, and storing the field lineage message in a message queue includes: Filtering unchanged table-level configuration information according to the relevant information to obtain modified table-level configuration information; Assemble the modified table-level configuration information to obtain field lineage information; Storing the field lineage information in the message queue; Analyze the lineage of data sources and modified data tables; Get the cache data stored by the field-level thread; Polling to determine whether the task corresponding to the field-level thread is completed; If it is determined that the task corresponding to the field-level thread is completed, responding to the analysis result of the field-level thread; If it is determined that the task corresponding to the field-level thread is not completed, performing the polling to determine whether the task corresponding to the field-level thread is completed; The relevant information includes data source information, table information, information on whether table-level configuration has been changed, field-level data processing information, and field-level output field mapping information; Field lineage message refers to the job-level thread traversing the output field set. For each field, the lineage and processing process information related to the current field is extracted from the field-level data processing information and the field-level output field mapping information, and the content is composed of the lineage and processing process information related to the current field.

2. The data lineage analysis method according to claim 1, characterized in that: The step of creating a job-level thread and a field-level thread includes: Use the main thread to start multiple job-level threads and field-level threads based on configuration.

3. The data lineage analysis method according to claim 1, characterized in that: The polling to determine whether the task corresponding to the field-level thread is completed includes: Determine whether the number of completed field-level tasks in the cached data is equal to the size of the message queue; If the number of completed field-level tasks in the cached data is equal to the size of the message queue, determining that the task corresponding to the field-level thread is completed; If the number of completed field-level tasks in the cache data is not equal to the size of the message queue, it is determined that the task corresponding to the field-level thread is not completed.

4. The data lineage analysis method according to claim 3, characterized in that: The analyzing of field lineage and field processing according to the field lineage message to obtain analysis results includes: Analyzing the lineage and field processing of the output fields in the lineage message to obtain lineage analysis results and field processing analysis results; The lineage analysis result and the analysis result of the field processing process are used to verify whether they are correct. When the verification is correct, the lineage analysis result and the analysis result of the field processing process are stored in the cache to obtain the analysis result, wherein the analysis result includes the cache data.

5. The data lineage analysis method according to claim 4, characterized in that: The storing of the analysis results and the related information in a database includes: The analysis results, data source lineage, modified data table lineage, and the related information are stored in a database.

6. A data lineage analysis device, characterized in that: include: A data source acquisition unit, used to acquire the data source to be analyzed; a configuration unit, configured to configure relevant information of a data processing process according to the data source to be analyzed; Creation unit, used to create job-level threads and field-level threads; a job-level analysis unit, configured to analyze the lineage message using the job-level thread in combination with the relevant information to obtain field lineage message, and store the field lineage message in a message queue; A message acquisition unit, configured to acquire the field lineage message from the message queue using the field-level thread; A field-level analysis unit, configured to analyze field lineage and field processing according to the field lineage information to obtain an analysis result; A storage unit, configured to store the analysis results and the related information in a database; The job-level analysis unit includes a filtering subunit, an assembling subunit, a storing subunit, a lineage analysis subunit, a cache data acquisition subunit, a task judgment subunit, and a response subunit; a filtering subunit, configured to filter the unchanged table-level configuration information according to the relevant information to obtain the modified table-level configuration information; The assembly subunit is used to assemble the modified table-level configuration information to obtain field lineage information; A storage subunit, configured to store the field lineage message into a message queue; The lineage analysis subunit is used to analyze the lineage of data sources and the lineage of modified data tables; The cache data acquisition subunit is used to obtain the cache data stored by the field-level thread; a task determination subunit, configured to poll and determine whether the task corresponding to the field-level thread is completed; if it is determined that the task corresponding to the field-level thread is not completed, executing the polling to determine whether the task corresponding to the field-level thread is completed; a response subunit, configured to respond to an analysis result of the field-level thread if it is determined that the task corresponding to the field-level thread is completed; The relevant information includes data source information, table information, information on whether table-level configuration has been changed, field-level data processing information, and field-level output field mapping information; Field lineage message refers to the job-level thread traversing the output field set. For each field, the lineage and processing process information related to the current field is extracted from the field-level data processing information and the field-level output field mapping information, and the content is composed of the lineage and processing process information related to the current field.

7. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 5 when executing the computer program.

8. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images