Blood relationship analysis method and device and storage medium

Through the combination of dynamic script collector and data blood relationship analyzer, blood relationship analysis of multi-source heterogeneous data is realized, which solves the problem that multi-source data cannot be analyzed in the existing technology, improves the timeliness and accuracy of the analysis, and adapts to The characteristics of different data sources cover a wide range of data sources.

CN118916416AInactive Publication Date: 2024-11-08SHENZHEN SMARTCITY TECH DEV GRP CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202411420666.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-12
Publication Date
2024-11-08
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The existing blood relationship analysis methods cannot effectively analyze blood relationships on multi-source heterogeneous data, rely too much on tools for a single data source, and lack comprehensive management of the entire life cycle of the data, so they cannot reflect the latest status of the data in real time.

Method used

Use a dynamic script collector to collect the pending data from the data source and perform asynchronous processing through the message queue. The data blood relationship analyzer dynamically determines the processing class according to the data source type, performs lexical, syntax, and semantic analysis, generates blood relationship analysis results, and transfers the results. Write to the graph database.

Benefits of technology

Achieve unified collection and flexible analysis of multi-source heterogeneous data, improve the timeliness and accuracy of blood relationship analysis, better understand the flow and transformation of business data, cover a wide range of data sources, and avoid the limitations of fixed tools. sex.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118916416A_ABST
    Figure CN118916416A_ABST
Patent Text Reader

Abstract

The invention discloses a blood relationship analysis method and device and a storage medium, and relates to the technical field of big data governance, and the method comprises the steps that a dynamic script collector collects to-be-processed data from a data source, and writes the to-be-processed data into a message queue, and the to-be-processed data comprises script information and an execution mode; the data consanguinity analyzer reads the to-be-processed data from the message queue, and determines an applicable data consanguinity analyzer processing class according to the data source type of the to-be-processed data; and the data consanguinity analyzer processing class performs consanguinity analysis on the to-be-processed data to obtain a corresponding consanguinity analysis result. The method aims at conducting blood relationship analysis on multi-source heterogeneous data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of big data governance technology, and in particular to bloodline analysis methods, equipment and storage media. Background Art

[0002] Data lineage refers to the natural relationship formed between data throughout its entire life cycle, from its generation, processing, processing, fusion, circulation to its final extinction.

[0003] In the related art, the log files of the database record the operation history, including data insertion, update and deletion operations, etc. These log files can be parsed using database log analysis tools or other professional tools provided by data source manufacturers to obtain data lineage relationships.

[0004] Current lineage analysis methods are too dependent on tools provided by specific data source vendors, which limits their application scope to the analysis of a single data source. However, in the data governance process, the life cycle of business data often involves multi-source heterogeneous data sources, especially in log files, where there are significant differences between different data sources. Therefore, current methods cannot perform lineage analysis on multi-source heterogeneous data.

[0005] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0006] The main purpose of this application is to provide a bloodline analysis method, device and storage medium, aiming to solve the technical problem that bloodline analysis cannot be performed on multi-source heterogeneous data.

[0007] To achieve the above objectives, the present application proposes a blood relationship analysis method, which comprises: The dynamic script collector collects data to be processed from a data source and writes the data to be processed into a message queue, wherein the data to be processed includes script information and execution mode; The data lineage analyzer reads the data to be processed from the message queue, and determines an applicable data lineage analyzer processing class according to the data source type of the data to be processed; The data lineage analyzer processing class performs lineage analysis on the data to be processed to obtain corresponding lineage analysis results.

[0008] In one embodiment, the step of the dynamic script collector collecting the data to be processed from the data source includes: The dynamic script collector intercepts raw data from the business system based on the interception model; The dynamic script collector collects the data to be processed in the original data based on the information collection model.

[0009] In one embodiment, the data lineage analyzer includes a data lineage analyzer registration class, and the step of determining an applicable data lineage analyzer processing class according to the data source type of the data to be processed includes: Determine an enumeration value of the data source type of the data to be processed; The data lineage analyzer processing class corresponding to the enumeration value is determined according to the data lineage analyzer registration class.

[0010] In one embodiment, before the step of determining the data lineage analyzer processing class corresponding to the enumeration value according to the data lineage analyzer registration class, the method further includes: Determine a thread-safe hash table associated with the data lineage analyzer registration class according to the global access point of the data lineage analyzer registration class; In the thread-safe hash table, obtain the data lineage analyzer registration class.

[0011] In one embodiment, the step of determining the data lineage analyzer processing class corresponding to the enumeration value according to the data lineage analyzer registration class includes: Based on the public interface of the data lineage analyzer registration class, determine the parent class data lineage analysis method corresponding to the enumeration value; Determine the subclass data lineage analysis method corresponding to the enumeration value based on the non-public interface of the data lineage analyzer registration class; The parent class data lineage analysis method and the child class data lineage analysis method are summarized to obtain the data lineage analyzer processing class.

[0012] In one embodiment, the data lineage analyzer processing class performs lineage analysis on the data to be processed, and the step of obtaining the corresponding lineage analysis result includes: The data lineage analysis method in the data lineage analyzer processing class is called to analyze the structure and semantics of the data to be processed to obtain the corresponding lineage analysis result.

[0013] In one embodiment, the step of calling the data lineage analysis method in the data lineage analyzer processing class to analyze the structure and semantics of the data to be processed to obtain the corresponding lineage analysis result includes: Performing lexical analysis, grammatical analysis, semantic analysis and data flow analysis on the data to be processed to determine the data flow direction of the data to be processed; The bloodline analysis result is generated according to the data dependency relationship and the data flow path of the data flow.

[0014] In one embodiment, after the step of performing lineage analysis on the data to be processed to obtain the corresponding lineage analysis result, the data lineage analyzer processing class further includes: The blood relationship analysis results are written into a graph database.

[0015] In addition, to achieve the above-mentioned purpose, the present application also proposes a blood relationship analysis device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the blood relationship analysis method as described above.

[0016] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the blood relationship analysis method described above are implemented.

[0017] One or more technical solutions proposed in this application have at least the following technical effects: Data is collected from data sources through a dynamic script collector. This dynamic script collector can process data of different formats and structures, ensuring that data can be collected uniformly in a multi-source environment. This method no longer relies on tools for a single data source, ensuring the flexibility and scalability of data collection, and covering a wide range of data sources. Moreover, the data lineage analyzer dynamically determines the corresponding data lineage analyzer processing class based on the data source of the data to be processed. This mechanism allows for flexible analysis of the characteristics of different data sources, avoiding the limitations of fixed tools. Throughout the entire process of this method, data from different data sources can be integrated and comprehensive lineage analysis can be performed. This comprehensive capability enables a better understanding of the flow and transformation of business data during the data governance process. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0020] Figure 1 A schematic diagram of the process provided for Example 1 of the blood relationship analysis method of the present application; Figure 2 A schematic diagram of the process provided for Example 2 of the blood relationship analysis method of this application; Figure 3 A schematic diagram of the process provided for Example 3 of the blood relationship analysis method of this application; Figure 4 A schematic diagram of the process provided for Example 4 of the blood relationship analysis method of the present application; Figure 5 A schematic diagram of the process provided for Example 5 of the blood relationship analysis method of this application; Figure 6 This is a schematic diagram of the overall brief process of the blood relationship analysis method of this application; Figure 7 A schematic diagram of a brief process of bloodline analysis of data in the bloodline analysis method of this application; Figure 8 This is a schematic diagram of the device structure of the hardware operating environment involved in the blood relationship analysis method in the embodiment of the present application.

[0021] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0022] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0023] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0024] The main solution of the embodiment of the present application is: a dynamic script collector collects the data to be processed from the data source, and writes the data to be processed into a message queue, wherein the data to be processed includes script information and execution method; a data lineage analyzer reads the data to be processed from the message queue, and determines the applicable data lineage analyzer processing class according to the data source type of the data to be processed; the data lineage analyzer processing class performs lineage analysis on the data to be processed to obtain corresponding lineage analysis results.

[0025] Current data lineage analysis methods: (1) Analyze data lineage based on metadata information: The database management system of the data source usually opens corresponding interfaces or methods, allowing users to obtain detailed metadata information through data dictionaries or special metadata collection tools. By deeply analyzing this metadata information, the lineage relationship between data can be identified more accurately.

[0026] (2) Obtaining data lineage based on database log files: Database log files record the operation history, including data insertion, update, and deletion operations. Using database log analysis tools or other professional tools provided by data source vendors, these log files can be parsed to obtain data lineage.

[0027] The current data lineage analysis methods have the following defects: (1) Only supports a single data source: The current lineage analysis method is too dependent on the data definition, description and corresponding tool support provided by the data source vendor, which limits its application scope to the analysis of a single data source. However, in the data governance process, the life cycle of business data often involves multi-source heterogeneous data sources. There are significant differences between different data sources in terms of metadata definition and acquisition of log files. Faced with this situation, the current lineage analysis method lacks compatibility and therefore cannot perform lineage analysis on multi-source heterogeneous data.

[0028] (2) Data lineage analysis lags: Analyzing data lineage relationships based on metadata information mainly relies on the scheduled metadata collection method to generate a lineage relationship map. However, this method has a significant lag and cannot reflect the latest status of the data in real time.

[0029] (3) Lack of management of the entire data life cycle: The current data governance process lacks comprehensive management of the entire data life cycle, and the construction of the data lineage relationship map is not perfect. For example, insufficient information collection on processes such as data generation and conversion leads to a lack of information on these processes in the data lineage relationship map.

[0030] It should be noted that the execution subject of this solution can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a blood relationship analysis device capable of realizing the above functions, etc. The following embodiments are described below taking the blood relationship analysis device as an example.

[0031] Based on this, the present application embodiment provides a blood relationship analysis method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the blood relationship analysis method of the present application.

[0032] In this embodiment, the blood relationship analysis method includes steps S10 to S30: Step S10, the dynamic script collector collects data to be processed from the data source and writes the data to be processed into a message queue, wherein the data to be processed includes script information and execution mode; It should be noted that the dynamic script collector is a component used for data collection, which can interact with various data sources and collect the required data.

[0033] This application supports not only lineage analysis of a single data source, but also lineage analysis of multi-source heterogeneous data sources. Among them, multi-source heterogeneous data sources, multi-source means that the data comes from multiple different sources, and heterogeneous means that these data have differences in structure, format, storage method, etc. It can be divided into the following types: Relational databases: These databases use SQL (Structured Query Language) for data management. Data is organized in tables, and tables can be connected to each other through relationships. For example, MySQL.

[0034] Non-relational databases: These databases do not rely on traditional table structures to store data, but use different data structures such as key-value pairs, documents, column storage, or graph models. For example, Spark SQL.

[0035] File system: refers to the data stored in files, which can be in various formats, such as CSV, Excel files, etc.

[0036] It should be noted that the data to be processed refers to the data that needs to be analyzed for lineage, including script information and execution methods. The script information includes data source type, data source connection information, script content, etc.; the execution methods include manual execution, execution through data processing scheduling tasks, etc., and their corresponding identifiers.

[0037] In one possible implementation, based on the adapter pattern: An adapter is designed for each different data source. The adapter is responsible for converting the data source's unique interface into a unified interface so that the dynamic script collector can interact with it indiscriminately. The dynamic script collector executes the collection method for a specific data source by calling the adapter's unified interface. After the adapter obtains data from the data source, it encapsulates the obtained data and returns it to the dynamic script collector through a unified interface.

[0038] In another feasible implementation, based on the interception model and the information collection model: It should be noted that the interception model uses the interceptor mechanism to intercept the original data. The specific process is as follows: Identify interception points. Determine which links in the business system are key points for data generation, such as specific method calls at the database access layer, service layer, or control layer.

[0039] Interceptor writing. Using the AOP framework, the interception logic is inserted before and after the key points. The interceptor can capture the information before and after the method call, such as the method name, content, execution time, execution result, etc.

[0040] Configure interceptors. Register interceptors in the business system's configuration file and specify their application scope, such as all database access operations, calls to specific service methods, etc.

[0041] When the business system is running, once a request or method call triggers the configured interception point, the interceptor will automatically execute its defined logic and capture and process data before and after the key point. In this way, the interceptor can intercept the original data from the business system.

[0042] It should be noted that the information collection model is used to define the script information and execution methods that support the execution of different data sources. The specific process is as follows: The raw data may contain various forms of text, database records, log files, etc., and the information collection model is responsible for identifying, parsing and extracting the script information and execution methods therein.

[0043] A set of parsing rules is defined in the information collection model to identify and extract script information and its execution methods from the original data. These rules may include regular expression matching, XML / JSON parsing, text processing in a specific format, etc.

[0044] Use the defined parsing rules to parse the original data and extract the script information and execution method.

[0045] Exemplarily, the parsing engine is designed as multiple modules, each module is responsible for processing a type of raw data, such as SQL, XML, etc. In this way, the parsing engine receives the raw data and selects the corresponding parsing module according to the data type. The parsing module traverses the raw data and applies the configured parsing rules to identify and extract the script information and execution mode.

[0046] It can be understood that through the interception model, the key points of data generation at multiple levels in the business system can be captured without being limited to a specific data source format or type. This enables flexible processing of data from different data sources without the need to write specific adapters for each data source. This flexibility enables faster adaptation to new data sources.

[0047] Based on this, the latter implementation method can achieve full-process lifecycle management. By building a unified data lineage information collection data model, the collected data can be deeply analyzed to analyze its source, processing process, destination and mutual relationship, so as to clearly depict the full life cycle lineage relationship of the data for the enterprise. This makes the origin and development of the data clear at a glance, helping enterprises to better understand and use the data.

[0048] Furthermore, the data to be processed is sent to a message queue. As a data transmission middleware, the message queue can play a role of buffering and decoupling. It allows dynamic script collectors and data lineage analyzers to work in an asynchronous manner, that is, the dynamic script collector can continuously write data to the queue, and the data lineage analyzer can read data from the queue for processing according to the processing capacity. Different from the timed scheduling collection method in the current technology, the present application uses asynchronous message middleware technology to realize quasi-real-time data lineage dependency analysis. This method greatly improves the timeliness and accuracy of data lineage analysis, making data analysis closer to business reality and more effectively supporting business decisions and operations.

[0049] Step S20, the data lineage analyzer reads the data to be processed from the message queue, and determines an applicable data lineage analyzer processing class according to the data source type of the data to be processed; Optionally, this solution may adopt a pluggable SDK (Software Development Kit) to support data lineage analyzers from multiple data sources.

[0050] It should be noted that a data lineage analyzer is a tool or system used to track and analyze the flow and changes of data in its life cycle, including the source of the data, the processing process, and the final destination. The data lineage analyzer includes several data lineage analyzer processing classes, each of which contains data processing methods for a specific data source.

[0051] Based on the implementation of this data lineage analyzer SDK, users only need to implement a lineage analyzer class for a specific data source to complete the development of the corresponding data source lineage analysis SDK. In the data source analyzer implementation class, you only need to conduct in-depth analysis based on the syntax and semantic definitions of the data source to quickly develop a new data source lineage analysis SDK.

[0052] Optionally, this solution can adopt a combination of strategy mode and factory mode to perform data source expansion configuration according to the data source type, so as to isolate and simplify different data sources to obtain different parsing methods.

[0053] In actual applications, the data lineage analyzer will set up one or more listeners to continuously monitor changes in the message queue. When new data to be processed is sent to the message queue, the listener will trigger the corresponding read operation, and the data lineage analyzer can read the message and quickly perform data analysis and processing, thereby obtaining the data to be processed in real time. This can achieve quasi-real-time data lineage analysis effects and improve data processing efficiency and analysis accuracy.

[0054] Then, the data lineage analyzer will parse the data to be processed. In the process of parsing the data, the data lineage analyzer will check the metadata of the data, including header information, tags, specific fields, etc. Since the metadata contains important information about the data source, the data lineage analyzer can identify the data source of the data to be processed by checking these metadata.

[0055] For example, the data format, storage location, data model, etc. in the metadata are analyzed, and then these key information are compared with the predefined classification standard, which contains the key information of all supported data source types. By matching the features, the data source type corresponding to the data to be processed is identified.

[0056] The data lineage analyzer then looks in its internal or external configuration for a processing class that matches the identified data source type.

[0057] Step S30: the data lineage analyzer processing class performs lineage analysis on the data to be processed to obtain corresponding lineage analysis results.

[0058] In a feasible implementation, the blood relationship analysis based on machine learning includes the following specific steps: According to the characteristics of the data and the needs of blood relationship analysis, select appropriate machine learning algorithms, such as neural networks, relationship classifiers, etc.; use the data set with known blood relationship as the training set to train the machine learning model; evaluate the performance of the model through methods such as cross-validation to ensure that the model has high accuracy and reliability. In this way, in the actual application process, the data to be processed can be input into the trained model to predict blood relationship.

[0059] It should be noted that the results of lineage analysis can cover a variety of relationships, including cluster lineage, system lineage, table-level lineage, and field lineage, which can clearly show the logical context of data processing. In this way, by quickly locating the impact range of abnormal data fields, the minimum data traceback range can be accurately determined, thereby reducing the cost of understanding data and solving data problems.

[0060] In addition, the traced blood relationship and related information can be stored in an appropriate storage medium, such as a graph database, a relational database, or a file system. For example, the Neo4j graph database is used as the storage medium, which has high performance and can store and query a large amount of node and edge information, and has efficient query and execution capabilities. This provides the data governance system with a full life cycle data blood relationship management function.

[0061] In addition, it can also provide API query interfaces for multi-dimensional data lineage view to meet customers' diverse application needs in data governance, quality monitoring and traceability. These API interfaces cover the multi-level relationships between the entire cluster, databases and tables, tables and fields, and can clearly display the dependencies and flow paths between data, helping customers to fully understand the life cycle and processing of data. This multi-dimensional data lineage view can meet customers' diverse needs in data governance, data quality monitoring, data traceability, etc., greatly improving the efficiency of data management and decision-making.

[0062] This embodiment provides a lineage analysis method, which collects data from a data source through a dynamic script collector. This dynamic script collector can process data of different formats and structures, ensuring that data can be collected uniformly in a multi-source environment. This method no longer relies on tools for a single data source, ensuring the flexibility and scalability of data collection, and can cover a wide range of data sources. Moreover, the data lineage analyzer dynamically determines the corresponding data lineage analyzer processing class based on the data source of the data to be processed. This mechanism allows for flexible analysis of the characteristics of different data sources, avoiding the limitations of fixed tools. Throughout the entire process of this method, data from different data sources can be integrated and a comprehensive lineage analysis can be performed. This comprehensive capability enables a better understanding of the flow and transformation of business data during the data governance process.

[0063] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can refer to the above introduction, and will not be repeated later. Figure 2 The data lineage analyzer includes a data lineage analyzer registration class. The step of determining the applicable data lineage analyzer processing class according to the data source type of the data to be processed includes steps A10 to A20: Step A10, determining the enumeration value of the data source type of the data to be processed; Step A20, determining the data lineage analyzer processing class corresponding to the enumeration value according to the data lineage analyzer registration class.

[0064] It should be noted that a data lineage analyzer registration class is created in advance. This registration class is a unified entry for implementing data lineage analyzer implementation classes that obtain heterogeneous data sources. The registration class is used to manage and maintain the mapping relationship between the data lineage analyzer processing class and the data source enumeration value. It usually contains a mapping table or mapping logic to map the data source enumeration value to the corresponding data lineage analyzer processing class.

[0065] In actual applications, the metadata in the data to be processed is parsed to extract the enumeration value of the data source. Then, the data lineage analyzer registration class is used to find the data lineage analyzer processing class corresponding to the enumeration value. This search process may be implemented through simple key-value pair queries, database queries, or more complex logic.

[0066] After determining the enumeration values ​​of different data source types, the corresponding data lineage analyzer processing class can be dynamically obtained by using the class reflection mechanism according to the data source type input parameter.

[0067] This embodiment provides a lineage analysis method, where each enumeration value corresponds to a specific data source. By extracting the enumeration value defined in the metadata, the data source can be quickly identified without going through a tedious analysis and matching process, thereby improving the processing speed. Moreover, when a new data source needs to be added, it is only necessary to add a new value to the enumeration and implement or register a new processing class accordingly. This method enables the system to flexibly adapt to changes.

[0068] Based on the second embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the above-mentioned second embodiment can refer to the above introduction, and will not be repeated in the following. Figure 3 Before step A20, the blood relationship analysis method further includes steps B10 to B20: Step B10, determining a thread-safe hash table associated with the data lineage analyzer registration class according to the global access point of the data lineage analyzer registration class; Step B20: Obtain the data lineage analyzer registration class in the thread-safe hash table.

[0069] In this solution, the registration class adopts a singleton design method, which only allows one data lineage analysis class object instance to be created, thereby providing a global access point for subsequent process analysis. This design reduces the overhead of frequent creation and destruction of objects and improves the overall performance of the system.

[0070] In actual applications, a certain form of reference is obtained through the global access point of the data lineage analyzer registration class, which can point to or represent an instance of the data lineage analyzer registration class. Based on this reference, the system can determine the thread-safe hash table ConcurrentHashMap associated with the data lineage analyzer registration class. This hash table may be used to store various information related to data lineage analysis, including but not limited to registered analysis methods, configuration parameters, status information, etc.

[0071] Once the thread-safe hash table is determined, the system will search and obtain the instance of the data lineage analyzer registration class in the hash table. Since the hash table is thread-safe, this acquisition process can ensure data consistency and access security in a multi-threaded environment.

[0072] This embodiment provides a lineage analysis method, which introduces a thread-safe memory cache mechanism ConcurrentHashMap in the data lineage analyzer registration class. When the system is started, the instance of the data lineage analyzer registration class is initialized in an automated manner and saved in the ConcurrentHashMap object, ensuring that the data lineage analyzer is quickly obtained in the processing flow, thereby improving the performance of the system.

[0073] Based on the second embodiment of the present application, in the fourth embodiment of the present application, the same or similar contents as those in the above-mentioned second embodiment can refer to the above introduction, and will not be repeated in the following. Figure 4 , step A20, including steps C10 to C30: Step C10, determining the parent class data lineage analysis method corresponding to the enumeration value based on the public interface of the data lineage analyzer registration class; Step C20, determining the subclass data lineage analysis method corresponding to the enumeration value based on the non-public interface of the data lineage analyzer registration class; Step C30, summarizing the parent class data lineage analysis method and the child class data lineage analysis method to obtain the data lineage analyzer processing class.

[0074] It should be noted that the public interface is associated with the parent class data lineage analysis method. These parent class methods generally adopt more general analysis logic and can meet basic analysis needs; the non-public interface is associated with the child class data lineage analysis method. These child class methods are usually expanded or concretized based on the parent class and can handle more specific analysis scenarios.

[0075] On the one hand, use the public interface provided by the data lineage analyzer registration class to pass in a specific enumeration value. The public interface will return the parent class data lineage analysis method corresponding to the enumeration value.

[0076] On the other hand, the non-public interface of the data lineage analyzer registration class is used to find the subclass data lineage analysis method corresponding to the enumeration value. This non-public interface may require specific permissions or access through inheritance. Subsequently, the non-public interface will find and return all subclass custom methods applicable to the specific enumeration value, which may be extensions or specializations of the parent class method.

[0077] At this point, all methods obtained from the parent class and child class are summarized and integrated to form a complete method set, and then a data lineage analyzer processing class is created, which will be responsible for executing the corresponding analysis logic and processing data lineage relationships.

[0078] This embodiment provides a lineage analysis method, which adopts the strategy pattern design method to encapsulate different data lineage analysis methods in a dedicated class, making the code structure clearer and easier to manage and maintain. In this way, different data source types can be replaced with each other, and users can easily choose the appropriate method according to the specific lineage analysis scenario. This flexibility enables the system to quickly adapt to different changes in requirements. Moreover, when it is necessary to add a new data source type or a new data lineage analysis method, it is only necessary to create a new strategy class and register it without making large-scale modifications to the existing processing class or code. This solution greatly improves the flexibility and scalability of the system.

[0079] Based on the first embodiment of the present application, in the fifth embodiment of the present application, the same or similar contents as those in the first embodiment can be referred to the above description, and will not be described in detail later. Figure 5 , step S30, including step D10: Step D10, calling the data lineage analysis method in the data lineage analyzer processing class to analyze the structure and semantics of the data to be processed to obtain the corresponding lineage analysis result.

[0080] Determine all data lineage analysis methods in the data lineage analyzer processing class. If there are parent and child class data lineage analysis methods, you need to determine the calling order of all methods based on the strategy or configuration to ensure that the dependencies between them are correctly handled.

[0081] Exemplarily, the context information and the data to be processed are used as parameters to call the parent class data lineage analysis method, wait for the method to be executed, pass the output of the parent class method as input to the child class method, or configure the child class method to perform supplementary analysis based on the parent class method.

[0082] Specifically, within the data lineage analysis method, the following operations are performed on the data to be processed: Structural analysis: traverse the table structure of the data to be processed, identify basic information such as table name, field name, data type, etc.; record the relationship between fields, such as foreign key constraints, primary key-foreign key relationships, etc.; analyze the flow path of data between different tables, including data insertion, update, deletion and other operations; identify key nodes and branches in the data flow, and record the source and destination of the data.

[0083] Semantic analysis: Review relevant documents, communicate with business teams, or analyze the data itself to understand the business meaning and purpose of the data; determine the role and importance of data in the business process; track changes in the meaning of data in different processing stages or business scenarios; record the logic and results of operations such as data conversion, aggregation, and calculation.

[0084] When executing each data lineage analysis method, the intermediate results or metadata generated are collected and all intermediate results are merged and sorted to form the final lineage analysis results.

[0085] Exemplarily, data lineage analysis is implemented according to the following process: (1) Lexical analysis Goal: Break sentences into words and symbols.

[0086] Process: Read the SQL statements in the data to be processed; use the lexical analysis tool to split the SQL statements into words and symbols; generate a list of lexical units (Tokens).

[0087] (2) Syntax analysis Goal: Generate an Abstract Syntax Tree (AST).

[0088] Process: Read the lexical unit list; use the syntax analysis tool to parse the lexical units into a tree structure according to the SQL syntax rules; confirm the SQL statement structure corresponding to each node and generate AST.

[0089] (3) Semantic analysis Objective: Ensure that SQL statements are semantically correct.

[0090] Process: Read the AST structure; perform type checking to verify whether the data type matches; check whether the function call is legal, such as checking whether the function parameter type is correct.

[0091] (4) Data flow analysis Objective: Understand the data flow and dependencies in SQL statements.

[0092] Process: Traverse the AST and identify clauses such as SELECT, FROM, WHERE, JOIN, and UNION; record the data dependencies in each clause, including which fields come from which tables; analyze the implementation details of operations such as join conditions and sorting, and sort out the data flow path.

[0093] (5) Generate blood relationship analysis results Objective: Generate data lineage relationships based on data flow analysis results.

[0094] Process: Using the analysis results from the previous step, integrate the dependencies and data flow paths to form a lineage analysis result.

[0095] It is understandable that through a series of steps such as lexical analysis, syntax analysis, semantic analysis and data flow analysis, all levels of SQL statements can be fully covered, starting from the most basic words and symbols, and gradually deepening into the structure, meaning and data flow relationship of the statement. This comprehensive analysis method can ensure the accuracy and completeness of data lineage analysis.

[0096] This embodiment provides a lineage analysis method, through which the structure and semantics of the data are deeply analyzed. This dual analysis scheme can more comprehensively understand the data to be processed. This scheme fully considers the differences between different data sources and adopts a variety of data lineage analysis methods to deal with these differences. For example, for the syntax and functions unique to certain data sources, special data lineage analysis methods can be used to identify and process these differences, thereby ensuring the accuracy and completeness of the data lineage analysis. This differentiated processing method enables the method to flexibly adapt to different data sources and query environments.

[0097] For example, to help understand the implementation process of the blood relationship analysis method obtained by combining this embodiment with the above-mentioned embodiment 1, please refer to Figure 6 , Figure 6 A brief schematic diagram of a blood relationship analysis method is provided, specifically: In data governance processes such as data modeling, online data development, and data processing operations, multi-source heterogeneous data is usually involved. For these multi-source heterogeneous data, data lineage analysis is performed through the following process.

[0098] (1) Data lineage script information collection Define a unified dynamic script collector, including defining a unified interception model and a unified information collection model.

[0099] The unified interception model uses the interceptor mechanism to intercept raw data from the business system; the unified information collection model collects the data to be processed from the raw data.

[0100] (2) Message Middleware The dynamic script collector writes the data to be processed into the message queue.

[0101] As a message middleware, the message queue can send messages to the data lineage analyzer in real time for the data to be processed written by the dynamic script collector.

[0102] (3) Data lineage analysis The data lineage analyzer analyzes the data to be processed and outputs the data lineage analysis results.

[0103] (4) Data lineage application Based on the results of data lineage analysis, a data lineage relationship map is formed.

[0104] The specific process of data lineage analysis is as follows: Figure 7 shown.

[0105] (1) Constructing the data lineage analyzer registration class The registration class adopts a singleton pattern design approach, which only allows one instance of a data lineage analysis class object to be created, providing a global access point to the instance for subsequent process analysis.

[0106] (2) Determine the data lineage analysis method corresponding to the data source type In the data lineage analyzer registration class, the required data lineage analysis method is dynamically determined based on the data source type input parameter using the class reflection mechanism.

[0107] (3) Determine the data lineage analyzer processing class corresponding to the data source type The data lineage analyzer processing class adopts the strategy pattern design method, obtains the specific data lineage analysis method from the strategy class, and then summarizes it to obtain the data lineage analyzer processing class.

[0108] (4) Data lineage analysis Call the data lineage analysis method in the data lineage analyzer processing class to analyze the data to be processed.

[0109] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the blood relationship analysis method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0110] The present application provides a bloodline analysis device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the bloodline analysis method in the above-mentioned embodiment one.

[0111] Reference below Figure 8, which shows a schematic diagram of the structure of a blood relationship analysis device suitable for implementing the embodiment of the present application. The blood relationship analysis device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 8 The bloodline analysis device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0112] like Figure 8 As shown, the blood relationship analysis device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM) 1004. In RAM 1004, various programs and data required for the operation of the blood relationship analysis device are also stored. The processing device 1001, ROM 1002, and RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the bloodline analysis device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a bloodline analysis device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0113] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0114] The blood relationship analysis device provided by the present application adopts the blood relationship analysis method in the above embodiment, which can solve the technical problem of being unable to perform blood relationship analysis on multi-source heterogeneous data. Compared with the prior art, the beneficial effects of the blood relationship analysis device provided by the present application are the same as the beneficial effects of the blood relationship analysis method provided by the above embodiment, and the other technical features of the blood relationship analysis device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0115] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0116] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0117] The present application provides a computer-readable storage medium having a computer program stored thereon, and the computer-readable program instructions are used to execute the blood relationship analysis method in the above-mentioned embodiment.

[0118] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0119] The above-mentioned computer-readable storage medium may be included in the bloodline analysis device; or it may exist independently without being assembled into the bloodline analysis device.

[0120] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the lineage analysis device, the lineage analysis device enables: the dynamic script collector to collect the data to be processed from the data source, and write the data to be processed into the message queue, wherein the data to be processed includes script information and execution method; the data lineage analyzer reads the data to be processed from the message queue, and determines the applicable data lineage analyzer processing class according to the data source type of the data to be processed; the data lineage analyzer processing class performs lineage analysis on the data to be processed to obtain corresponding lineage analysis results.

[0121] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0122] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0123] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0124] The readable storage medium provided in the present application is a computer-readable storage medium, which stores a computer program for executing the above-mentioned blood relationship analysis method, and can solve the technical problem that blood relationship analysis cannot be performed on multi-source heterogeneous data. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the blood relationship analysis method provided in the above-mentioned embodiment, and will not be repeated here.

[0125] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A method for blood relationship analysis, characterized in that: The method comprises: The dynamic script collector collects data to be processed from a data source and writes the data to be processed into a message queue, wherein the data to be processed includes script information and execution mode; The data lineage analyzer reads the data to be processed from the message queue, and determines an applicable data lineage analyzer processing class according to the data source type of the data to be processed; The data lineage analyzer processing class performs lineage analysis on the data to be processed to obtain corresponding lineage analysis results.

2. The method according to claim 1, characterized in that The step of the dynamic script collector collecting the data to be processed from the data source comprises: The dynamic script collector intercepts raw data from the business system based on the interception model; The dynamic script collector collects the data to be processed in the original data based on the information collection model.

3. The method according to claim 1, characterized in that The data lineage analyzer includes a data lineage analyzer registration class. The step of determining an applicable data lineage analyzer processing class according to the data source type of the data to be processed includes: Determine an enumeration value of the data source type of the data to be processed; The data lineage analyzer processing class corresponding to the enumeration value is determined according to the data lineage analyzer registration class.

4. The method according to claim 3, characterized in that Before the step of determining the data lineage analyzer processing class corresponding to the enumeration value according to the data lineage analyzer registration class, the step further includes: Determine a thread-safe hash table associated with the data lineage analyzer registration class according to the global access point of the data lineage analyzer registration class; In the thread-safe hash table, obtain the data lineage analyzer registration class.

5. The method according to claim 3, characterized in that The step of determining the data lineage analyzer processing class corresponding to the enumeration value according to the data lineage analyzer registration class comprises: Based on the public interface of the data lineage analyzer registration class, determine the parent class data lineage analysis method corresponding to the enumeration value; Determine the subclass data lineage analysis method corresponding to the enumeration value based on the non-public interface of the data lineage analyzer registration class; The parent class data lineage analysis method and the child class data lineage analysis method are summarized to obtain the data lineage analyzer processing class.

6. The method according to claim 1, characterized in that The data lineage analyzer processing class performs lineage analysis on the data to be processed, and the steps of obtaining corresponding lineage analysis results include: The data lineage analysis method in the data lineage analyzer processing class is called to analyze the structure and semantics of the data to be processed to obtain the corresponding lineage analysis result.

7. The method according to claim 6, characterized in that The step of calling the data lineage analysis method in the data lineage analyzer processing class to analyze the structure and semantics of the data to be processed to obtain the corresponding lineage analysis result includes: Performing lexical analysis, grammatical analysis, semantic analysis and data flow analysis on the data to be processed to determine the data flow direction of the data to be processed; The bloodline analysis result is generated according to the data dependency relationship and the data flow path of the data flow.

8. The method according to claim 1, characterized in that After the data lineage analyzer processing class performs lineage analysis on the data to be processed to obtain the corresponding lineage analysis result, the method further includes: The blood relationship analysis results are written into a graph database.

9. A blood relationship analysis device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the blood relationship analysis method according to any one of claims 1 to 8.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the blood relationship analysis method as described in any one of claims 1 to 8 are implemented.

Citation Information

Patent Citations

  • Railway wagon material management and accounting data management method, device and equipment

    CN116483897A

  • Data blood relationship management method and system

    CN117033410A

  • Metadata collection method and device and storage medium

    CN118626550A