Data lineage analysis methods, devices, equipment and storage media

Abstract

This invention relates to data analysis technology and discloses a data lineage analysis method, comprising: acquiring raw data from real-time and offline data streams; identifying the data type of the raw data and selecting a preset parsing model based on the data type; parsing the raw data using the preset parsing model to obtain the data nodes, task nodes, and data flow relationships between each data node and task node; and constructing a lineage graph based on the data nodes, task nodes, and data flow relationships. Furthermore, this invention also relates to blockchain technology, allowing test reports to be stored on blockchain nodes. This invention also proposes a data lineage analysis method device, electronic device, and storage medium. This invention can improve the accuracy of lineage relationships for electronic transaction data.

Description

Technical Field

[0001] This invention relates to the field of data analysis technology, and in particular to a data lineage analysis method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] With the development of the digital age, massive amounts of electronic transaction data are constantly being generated (such as electronic transaction data generated by banking systems, insurance systems, etc.). How to utilize this electronic transaction data has become crucial for every enterprise. Currently, in the field of big data governance, data lineage analysis has been proposed. By collecting and analyzing electronic transaction data, the connections between electronic transaction data can be sorted out, thereby forming the desired data map and data assets. This enables traceability of data fusion processing and plays a very important role in analyzing the impact of electronic transaction data changes, troubleshooting business problems, reducing maintenance costs, and improving development efficiency.

[0003] Existing methods for analyzing and displaying data lineage are generally rigid and singular: in terms of electronic transaction data collection, current methods rely solely on database or log collection, which is relatively weak and lacks scalability, resulting in the loss of most useful electronic transaction data and thus low accuracy in data lineage analysis; most data analysis relies on parsing SQL statements to analyze relationships between data, but analysis of non-SQL data is rare, and even complex SQL relationship data lineage analysis is not possible, which hinders diversified data collection and analysis, affects in-depth and refined lineage analysis, and thus results in low accuracy in data lineage analysis. Summary of the Invention

[0004] This invention provides a method, apparatus, and computer-readable storage medium for analyzing data lineage, with the main objective of solving the problem of low accuracy in data lineage analysis.

[0005] To achieve the above objectives, the present invention provides a data kinship analysis method, comprising:

[0006] Obtain raw data from both real-time and offline links;

[0007] Identify the data type of the original data and select a preset parsing model based on the data type;

[0008] The original data is parsed using the preset parsing model to obtain the data nodes, task nodes, and data flow relationships between the data nodes and task nodes of the original data.

[0009] A lineage diagram is constructed based on the data nodes, the task nodes, and the data flow relationships.

[0010] Optionally, identifying the data type of the original data includes:

[0011] The original data is segmented into words to obtain the original data word sequence;

[0012] The original data word sequence is quantized using a preset word vector model to obtain the original data vector sequence;

[0013] Obtain multiple decision trees in a pre-defined random forest classification model, as well as the decision dimension index and decision conditions of at least one layer of nodes in each decision tree;

[0014] Based on the decision dimension index of the first node in the random forest classification model, feature extraction is performed on the original data vector sequence to obtain the feature values ​​of the original data vector sequence on the split dimension of the first node;

[0015] The feature value is judged based on the decision condition of the first node, and the second node to be traversed is determined from the branch nodes of the first node based on the judgment result.

[0016] Based on the current decision dimension index and decision conditions, continue to extract the feature values ​​of the original data vector sequence at the second node and determine the next node to be traversed until the decision tree traversal is completed, and obtain the data type of the original data.

[0017] Optionally, selecting a preset parsing model based on the data type includes:

[0018] When the data type is SQL-type data, select the preset SQL parsing model;

[0019] When the data type is JSON, select the preset JSON parsing model;

[0020] When the data type is general-purpose data, select the preset text parsing model.

[0021] Optionally, the step of parsing the original data using the preset parsing model to obtain the data nodes, task nodes, and data flow relationships between the data nodes and task nodes of the original data includes:

[0022] When the preset parsing model is a preset SQL parsing model, regularized SQL statements are extracted from the original data;

[0023] The characters in the regularized SQL statement are converted into a symbol stream using a pre-built lexical analyzer;

[0024] The symbol stream is converted into a syntax tree using a pre-built parser;

[0025] The syntax tree is parsed into an abstract syntax tree by a preset compiler;

[0026] Extract the operation type of the regularized SQL statement from the abstract syntax tree and use it as a task node;

[0027] Based on the operation type, operation field information and table name information are extracted from the corresponding node in the abstract syntax tree as data nodes. The node relationship corresponding to the operation field information and table name information is extracted from the abstract syntax tree as the data flow relationship between the data nodes and each node of the task node.

[0028] Optionally, constructing the lineage graph based on the data nodes, the task nodes, and the data flow relationship includes:

[0029] Obtain an existing kinship graph, and extract the relevant nodes of the data node from the existing kinship graph;

[0030] Using a graph tool, a lineage graph is constructed based on the data nodes, task nodes, data flow relationships, and related nodes.

[0031] Optionally, obtaining the raw target data of the real-time link and the offline link includes:

[0032] Write the subscribed or pushed real-time change data to the message queue, and retrieve the real-time data from the message queue;

[0033] Use pre-set scheduling tasks to periodically acquire offline data;

[0034] Abnormal data is removed from the real-time data and the offline data to obtain the original target data.

[0035] Optionally, the step of periodically acquiring offline data using a preset scheduling task includes:

[0036] Select a preset task type based on user needs;

[0037] Obtain the code block corresponding to the task type and execute the code block to capture offline data.

[0038] To address the aforementioned problems, the present invention also provides a data lineage analysis method and apparatus, the apparatus comprising:

[0039] The data acquisition module is used to acquire raw data from real-time and offline links;

[0040] The parsing model selection module is used to identify the data type of the original data and select a preset parsing model according to the data type.

[0041] The parsing module is used to parse the original data using the preset parsing model to obtain the data nodes, task nodes, and data flow relationships between the data nodes and each node of the task nodes in the original data.

[0042] A lineage graph construction module is used to construct a lineage graph based on the data nodes, the task nodes, and the data flow relationship.

[0043] To address the above problems, the present invention also provides an electronic device, the electronic device comprising:

[0044] At least one processor; and,

[0045] A memory communicatively connected to the at least one processor; wherein,

[0046] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the data lineage analysis method described above.

[0047] To address the aforementioned problems, the present invention also provides a computer-readable storage medium storing at least one computer program, which is executed by a processor in an electronic device to implement the data lineage analysis method described above.

[0048] This invention acquires raw data from both real-time and offline links, resulting in a wider and more comprehensive data source and thus more accurate raw data. Furthermore, it identifies the data type of the raw data, selects a preset parsing model based on the data type, and chooses different parsing models for different data types to parse the corresponding raw data. This yields data nodes, task nodes, and the data flow relationships between the data nodes and task nodes, adapting to different data types and improving data analysis efficiency and accuracy. Finally, a lineage graph is constructed based on the data nodes, task nodes, and data flow relationships, and provided to the user. Therefore, the data lineage analysis method, apparatus, electronic device, and computer-readable storage medium proposed in this invention can solve the problem of low accuracy in data lineage analysis. Attached Figure Description

[0049] Figure 1 This is a flowchart illustrating a data kinship analysis method provided in an embodiment of the present invention;

[0050] Figure 2 for Figure 1 The diagram shows a detailed implementation flow of one step in the data kinship analysis method.

[0051] Figure 3 for Figure 1 The diagram shows a detailed implementation flow of another step in the data kinship analysis method.

[0052] Figure 4 This is a functional block diagram of a data kinship analysis method and apparatus provided in an embodiment of the present invention;

[0053] Figure 5 This is a schematic diagram of the structure of an electronic device for implementing the data lineage analysis method according to an embodiment of the present invention.

[0054] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0055] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0056] This application provides a data lineage analysis method. The execution subject of the data lineage analysis method includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this application: a server, a terminal, etc. In other words, the data lineage analysis method can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0057] Reference Figure 1 The diagram shown is a flowchart illustrating a data kinship analysis method according to an embodiment of the present invention. In this embodiment, the data kinship analysis method includes:

[0058] S1. Obtain the raw data of the real-time link and the offline link.

[0059] In this embodiment of the invention, the real-time link is the link corresponding to the real-time online data of the business system that needs to be collected.

[0060] In this embodiment of the invention, the offline link is the link corresponding to the offline data of the business system that needs to be collected.

[0061] In this embodiment of the invention, the original data may come from data generated by the data development platform, network data from third-party services such as APP and WEB, electronic transaction data such as local files or log files.

[0062] For details, please refer to Figure 2 As shown, obtaining the raw target data of the real-time link and offline link in S1 includes:

[0063] S11. Write the subscribed or pushed real-time change data into a message queue, and retrieve the real-time data from the message queue;

[0064] S12. Use a preset scheduling task to periodically acquire offline data;

[0065] S13. Remove abnormal data from the real-time data and the offline data to obtain the original target data.

[0066] In this embodiment of the invention, the preset scheduling task can encapsulate different task types, such as Python, Shell, FLINK, etc., into a unified task model. Different tasks corresponding to different task types can be selected according to different user needs, realizing the dynamic expansion of the scheduling task and capturing offline data. For example, to add a task of type FLINK, the corresponding task model can be encapsulated into a JAR file and uploaded to the preset scheduling platform. Only the task rules need to be specified to execute the specific task.

[0067] Furthermore, the step of periodically acquiring offline data using a preset scheduling task includes:

[0068] Select a preset task type based on user needs;

[0069] Obtain the code block corresponding to the task type and execute the code block to capture offline data.

[0070] In another embodiment of the present invention, an ETL tool can be used to collect raw offline data and raw real-time data from various business systems, and the raw message data of the various business systems (shopping platform, claims, insurance, online banking, etc.) can be obtained from the message processing engine; abnormal data in the raw offline data, the raw real-time data, and the raw message data can be removed to obtain offline data, real-time data, and message data, and the offline data, the real-time data, and the message data can be used as the raw target data.

[0071] In this embodiment of the invention, the ETL (Extract-Transform-Load) tool is a data extraction, transformation, and loading tool, such as Kettle, Talend, Informatica, Datax, etc.

[0072] In this embodiment of the invention, real-time and offline data are acquired, and the data sources are more extensive and comprehensive, making the original data more accurate. The preset scheduling tasks can be flexibly configured with scheduling rules, and different task types can be selected according to different user needs, realizing the dynamic expansion of scheduling tasks and strong data capture capabilities.

[0073] S2. Identify the data type of the original data and select a preset parsing model based on the data type.

[0074] In this embodiment of the invention, the data types include: SQL-type data, JSON-type data, general-purpose data, etc.

[0075] Specifically, the data type for identifying the original data described in S2 includes:

[0076] The original data is segmented into words to obtain the original data word sequence;

[0077] The original data word sequence is quantized using a preset word vector model to obtain the original data vector sequence;

[0078] Obtain multiple decision trees in a pre-defined random forest classification model, as well as the decision dimension index and decision conditions of at least one layer of nodes in each decision tree;

[0079] Based on the decision dimension index of the first node in the random forest classification model, feature extraction is performed on the original data vector sequence to obtain the feature values ​​of the original data vector sequence on the split dimension of the first node;

[0080] The feature value is judged based on the decision condition of the first node, and the second node to be traversed is determined from the branch nodes of the first node based on the judgment result.

[0081] Based on the current decision dimension index and decision conditions, continue to extract the feature values ​​of the original data vector sequence at the second node and determine the next node to be traversed until the decision tree traversal is completed, and obtain the data type of the original data.

[0082] In this embodiment of the invention, the preset random forest classification model is a model that integrates multiple trees using the idea of ​​ensemble learning. Its basic unit is a decision tree. Taking a classification problem as an example, each decision tree is a classifier. For an input sample, N trees will have N classification results. The random forest integrates all the classification voting results and designates the category with the most votes as the final output, thereby obtaining the optimal category.

[0083] In this embodiment of the invention, the preset parsing model can be divided into SQL parsing model, JSON parsing model, and text parsing model.

[0084] Furthermore, the selection of a preset parsing model based on the data type in S2 includes:

[0085] When the data type is SQL-type data, select the preset SQL parsing model;

[0086] When the data type is JSON, select the preset JSON parsing model;

[0087] When the data type is general-purpose data, select the preset text parsing model.

[0088] In this embodiment of the invention, different parsing models are selected for different data types to parse the corresponding raw data, which helps to improve the efficiency and accuracy of data analysis.

[0089] S3. Analyze the original data using the preset parsing model to obtain the data nodes, task nodes, and data flow relationships between the data nodes and task nodes of the original data.

[0090] For details, please refer to Figure 3 As shown, S3 includes:

[0091] S31. When the preset parsing model is a preset SQL parsing model, extract the regularized SQL statement from the original data;

[0092] S32. Use a pre-built lexical analyzer to convert the characters in the regularized SQL statement into a symbol stream;

[0093] S33. The symbol stream is converted into a syntax tree using a pre-built parser;

[0094] S34. The syntax tree is parsed into an abstract syntax tree by a preset compiler;

[0095] S35. Extract the operation type of the regularized SQL statement from the abstract syntax tree and use it as a task node;

[0096] S36. Based on the operation type, extract the operation field information and table name information from the corresponding node in the abstract syntax tree as data nodes, and extract the node relationship corresponding to the operation field information and table name information from the abstract syntax tree as the data flow relationship between the data nodes and each node of the task node.

[0097] In this embodiment of the invention, the lexical analyzer is a computer science program that converts a sequence of characters into a sequence of tokens. It analyzes and quantifies a meaningless stream of characters, translating it into a discrete sequence of tokens, including keywords, identifiers, etc.

[0098] Furthermore, in this embodiment of the invention, the syntactic analysis (also called parsing) is a program that analyzes input text composed of word sequences according to a given formal grammar and determines its syntactic structure. Its function is to perform grammatical checks on the word sequences and construct a syntax tree composed of the input word sequences.

[0099] In this embodiment of the invention, the operation types are divided into query operation type, read operation type, update operation type, insert operation type, and delete operation type. For example, the query operation type related statement can be the select part of the SQL statement.

[0100] In another embodiment of the present invention, when the preset parsing model is a preset JSON parsing model, regularized JSON statements are extracted from the original data; the regularized JSON statements are parsed according to JSON syntax rules to obtain operation type, operation field information, table name information, and the data flow relationship between the operation type, the operation field information, and the table name information. The operation type is used as a task node, and the operation field information and the table name information are used as data nodes. For example, if the operation type is a query, the operation field information is a person's policy cost, and the table name information is a policy summary table for XX days, it means that a person's policy cost can be queried from the policy summary table for XX months and the policy summary table for XX days, indicating that there is a data correspondence between the policy cost of a person in the policy summary table for XX months and the policy summary table for XX days.

[0101] In this embodiment of the invention, JSON (JavaScript Object Notation) is a lightweight data exchange format that is easy to read and write, and also easy for machines to parse and generate.

[0102] In this embodiment of the invention, the preset text parsing model can parse data nodes, task nodes, and the data flow relationship between each node of the data nodes and the task nodes from the original data of general data type.

[0103] In this embodiment of the invention, different parsing models are used to parse raw data of different data types, which helps to improve the efficiency and accuracy of data analysis.

[0104] S4. Construct a lineage diagram based on the data nodes, the task nodes, and the data flow relationship.

[0105] Specifically, S4 includes:

[0106] Obtain an existing kinship graph, and extract the relevant nodes of the data node from the existing kinship graph;

[0107] Using a graph tool, a lineage graph is constructed based on the data nodes, task nodes, data flow relationships, and related nodes.

[0108] In another embodiment of the present invention, when the system does not have an existing lineage diagram, a lineage diagram can be constructed directly using a graph library tool based on the data nodes, the task nodes, and the data flow relationship.

[0109] In this embodiment of the invention, a graph database can be used to construct a lineage graph based on the data nodes, the task nodes, and the data flow relationships. The graph database can store various types of intermediate or final data lineage relationships. The graph database can be a Neo4j graph database, a Nebula Graph graph database, or similar databases.

[0110] In another embodiment of the present invention, the data node, the task node, and the data flow relationship can also be encapsulated as lineage data corresponding to the API interface for other data ports to call.

[0111] In another embodiment of the present invention, the data node, the task node, and the data flow relationship can be encapsulated into a JSON object, and the JSON object can be converted into a lineage diagram.

[0112] In this embodiment of the invention, the bloodline diagram can be directly displayed on the user's device.

[0113] This invention acquires raw data from both real-time and offline data sources, resulting in a wider and more comprehensive data source and thus more accurate raw data. Furthermore, it identifies the data type of the raw data, selects a preset parsing model based on the data type, and chooses different parsing models for different data types to parse the corresponding raw data. This yields data nodes, task nodes, and the data flow relationships between the data nodes and task nodes, adapting to different data types and improving data analysis efficiency and accuracy. Finally, a lineage graph is constructed based on the data nodes, task nodes, and data flow relationships, and provided to the user. Therefore, the data lineage analysis method proposed in this invention can solve the problem of low accuracy in data lineage analysis.

[0114] like Figure 4 The diagram shown is a functional block diagram of a data lineage analysis method and apparatus provided in an embodiment of the present invention.

[0115] The data kinship analysis method device 100 of the present invention can be installed in an electronic device. Depending on the functions implemented, the data kinship analysis method device 100 may include a data acquisition module 101, a parsing model selection module 102, a parsing module 103, and a kinship graph construction module 104. The module described in this invention can also be called a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and are stored in the memory of the electronic device.

[0116] In this embodiment, the functions of each module / unit are as follows:

[0117] The data acquisition module 101 is used to acquire the raw data of the real-time link and the offline link;

[0118] The parsing model selection module 102 is used to identify the data type of the original data and select a preset parsing model according to the data type.

[0119] The parsing module 103 is used to parse the original data using the preset parsing model to obtain the data nodes, task nodes, and data flow relationships between the data nodes and each node of the task nodes in the original data.

[0120] The bloodline graph construction module 104 is used to construct a bloodline graph based on the data nodes, the task nodes, and the data flow relationship.

[0121] In detail, each module in the data kinship analysis method apparatus 100 described in this embodiment of the invention adopts the same characteristics as described above during use. Figures 1 to 3The data lineage analysis method described herein uses the same technical means and can produce the same technical effect, so it will not be elaborated here.

[0122] like Figure 5 The diagram shown is a structural schematic of an electronic device for implementing a data lineage analysis method according to an embodiment of the present invention.

[0123] The electronic device 1 may include a processor 10, a memory 11, a communication bus 12, and a communication interface 13. It may also include a computer program stored in the memory 11 and capable of running on the processor 10, such as a data lineage analysis method program.

[0124] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the memory 11 (e.g., executing data lineage analysis methods) and calls data stored in the memory 11 to perform various functions of the electronic device and process data.

[0125] The memory 11 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of an electronic device, such as a portable hard drive. In other embodiments, the memory 11 can be an external storage device of the electronic device, such as a plug-in portable hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc. Furthermore, the memory 11 can include both internal and external storage units of the electronic device. The memory 11 can be used not only to store application software and various types of data installed on the electronic device, such as the code of a data lineage analysis method program, but also to temporarily store data that has been output or will be output.

[0126] The communication bus 12 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 11 and at least one processor 10, etc.

[0127] The communication interface 13 is used for communication between the aforementioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, Bluetooth interface, etc.), typically used to establish communication connections between the electronic device and other electronic devices. The user interface may be a display, an input unit (such as a keyboard), or optionally, a standard wired or wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device and to display a visual user interface.

[0128] Figure 5 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 5 The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0129] For example, although not shown, the electronic device may also include a power supply (such as a battery) to power the various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0130] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0131] The data lineage analysis method program stored in the memory 11 of the electronic device 1 is a combination of multiple instructions, which, when run in the processor 10, can achieve the following:

[0132] Obtain raw data from both real-time and offline links;

[0133] Identify the data type of the original data and select a preset parsing model based on the data type;

[0134] The original data is parsed using the preset parsing model to obtain the data nodes, task nodes, and data flow relationships between the data nodes and task nodes of the original data.

[0135] A lineage diagram is constructed based on the data nodes, the task nodes, and the data flow relationships.

[0136] Specifically, the specific implementation method of the processor 10 for the above instructions can be referred to the description of the relevant steps in the corresponding embodiment of the accompanying drawings, and will not be repeated here.

[0137] Furthermore, if the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0138] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor of an electronic device, can perform the following:

[0139] Obtain raw data from both real-time and offline links;

[0140] Identify the data type of the original data and select a preset parsing model based on the data type;

[0141] The original data is parsed using the preset parsing model to obtain the data nodes, task nodes, and data flow relationships between the data nodes and task nodes of the original data.

[0142] A lineage diagram is constructed based on the data nodes, the task nodes, and the data flow relationships.

[0143] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0144] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0145] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0146] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0147] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0148] The blockchain referred to in this invention is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0149] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0150] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a system claim may also be implemented by a single unit or device through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0151] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A data kinship analysis method, characterized in that, The method comprises: acquiring original data of real-time links and offline links; identifying the data type of the original data, and selecting a preset analysis model according to the data type; analyzing the original data by using the preset analysis model to obtain data nodes, task nodes, and data flow direction relationships between each node of the data nodes and the task nodes; constructing a blood relationship graph according to the data nodes, the task nodes, and the data flow direction relationships; wherein the identification of the data type of the original data comprises: performing word segmentation on the original data to obtain an original data word sequence; quantifying the original data word sequence by using a preset word vector model to obtain an original data vector sequence; acquiring a plurality of decision trees in a preset random forest classification model and a decision dimension index and a decision condition of at least one layer of nodes in each decision tree; extracting features of the original data vector sequence at a first node of the random forest classification model according to the decision dimension index of the first node to obtain a feature value of the original data vector sequence at the split dimension of the first node; judging the feature value according to the decision condition of the first node, and determining a second node to be traversed from the branch nodes of the first node according to the judgment result; continuing to extract the feature value of the original data vector sequence at the second node according to the current decision dimension index and the decision condition, and determining a next node to be traversed until the decision tree is traversed to obtain the data type of the original data.

2. The data kinship analysis method of claim 1, wherein, The selection of the preset analysis model according to the data type comprises: when the data type is SQL data, a preset SQL analysis model is selected; when the data type is JSON data, a preset JSON analysis model is selected; when the data type is general data, a preset text analysis model is selected.

3. The data pedigree analysis method of claim 1, wherein, The analysis of the original data by using the preset analysis model to obtain data nodes, task nodes, and data flow direction relationships between each node of the data nodes and the task nodes comprises: when the preset analysis model is a preset SQL analysis model, regularized SQL statements are extracted from the original data; characters in the regularized SQL statements are converted into symbol streams by using a pre-constructed lexical analyzer; the symbol streams are converted into a syntax tree by using a pre-constructed syntax analyzer; the syntax tree is parsed into an abstract syntax tree by a preset compiler; operation types of the regularized SQL statements are extracted from the abstract syntax tree as task nodes; operation field information and table name information are extracted from corresponding nodes in the abstract syntax tree as data nodes according to the operation types, and node relationships of the operation field information and the table name information are extracted from the abstract syntax tree as data flow direction relationships between each node of the data nodes and the task nodes.

4. The data pedigree analysis method of claim 1, wherein, The construction of the blood relationship graph according to the data nodes, the task nodes, and the data flow direction relationships comprises: acquiring an existing blood relationship graph, and acquiring related nodes of the data nodes from the existing blood relationship graph; A blood relationship graph is constructed according to the data nodes, the task nodes, the data flow direction relationship and the related nodes by using a library tool.

5. The data pedigree analysis method of claim 1, wherein, The raw target data of the real-time link and the offline link is obtained, including: Real-time change data is written to a message queue through subscription or pushing, and real-time data is obtained from the message queue; Offline data is obtained in a timing manner by using a preset scheduling task; Abnormal data in the real-time data and the offline data is removed to obtain the raw target data.

6. The data kinship analysis method of claim 5, wherein, The offline data is obtained in a timing manner by using a preset scheduling task, including: A preset task type is selected according to a user demand; A code block corresponding to the task type is obtained, and the code block is executed to capture offline data.

7. A data kinship analysis method apparatus for implementing the data kinship analysis method according to any one of claims 1 to 6, characterized in that, The apparatus includes: A data acquisition module is configured to acquire raw data of a real-time link and an offline link; An analysis model selection module is configured to identify a data type of the raw data, and select a preset analysis model according to the data type; An analysis module is configured to analyze the raw data by using the preset analysis model to obtain data nodes, task nodes and a data flow direction relationship between each node of the data nodes and the task nodes; A blood relationship graph construction module is configured to construct a blood relationship graph according to the data nodes, the task nodes and the data flow direction relationship.

8. An electronic device, comprising: The electronic device includes: At least one processor; and A memory connected in communication with the at least one processor; wherein The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the data blood relationship analysis method of any one of claims 1 to 6.

9. A computer readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to implement the data blood relationship analysis method of any one of claims 1 to 6.

Images