Visual data platform full-link data flow tracing method, system, device and medium

Abstract

The invention relates to the technical field of big data management and data analysis, and discloses a visual data platform full-link data flow traceability method, system and device and a medium, and the method comprises the steps: monitoring a job scheduling process submission event, analyzing the input and output configuration information of each job node, converting the information into a data flow model, and storing the data flow model in a traceability database; static analysis is conducted on node scripts and codes, and analysis results are stored in a traceability database; generating a unique identifier for each data record, and recording related information in a data circulation process; and writing the data flow relationship into a graph database Neo4j through a Kafka asynchronous message, and performing data traceability query by a user through a visual interface. The method has the beneficial effects that the data quality and safety are improved, and the limitation and complexity of a mainstream open source tool in the aspect of a data blood relationship analysis function are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of big data governance and data analysis, and specifically to a full-link data flow tracing method, system, equipment and medium for a visual data platform. Background Art

[0002] With the continuous development of big data technology, data platforms have become an important tool for enterprises to process, store and analyze data. However, in data platforms, data often goes through multiple links from generation to application, including data collection, cleaning, storage, processing, analysis and application. In these links, the quality and security of data are affected by many factors, such as operational errors, system vulnerabilities, etc.

[0003] The existing data platform lacks an effective data flow tracking mechanism and cannot fully record the flow of data in each link from generation to application. It is impossible to intuitively present the flow of data in each link. Even if some systems can record data flow logs, this information is often stored in text form, which is difficult for non-technical personnel to understand and analyze. This makes it difficult to trace the historical path of the data when data problems occur, and it is impossible to determine the specific location and cause of the problem. Once a data problem occurs, how to quickly locate and solve the problem becomes a major challenge. Therefore, a method that can fully track and visualize the data flow process is needed to improve data quality and security. Summary of the invention

[0004] In view of the above-mentioned problems, the present invention is proposed.

[0005] Therefore, the technical problem solved by the present invention is: how to comprehensively track the flow of data in the data platform and display each link and status of the data flow through visualization means.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions: a method for tracing the source of full-link data flow on a visualization data platform, comprising:

[0007] Monitor the submission events of the job scheduling process, parse the input and output configuration information of each job node, convert it into a data flow model and save it in the traceability database;

[0008] Perform static analysis on node scripts and codes, save the analysis results in the traceability database, and analyze the data flow when the job scheduling process is running;

[0009] Generate a unique identifier for each data record, record relevant information during the data flow, and store the information in the traceability database;

[0010] The data flow relationship is written into the graph database Neo4j through Kafka asynchronous messages, and the association relationship between the data record ID and the flow information in the traceability database is established. Users can perform data traceability queries through a visual interface.

[0011] As a preferred solution of the full-link data flow tracing method of the visualization data platform described in the present invention, wherein: the monitoring of the job scheduling process submission event includes, in the job scheduling system, configuring an event listener to capture the job submission event; using a custom event processor to monitor the submission event, and when the event is captured, extracting the job ID, job name, and scheduling time metadata;

[0012] The captured metadata is stored in the database, and the job configuration file is parsed using the SQL database storage job metadata table to extract the input and output configuration information of each job node.

[0013] As a preferred solution of the full-link data flow tracing method of the visualization data platform described in the present invention, the static analysis includes establishing a data flow model, assuming that each operation node in the code is , whose input data source is , the output data source is:

[0014] The type of operation performed by each node is represented as ; Build a directed graph model and represent the data flow as a directed graph ,in:

[0015] Represents a collection of nodes. It is a set of directed edges, indicating the data flow relationship:

[0016] in, Represents a slave node To Node The data flow relationship edge; establish a data flow relationship, if the node The output data source is the node , then there is a directed edge :

[0017] Establish a static data flow path, data from the node Flow to Node Path It is expressed as:

[0018] Dynamic data operations and time-dependent node operations at runtime include, assuming that at time The operation nodes running are , whose input and output data sources are and ;

[0019] Establish a dynamic data flow model, and the data flow relationship at runtime is expressed as a time series model , where each time point The data flow is expressed as:

[0020] Dynamic path update: at time Generated data flow path It is expressed as:

[0021] Combine static and dynamic models, update the static model, and transfer the dynamic data to the results Static data flow model Combined to form an updated data flow diagram The formula is:

[0022] The data flow graph is fully represented. Combining the static and dynamic data flow relationships, the data flow path is finally represented as:

[0023] From the starting point To the end node The complete data flow path includes all static and dynamic nodes and edges.

[0024] As a preferred solution of the full-link data flow tracing method of the visualization data platform described in the present invention, wherein: the generating of a unique identifier for each data record includes, initializing the input content includes, , starting node , target node , edge weight function , maximum parallelism ;

[0025] The output includes, from the starting node To the target node The shortest path and the least cost path; initialize the queue and priority queue , Indicates the node storing the current level. Represents a priority queue, according to edge weight Sort the nodes; put the starting node Put in queue Initialize its cost ;

[0026] Initialize the node collection , used to store visited nodes, initially empty, for parallel breadth-first search;

[0027] When the queue Not empty and the target node was not found , do the following:

[0028] Process nodes at the same level in parallel and queue The nodes in the Divide into multiple subsets, each of which is assigned to a processing unit; process all neighboring nodes of the nodes in each subset in parallel; process each node Neighbor nodes ;

[0029] For each unvisited neighbor node , calculate from the starting node arrive The cumulative cost is expressed as:

[0030] in, Indicates from arrive The edge, represents the weight of the edge;

[0031] like If it has not been visited or a lower-cost path is found, update The price , and Put in priority queue middle:

[0032] Select the next node from the priority queue and follow the priority queue The node cost in the sequence is small to large, and the next node is selected for access and put into the queue. , continue the next round of breadth-first search.

[0033] As a preferred solution of the full-link data flow tracing method of the visualization data platform described in the present invention, the next round of breadth-first search includes dynamically adjusting the maximum parallelism and the priority of the priority queue to cope with changes in system resources and current load; if the target node is found , the search is terminated and the output is arrive The shortest path; if all nodes are traversed , then the output is The shortest paths to all other nodes;

[0034] If you need to output the path instead of just the path length, Back to the starting point , output the shortest path;

[0035] Priority update of priority queue, priority queue The set of nodes sorted by cost in is: ,

[0036] Every time from Take out the node with the smallest cost for the next traversal;

[0037] Dynamically adjust the degree of parallelism, parallelism The adjustment is based on the current system load and remaining resources Decide: , in, Indicates the maximum degree of parallelism allowed by the system. Indicates currently available resources. Indicates the current load.

[0038] As a preferred solution of the full-link data flow tracing method of the visualization data platform described in the present invention, the Kafka asynchronous message includes generating a data flow message, generating a data flow message containing key information of the data record ID, operation type, and timestamp at each node of data processing; each message is represented as a triple: , in, A unique identifier representing a data record; Indicates the operation type; Indicates timestamp; Kafka message sending, sends data flow message to Kafka topic to realize asynchronous transmission of message; Assume that each data flow event generates a message , then the topic of the Kafka message queue Represented as a sequence of messages: , in, Indicates the first messages; the messages are arranged in chronological order, that is, for any ,have .

[0039] As a preferred solution of the full-link data flow tracing method of the visualization data platform described in the present invention, it is characterized in that: the graph database Neo4j includes configuring Kafka consumers to consume data flow messages from the Kafka queue in real time; the consumer extracts messages from the queue , and parse out the ID, operation type and timestamp, and write the message to the Neo4j graph database: In Neo4j, based on the data in the message record Create and update nodes; depending on the operation type and timestamp Create or update the relationship between nodes in Neo4j to form a data flow path;

[0040] Each data record Corresponds to a node in Neo4j , the data node formula is expressed as:

[0041] Relational modeling of data operations, assuming two data nodes and There is a data flow relationship, represented by a directed edge : , in, Indicates the time when the operation occurred; Kafka message The information in will be mapped to the Neo4j graph database, and the nodes and relationships will be updated. The formula is: , Each message Corresponding to one or more such edge relations .

[0042] As a preferred solution of the full-link data flow tracing method of the visualization data platform described in the present invention, wherein: the establishment of data records The relationship with the information flowing in the traceability database includes building a Neo4j graph model and visually modeling nodes and relationships. In Neo4j, nodes are used to represent each data record. , use edges to represent the operations and order of data flow; through the construction of the graph, establish a complete data flow path, so that the flow history of each node can be traced by traversing the edges in the graph;

[0043] Design By node collection and edge set The flow history formula of the node structure of the Neo4j graph is expressed as: ; ;

[0044] The integration of Kafka message queue and Neo4j graph database realizes asynchronous writing and storage of data flow information. Users can easily query and track data flow paths through a visual interface.

[0045] As a preferred solution of the full-link data flow tracing system of the visualization data platform described in the present invention, wherein:

[0046] The event monitoring module monitors the job submission events in the job scheduler in real time, captures the job metadata, and stores the extracted job metadata in the traceability database Neo4j of the graph database module;

[0047] The data flow model module parses the input and output configuration information of each job node, builds a data flow model, and performs static analysis on the scripts and codes of the job nodes; generates a unique identifier for each data record and records data flow information;

[0048] The graph database module builds the graph database Neo4j. Users can use the graphical interface to perform data source tracing queries and view the full-link information of data flow.

[0049] The integration and visualization query module asynchronously sends data flow information to the graph database module through the Kafka message queue, and establishes a graph model of data nodes and their flow relationships in the graph database Neo4j.

[0050] A computer device comprises: a memory and a processor; the memory stores a computer program, comprising: the steps of implementing any one of the methods of the present invention when the processor executes the computer program.

[0051] A computer-readable storage medium stores a computer program, comprising: when the computer program is executed by a processor, the steps of implementing any one of the methods of the present invention are implemented.

[0052] The beneficial effects of the present invention are as follows: data quality and security are improved. By comprehensively tracking the data flow process in the data platform and recording detailed information for each link, it helps to promptly discover and correct problems in the data flow process, thereby improving data quality and security. Improved data traceability efficiency: by displaying data flow information through a visual interface, users can intuitively view the full-link flow process of data in the data platform, quickly locate data problems, and improve the efficiency and accuracy of data traceability. A flexible data display method is provided. By providing a variety of visual components such as data flow diagrams, timelines, and operation lists, users can choose a suitable display method as needed to better understand and analyze the data flow process. The implementation process of the present invention solves the limitations and complexity of mainstream open source tools in data lineage analysis functions through multi-level and multi-dimensional data flow analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0054] Figure 1 An overall flow chart of data flow tracing of a full-link data flow tracing method of a visual data platform provided in the first embodiment of the present invention.

[0055] Figure 2 A module diagram of a full-link data flow tracing system for a visual data platform provided for the first embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0057] Example 1, reference Figure 1-2 , is an embodiment of the present invention, and provides a method for tracing the source of full-link data flow on a visual data platform, including:

[0058] S1: Listen to the job scheduling process submission events, parse the input and output configuration information of each job node, convert it into a data flow model and save it in the traceability database.

[0059] Furthermore, monitoring the job scheduling process submission event includes, in the job scheduling system, configuring an event listener to capture the job submission event; using a custom event handler to monitor the submission event, and when the event is captured, extracting the job ID, job name, and scheduling time metadata.

[0060] Furthermore, the captured metadata is stored in a database, and the SQL database is used to store the job metadata table to parse the job configuration file and extract the input and output configuration information of each job node.

[0061] Furthermore, we monitor the submission events of the job scheduling process and parse the input and output configuration information of each job node. Different links usually have different scheduling configuration parameters, such as library table data integration, which usually configures the data source table and destination table as well as field mapping information; interface data integration, which usually configures the interface source URL, response body data format and data retrieval path, and field storage mapping information; file integration, which usually configures the source FTP path, target file FTP path, mapping relationship, etc. These configuration information are usually modeled in a semi-structured form such as JSON format.

[0062] It should be noted that the job monitor parses these node models through the self-developed DataTransfProxyLog plug-in to extract useful data flow information. This plug-in can achieve the universality of data flow analysis functions in various scenarios by configuring model extraction parameters for different types of service components.

[0063] S2: Perform static analysis on node scripts and codes, save the analysis results in the traceability database, and analyze the data flow when the job scheduling process is running.

[0064] Furthermore, the static analysis includes constructing a data flow model assuming that each operation node in the code is , whose input data source is , the output data source is:

[0065] Furthermore, the type of operation performed by each node is represented as ; Build a directed graph model and represent the data flow as a directed graph ,in:

[0066] Furthermore, to represent a set of nodes, It is a set of directed edges, indicating the data flow relationship: , in, Represents a slave node To Node The data flow relationship edge; establish a data flow relationship, if the node The output data source is the node , then there is a directed edge :

[0067] Furthermore, a static data flow path is established, and data from the node Flow to Node Path It is expressed as:

[0068] Dynamic data operations and time-dependent node operations at runtime include, assuming that at time The operation nodes running are , whose input and output data sources are and .

[0069] Furthermore, a dynamic data flow model is established, and the data flow relationship at runtime is represented as a time series model. , where each time point The data flow is expressed as:

[0070] Dynamic path update, in time Generated data flow path It is expressed as:

[0071] Furthermore, by combining static and dynamic models, the static model is updated and the dynamic data flow results are Static data flow model Combined to form an updated data flow diagram The formula is:

[0072] Furthermore, the data flow graph is fully represented. Combining the static and dynamic data flow relationships, the data flow path is finally represented as:

[0073] From the starting point To the end node The complete data flow path includes all static and dynamic nodes and edges.

[0074] Furthermore, based on the compilation principle, by scanning and parsing the source code, as well as statically analyzing and listing the paths involved in the program logic, an objective reflection of the data flow can be achieved. This step is also configured in the DataTransfProxyLog plug-in to realize the automatic parsing of static resources. Usually, the static analysis function can be enabled for SQL development task components. This plug-in is compatible with the current mainstream database operation syntax, views, triggers, and stored procedures, and performs in-depth analysis of database metadata relationships, specifically parsing SQL statements, stored procedures, ETL processes and other files. Due to complex code and application environment, automatic parsing can cover 70-95% of enterprise data.

[0075] It should be noted that the data flow analysis and processing process in the above links all use the asynchronous listening processing mode of the plug-in. The analysis results will be sent to Kafka messages and finally saved in the graph database Neo4j to establish the association between the data record ID and the flow information in the traceability database, ensuring that the corresponding full-link flow information can be quickly queried through the data record ID. This association can be automatically established by the data platform or manually maintained by the user.

[0076] S3: Generate a unique identifier for each data record, record relevant information during the data flow, and store the information in the traceability database.

[0077] Further, generating a unique identifier for each data record includes initializing input content including: , starting node , target node , edge weight function , maximum parallelism .

[0078] Furthermore, the output includes, from the starting node To the target node The shortest path and the least cost path; initialize the queue and priority queue , Indicates the node storing the current level. Represents a priority queue, according to edge weight Sort the nodes; put the starting node Put in queue Initialize its cost .

[0079] Furthermore, initialize the node set , used to store visited nodes, initially empty, parallel breadth-first search.

[0080] Furthermore, when the queue Not empty and the target node was not found , do the following:

[0081] Process nodes at the same level in parallel and queue The nodes in the Divide into multiple subsets, each of which is assigned to a processing unit; process all neighboring nodes of the nodes in each subset in parallel; process each node Neighbor nodes .

[0082] Furthermore, for each unvisited neighbor node , calculate from the starting node arrive The cumulative cost is expressed as: , in, Indicates from arrive The edge, Represents the weight of the edge.

[0083] Furthermore, if If it has not been visited, or a lower-cost path is found, update The price , and Put in priority queue middle:

[0084] Furthermore, the next node is selected from the priority queue according to the priority queue. The node cost in the sequence is small to large, and the next node is selected for access and put into the queue. , continue the next round of breadth-first search.

[0085] Furthermore, dynamically adjust the maximum degree of parallelism and the priority of the priority queue to cope with changes in system resources and current load; if the target node is found , the search is terminated and the output is arrive The shortest path; if all nodes are traversed , then the output is The shortest paths to all other nodes.

[0086] Furthermore, if you need to output the path instead of just the path length, Back to the starting point , output the shortest path.

[0087] Furthermore, the priority of the priority queue is updated. The set of nodes sorted by cost in is: ,

[0088] Furthermore, each time from Take out the node with the lowest cost for the next traversal; dynamically adjust the degree of parallelism. The adjustment is based on the current system load and remaining resources Decide: , in, Indicates the maximum degree of parallelism allowed by the system. Indicates currently available resources. Indicates the current load.

[0089] It should be noted that a visual interface is provided to display components such as data flow diagrams, timelines, and operation lists. Users can intuitively view the full-link flow process of data in the data platform by dragging, clicking, and other operations. The visual interface can be customized according to user needs to meet the viewing and analysis needs of different users. Allow users to perform data traceability queries through the visual interface. Users can enter data record IDs or other key information, and the system will query the corresponding full-link flow information based on the association relationship and display it on the visual interface. Users can quickly locate data problems based on the flow information and take corresponding solutions. At the same time, the system can also provide analysis and suggestion functions for data problems to help users better understand the causes of the problems and develop corresponding solutions.

[0090] S4: The data flow relationship is written into the graph database Neo4j through Kafka asynchronous messages, and the association relationship between the data record ID and the flow information in the traceability database is established. Users can perform data traceability queries through a visual interface.

[0091] Furthermore, the Kafka asynchronous message includes data flow message generation, which generates data flow messages containing key information such as data record ID, operation type, and timestamp at each node of data processing; each message is represented as a triple: , in, A unique identifier representing a data record; Indicates the operation type; Indicates timestamp; Kafka message sending, sends data flow message to Kafka topic to realize asynchronous transmission of message; Assume that each data flow event generates a message , then the topic of the Kafka message queue Represented as a sequence of messages: , in, Indicates the first messages; the messages are arranged in chronological order, that is, for any ,have .

[0092] Furthermore, Kafka consumers consume messages from the queue in sequence and take them out for processing. Consumed the message , then the consumption operation can be expressed as: , in, , indicating that consumption occurs after the message is generated. Consumption sequence: Kafka consumers consume messages in the queue in sequence, so the consumption sequence is also ordered, which can be expressed as: , in, Indicates the message The time consumed, and ; As a message queue, the state of a Kafka topic changes dynamically with the generation and consumption of messages; At this moment, the state of the Kafka queue It can be represented as a collection of messages that are not currently consumed: , in, Indicates the message has been generated, and Indicates that the message has not been consumed yet.

[0093] Furthermore, the Kafka message flow can be viewed as a continuous process from production to consumption. is a time function, which means that At this moment, the dynamic process of messages from production to consumption: , in, Represents the set of messages generated at time t; Represents the set of messages consumed at time t. Message generation rate and consumption rate: , in, Indicates at time A collection of messages generated at the moment. Indicates at time The collection of messages consumed at any given moment. Message generation rate Indicates at time The number of messages generated per unit time, the message consumption rate Indicates at time The number of messages consumed per unit time: ,

[0094] Furthermore, the queue length: at time Number of messages in the Kafka queue at time It can be expressed as the integral difference of message generation rate and consumption rate: , in, Indicates the number of messages in the current queue waiting to be consumed. Message flow stability: In order to ensure the stable operation of the Kafka system, the message generation rate and consumption rate A balance needs to be maintained; if the system is to remain stable, the queue length Should be finite, the stability condition requires:

[0095] Furthermore, in the long run, the rate at which messages are produced and consumed must be equal to prevent the queue from growing indefinitely.

[0096] Furthermore, the graph database Neo4j includes configuring Kafka consumers to consume data flow messages from the Kafka queue in real time; the consumer extracts messages from the queue , and parse out the ID, operation type and timestamp, and write the message to the Neo4j graph database.

[0097] Furthermore, in Neo4j, according to the data record in the message Create and update nodes; depending on the operation type and timestamp Create or update the relationship between nodes in Neo4j to form a data flow path.

[0098] Furthermore, each data record Corresponds to a node in Neo4j , the data node formula is expressed as:

[0099] Furthermore, the relational modeling of data operations assumes that two data nodes and There is a data flow relationship, represented by a directed edge : , in, Indicates the time when the operation occurred; Kafka message The information in will be mapped to the Neo4j graph database, and the nodes and relationships will be updated. The formula is:

[0100] Furthermore, each message Corresponding to one or more such edge relations The data record is established The relationship with the information flowing in the traceability database includes building a Neo4j graph model and visually modeling nodes and relationships. In Neo4j, nodes are used to represent each data record. , use edges to represent the operations and order of data flow; through the construction of the graph, a complete data flow path is established, so that the flow history of each node can be tracked by traversing the edges in the graph.

[0101] Furthermore, By node collection and edge set The flow history formula of the node structure of the Neo4j graph is expressed as: ,

[0102] Furthermore, the integration of Kafka message queue and Neo4j graph database realizes asynchronous writing and storage of data flow information, and users can easily query and track data flow paths through a visual interface.

[0103] It should be noted that the mathematical modeling framework of the Kafka message queue includes the description of message generation, message transmission, message consumption, queue status, and message flow. Through these mathematical representations, the performance of the Kafka system can be theoretically analyzed and optimized to ensure the stability and efficiency of message flow.

[0104] On the other hand, this embodiment also provides a visual data platform full-link data flow traceability system, such as Figure 2 As shown, it includes:

[0105] The event monitoring module monitors the job submission events in the job scheduler in real time, captures the job metadata, and stores the extracted job metadata in the traceability database.

[0106] The data flow model module parses the input and output configuration information of each job node, builds a data flow model, and performs static analysis on the scripts and codes of the job nodes.

[0107] The identification module generates a unique identifier for each data record and records data flow information.

[0108] The integrated and visualized query module asynchronously sends data flow information to the graph database Neo4j through the Kafka message queue, and establishes a graph model of data nodes and their flow relationships in Neo4j. Users can perform data traceability queries through a graphical interface and view the full-link information of data flow.

[0109] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0110] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0111] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0112] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0113] Example 2: The following is an embodiment of the present invention, which provides a method for tracing the full-link data flow of a visualization data platform. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0114] In order to verify the innovation and advantages of a full-link data flow traceability method for a visual data platform, the following experiment was designed and implemented. The experimental environment is a typical enterprise-level data platform, covering multiple links such as data collection, cleaning, storage, processing, analysis and application. Each module within the platform is composed of independent job nodes, which are coordinated and managed through the job scheduling process.

[0115] The experiment first built a simulated data platform and configured multiple job scheduling processes, including data collection nodes, data cleaning nodes, data storage nodes, data processing nodes, and data analysis nodes. Each node is configured with input and output data sources, and there are complex dependencies between nodes. To ensure the comprehensiveness of the experiment, the experiment introduced several common data problems, such as data loss, data tampering, and data delay.

[0116] The experiment is divided into the following steps:

[0117] Job scheduling process monitoring and data analysis: In the experimental environment, the job submission events in the job scheduler are monitored in real time. Whenever a new job is submitted, the system automatically analyzes the input and output configuration information of the job, converts this information into a data flow model, and saves it in the traceability database. This step ensures that the system can grasp the source, destination, and processing process of each data in real time.

[0118] Static analysis and data model storage: The experiment statically analyzes the scripts and codes of each job node, extracts the specific logic of data operations, such as filtering, aggregation, conversion, etc., and integrates the analysis results with the data flow model and stores them in the traceability database. This process ensures the accuracy and integrity of the data flow path.

[0119] Data flow records and unique identifier generation: Generate a unique identifier for each data record, and record relevant information in real time during the data flow process, such as operation time, operation type, input and output data sources, etc. All information is stored in the traceability database for subsequent data tracking and traceability queries.

[0120] Kafka and Neo4j integration and visualization: Data flow information is asynchronously transmitted to the Neo4j graph database through the Kafka message queue to establish the association between the data record ID and the flow information. Users can intuitively query and track the data flow path through the visualization interface, which is convenient for quickly locating data problems.

[0121] In the experiment, in order to compare the effects of the existing technology and the present invention, two environments were constructed: one was a traditional data platform that did not use the method of the present invention, and the other was an innovative data platform that applied the present invention. The same data problem was introduced into both environments, and key indicators in the data processing process were recorded, such as problem location time, data flow path accuracy, operation time, data loss, etc.

[0122] The data platform of the present invention is superior to the traditional data platform in all key performance indicators. First, in terms of problem location time, the data platform of the present invention only takes an average of 10 minutes to accurately locate the root cause of the data problem, while the traditional data platform takes 45 minutes. This advantage is due to the real-time data flow model and static analysis method in the present invention, which can immediately provide accurate flow path and operation information when data problems occur, so as to quickly locate the problem. The delay time of the traditional platform is 120 seconds, while the delay time of the present invention is only 10 seconds. The integration solution of the Kafka message queue and the Neo4j graph database of the present invention can significantly reduce the time delay in the data flow process, thereby improving the real-time performance of data processing. In terms of the success rate of traceability queries, the data platform of the present invention reached 98%, which is significantly higher than the 65% of the traditional data platform. This improvement further proves the advantage of the present invention in data traceability capabilities, and can provide accurate and fast data traceability query support in a variety of complex scenarios.

[0123] In terms of data comparison of the improved algorithms, the path accuracy of the improved algorithm of the present invention reached 98%, while the accuracy of the log recording and analysis algorithm was only 60%, the database-level data tracing algorithm was 85%, and the ETL-based tracing algorithm was 75%. This difference shows the accuracy of the present invention in data flow modeling, which ensures the high accuracy of the data flow path by recording the input and output configuration and operation information of each job node in detail.

[0124] In terms of data loss rate, the improved algorithm of the present invention reduces the data loss rate to 1%, compared with 12% (logging and analysis algorithm), 8% (database-level data tracing algorithm) and 10% (ETL-based tracing algorithm) of other algorithms. This significant improvement is mainly due to the real-time monitoring and unique identifier generation technology in the present invention, which ensures the integrity and security of each data record throughout the entire flow process.

[0125] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A method for tracing the source of full-link data flow on a visual data platform, characterized in that: include: Monitor the submission events of the job scheduling process, parse the input and output configuration information of each job node, convert it into a data flow model and save it in the traceability database; Perform static analysis on node scripts and codes, save the analysis results in the traceability database, and analyze the data flow when the job scheduling process is running; Generate a unique identifier for each data record, record relevant information during the data flow, and store the information in the traceability database; The data flow relationship is written into the graph database Neo4j through Kafka asynchronous messages, and the association relationship between the data record ID and the flow information in the traceability database is established. Users can perform data traceability queries through a visual interface.

2. The method for tracing the source of full-link data flow on a visualization data platform according to claim 1, characterized in that: The monitoring of the job scheduling process submission event includes configuring an event listener in the job scheduling system to capture the job submission event; Use a custom event handler to listen for submission events and extract the job ID, job name, and scheduling time metadata when the event is captured; The captured metadata is stored in the database, and the job configuration file is parsed using the SQL database storage job metadata table to extract the input and output configuration information of each job node.

3. The method for tracing the source of full-link data flow on a visualization data platform according to claim 1, characterized in that: The static analysis includes establishing a data flow model, assuming that each operation node in the code is , whose input data source is , the output data source is: ； The type of operation performed by each node is represented as ; Build a directed graph model and represent the data flow as a directed graph ,in: ； Represents a collection of nodes. It is a set of directed edges, indicating the data flow relationship: ； in, Represents a slave node To Node The data flow relationship edge; establish a data flow relationship, if the node The output data source is the node , then there is a directed edge : ； Establish a static data flow path, data from the node Flow to Node Path It is expressed as: ； Dynamic data operations and time-dependent node operations at runtime include, assuming that at time The operation nodes running are , whose input and output data sources are and ; Establish a dynamic data flow model, and the data flow relationship at runtime is expressed as a time series model , where each time point The data flow is expressed as: ； Dynamic path update: at time Generated data flow path It is expressed as: ； Combine static and dynamic models, update the static model, and transfer the dynamic data to the results Static data flow model Combined to form an updated data flow diagram The formula is: ； The data flow graph is fully represented. Combining the static and dynamic data flow relationships, the data flow path is finally represented as: ； From the starting point To the end node The complete data flow path includes all static and dynamic nodes and edges.

4. The method for tracing the source of full-link data flow on a visualization data platform according to claim 1 or 2, characterized in that: The generating of a unique identifier for each data record includes initializing input content including: , starting node , target node , edge weight function , maximum parallelism ; The output includes, from the starting node To the target node The shortest path and the least cost path; initialize the queue and priority queue , Indicates the node storing the current level. Represents a priority queue, according to edge weight Sort the nodes; put the starting node Put in queue Initialize its cost ; Initialize the node collection , used to store visited nodes, initially empty, for parallel breadth-first search; When the queue Not empty and the target node was not found , do the following: Process nodes at the same level in parallel and queue The nodes in the Divide into multiple subsets, each of which is assigned to a processing unit; process all neighboring nodes of the nodes in each subset in parallel; process each node Neighbor nodes ; For each unvisited neighbor node , calculate from the starting node arrive The cumulative cost is expressed as: ； in, Indicates from arrive The edge, represents the weight of the edge; like If it has not been visited or a lower-cost path is found, update The cost , and Put in priority queue middle: ， Select the next node from the priority queue and follow the priority queue The node cost in the sequence is small to large, and the next node is selected for access and put into the queue. , continue the next round of breadth-first search.

5. The method for tracing the source of full-link data flow on a visualization data platform according to claim 4, characterized in that: The next round of breadth-first search includes dynamically adjusting the maximum parallelism and the priority of the priority queue to cope with changes in system resources and current load; if the target node is found , the search is terminated and the output is arrive The shortest path; If all nodes are traversed , then the output is The shortest paths to all other nodes; If you need to output the path instead of just the path length, Back to the starting point , output the shortest path; Priority update of priority queue, priority queue The set of nodes sorted by cost in is: ； ； Every time from Take out the node with the smallest cost for the next traversal; Dynamically adjust the degree of parallelism, parallelism The adjustment is based on the current system load and remaining resources Decide: ， in, Indicates the maximum degree of parallelism allowed by the system. Indicates currently available resources. Indicates the current load.

6. The method for tracing the source of full-link data flow on a visualization data platform according to any one of claims 1, 2 or 3, characterized in that: The Kafka asynchronous message includes generating a data flow message, and generating a data flow message containing key information of data record ID, operation type, and timestamp at each node of data processing; each message is represented as a triple: ， in, A unique identifier representing a data record; Indicates the operation type; Indicates timestamp; Kafka message sending, sends data flow message to Kafka topic to realize asynchronous transmission of message; assume that each data flow event generates a message , then the topic of the Kafka message queue Represented as a sequence of messages: ， in, Indicates the first messages; the messages are arranged in chronological order, that is, for any ,have .

7. The method for tracing the source of full-link data flow on a visualization data platform as described in any one of claims 1, 3, and 5, characterized in that: The graph database Neo4j includes configuring Kafka consumers to consume data flow messages from the Kafka queue in real time; the consumer extracts messages from the queue , and parse out the ID, operation type and timestamp, and write the message to the Neo4j graph database: In Neo4j, based on the data in the message record Create and update nodes; depending on the operation type and timestamp Create or update the relationship between nodes in Neo4j to form a data flow path; Each data record Corresponds to a node in Neo4j , the data node formula is expressed as: ， Relational modeling of data operations, assuming two data nodes and There is a data flow relationship, represented by a directed edge : ； in, Indicates the time when the operation occurred; Kafka message The information in will be mapped to the Neo4j graph database, and the nodes and relationships will be updated. The formula is: ， Each message Corresponding to one or more such edge relations .

8. The method for tracing the source of full-link data flow on a visualization data platform according to claim 7, characterized in that: The data record is established The relationship with the information flowing in the traceability database includes building a Neo4j graph model and visually modeling nodes and relationships. In Neo4j, nodes are used to represent each data record. , use edges to represent the operations and order of data flow; through the construction of the graph, establish a complete data flow path, so that the flow history of each node can be traced by traversing the edges in the graph; Design By node collection and edge set The flow history formula of the node structure of the Neo4j graph is expressed as: ， ， The integration of Kafka message queue and Neo4j graph database realizes asynchronous writing and storage of data flow information. Users can easily query and track data flow paths through a visual interface.

9. A visual data platform full-link data flow traceability system, characterized by: include, An event monitoring module (100) monitors job submission events in the job scheduler in real time, captures job metadata, and stores the extracted job metadata in the traceability database Neo4j of the graph database module (300); The data flow model module (200) parses the input and output configuration information of each job node, builds a data flow model, and performs static analysis on the scripts and codes of the job nodes; Generate a unique identifier for each data record and record data flow information; The graph database module (300) builds the graph database Neo4j, and users can perform data traceability query through a graphical interface and view the full-link information of data flow; The integration and visualization query module (400) asynchronously sends the data flow information to the graph database module (300) through the Kafka message queue, and establishes a graph model of the data nodes and their flow relationships in the graph database Neo4j.

10. A computer device comprising: Memory and processor; The memory stores a computer program, which is characterized in that: when the processor executes the computer program, it implements the steps of the full-link data flow tracing method of the visual data platform.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the full-link data flow tracing method of the visual data platform are implemented.

Images