Data full-link tracking analysis method and device, electronic equipment and storage medium

Abstract

This invention discloses a data end-to-end tracking and analysis method, apparatus, electronic device, and storage medium, which can be widely applied in the field of big data processing technology. The invention records a first trajectory of multi-source collected data, a second trajectory of real-time calculated data, and a third trajectory of the stored results. Based on the first trajectory, it generates first node data in a graph database; based on the third trajectory, it generates second node data in the same graph database. Then, it calculates a first correlation between the first and second node data, thereby obtaining the degree of correlation between each node's data. Finally, it generates an end-to-end graph based on the first node data, second node data, second trajectory, and first correlation. The end-to-end graph is then used to analyze and display the entire data chain, making the displayed operating status more consistent with the actual operating status, and thus enabling timely detection of network operation problems.

Description

Technical Field

[0001] This invention relates to the field of big data processing technology, and in particular to a data end-to-end tracking and analysis method, apparatus, electronic device, and storage medium. Background Technology

[0002] In related technologies, the real-time data collection and computation of big data platforms typically involves multiple stages. An anomaly in any stage can lead to task failure. During analysis, if only the status of one stage is analyzed, it is difficult to detect runtime problems, which may result in discrepancies between the observed runtime status and the actual runtime situation. Summary of the Invention

[0003] This invention aims to at least solve one of the technical problems existing in the prior art. To this end, this invention proposes a data end-to-end tracking and analysis method, apparatus, electronic device, and storage medium, which can reduce the probability of discrepancies between the observed operating state and the actual operating state.

[0004] On one hand, embodiments of the present invention provide a data end-to-end tracing and analysis method, including the following steps:

[0005] Record the first trajectory of multi-source collected data, and generate the first node data of the graph database based on the first trajectory;

[0006] Record the second trajectory of real-time computational data;

[0007] The third trajectory of the data entry result is recorded. After generating the second node data of the graph database based on the third trajectory, the first association relationship between the first node data and the second node data is calculated.

[0008] A full-link diagram is generated based on the first node data, the second node data, the second trajectory, and the first correlation relationship;

[0009] The entire data flow is analyzed and displayed based on the aforementioned end-to-end diagram.

[0010] In some embodiments, the first trajectory for recording multi-source acquired data, and the generation of first node data for a graph database based on the first trajectory, include:

[0011] The multi-source collected data is written to a message middleware, which is used to store the multi-source collected data and the first trajectory of the multi-source collected data.

[0012] Multi-source data collected from the first trajectory is read from the message middleware in an asynchronous multi-threaded manner, and the read multi-source data is used as the first node data of the graph database.

[0013] In some embodiments, reading multi-source collected data on the first trajectory from the message middleware via asynchronous multi-threading includes:

[0014] Real-time monitoring of messages from the message middleware;

[0015] When a message that meets the first preset requirement is detected in the message middleware, the message that meets the first preset requirement and the corresponding displacement point are obtained; wherein, multiple displacement points constitute the first trajectory;

[0016] Based on the displacement point, messages that meet the first preset requirements are read using an asynchronous multi-threaded approach.

[0017] In some embodiments, the second trajectory for recording real-time computational data includes:

[0018] Obtain the source data from the real-time computing data;

[0019] The source data is aggregated to obtain aggregated data;

[0020] Calculate the second association between the source data and the aggregated data;

[0021] The second trajectory of the real-time computing data is generated based on the second association.

[0022] In some embodiments, after the step of generating a full-link graph based on the first node data, the second node data, the second trajectory, and the first association relationship, the method further includes:

[0023] Generate a globally unique identifier for each data item in the entire link graph;

[0024] The end-to-end graph is segmented and stored according to the globally unique identifier, and the indexing method is set according to the data type in the end-to-end graph.

[0025] In some embodiments, prior to the step of analyzing and displaying the entire data link based on the end-to-end diagram, the method further includes:

[0026] Set a collection time label for each collected data point according to the time caliber.

[0027] A calculation process diagram is generated based on the processing sequence number corresponding to the collected data;

[0028] The abnormal dataset is determined based on the collection time stamp and the calculation process diagram.

[0029] In some embodiments, the step of analyzing and displaying the entire data link based on the end-to-end diagram includes:

[0030] The target data is obtained by querying the full-link graph based on the query data;

[0031] After obtaining upstream and downstream data from the target data in the full-link diagram, a third association relationship between the upstream and downstream data and the target data is obtained.

[0032] A data tracking and analysis graph is generated based on the upstream and downstream data, the third correlation, and the abnormal dataset.

[0033] The data tracking and analysis chart is then displayed.

[0034] On the other hand, embodiments of the present invention provide a data end-to-end tracking and analysis device, comprising:

[0035] The first module is used to record the first trajectory of multi-source collected data and generate the first node data of the graph database based on the first trajectory.

[0036] The second module is used to record the second trajectory of real-time computational data;

[0037] The third module is used to record the third trajectory of the data entry result, generate the second node data of the graph database based on the third trajectory, and calculate the first association relationship between the first node data and the second node data.

[0038] The fourth module is used to generate a full-link diagram based on the first node data, the second node data, the second trajectory, and the first association relationship;

[0039] The fifth module is used to analyze and display the entire data link based on the full-link diagram.

[0040] On the other hand, embodiments of the present invention provide an electronic device, including:

[0041] At least one memory for storing programs;

[0042] At least one processor is used to load the program to execute the data end-to-end tracing and analysis method.

[0043] On the other hand, embodiments of the present invention provide a computer storage medium storing a computer-executable program, which, when executed by a processor, is used to implement the data end-to-end tracing and analysis method.

[0044] The data end-to-end tracing and analysis method provided in this invention has the following beneficial effects:

[0045] This embodiment records the first trajectory of multi-source collected data, the second trajectory of real-time calculated data, and the third trajectory of the data stored in the database. Based on the first trajectory, it generates the first node data of the graph database, and based on the third trajectory, it generates the second node data of the graph database. Then, it calculates the first association relationship between the first and second node data, thereby obtaining the degree of association between each node data. Finally, it generates a full-link graph based on the first node data, the second node data, the second trajectory, and the first association relationship. The full-link graph is then used to analyze and display the entire data link, making the displayed operating status more consistent with the actual operating status, and thus enabling timely detection of network operation problems.

[0046] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0047] The present invention will be further described below with reference to the accompanying drawings and embodiments, wherein:

[0048] Figure 1 This is a flowchart of a data end-to-end tracing and analysis method according to an embodiment of the present invention;

[0049] Figure 2 This is a schematic diagram of a data calculation process according to an embodiment of the present invention;

[0050] Figure 3 This is a schematic diagram of a data end-to-end link according to an embodiment of the present invention;

[0051] Figure 4 This is a business process diagram of a real-time computing task according to an embodiment of the present invention;

[0052] Figure 5 This is a schematic diagram illustrating an application scenario of a data end-to-end tracing and analysis method according to an embodiment of the present invention;

[0053] Figure 6 This is a schematic diagram illustrating a data collection and recording implementation according to an embodiment of the present invention;

[0054] Figure 7 This is a schematic diagram of an asynchronous message processing embodiment of the present invention;

[0055] Figure 8 This is a schematic diagram of a data trajectory recording of a calculation point according to an embodiment of the present invention;

[0056] Figure 9 This is a schematic diagram of the recording trigger point of a calculation process according to an embodiment of the present invention;

[0057] Figure 10 This is a schematic diagram illustrating an efficient data writing process according to an embodiment of the present invention;

[0058] Figure 11 This is a schematic diagram of an index construction according to an embodiment of the present invention;

[0059] Figure 12 This is a schematic diagram illustrating the discovery of a problem dataset according to an embodiment of the present invention;

[0060] Figure 13 This is a schematic diagram of a data end-to-end tracing and query process according to an embodiment of the present invention. Detailed Implementation

[0061] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0062] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0063] In the description of this invention, unless otherwise explicitly defined, terms such as "set up," "install," and "connect" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0064] In the description of this invention, the terms "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0065] In related technologies, the real-time data collection and computation of big data platforms typically involves multiple stages. When problems arise, it is often difficult to locate and analyze the issue at any single stage. It requires tracing the entire data computation chain from the data source, message middleware, computation program, to the target database to find the cause. Furthermore, because the data flows in real time, it is difficult to verify data consistency over a fixed time period. Current methods for troubleshooting mainly suffer from the following drawbacks:

[0066] First, judging the running status of real-time computing tasks is complex. Since computing tasks consist of multiple stages, any abnormality in any stage will cause the task to fail. If only the status of the computing stage is obtained, the observed running status will be inconsistent with the actual running situation, making it impossible to detect runtime problems in a timely manner.

[0067] Secondly, data loss is difficult to detect. Because real-time computing involves real-time data cleaning and processing, it is difficult to verify the consistency between the data after it is entered into the database and the data collected at that time. It is impossible to determine whether the inconsistency is due to filtering operations or data loss.

[0068] Thirdly, data auditing is difficult. Because real-time data is streamed, there is always a time difference between the time the data is read and the time it is entered into the database. This makes it very difficult to compare the total amount of data for a given time period. It is impossible to confirm the accuracy of the data by whether the total amount is consistent, nor is it possible to accurately assess whether there is any data loss.

[0069] Fourthly, when performing real-time computation on multi-source data, it is difficult to map the entire data chain. Because the computation process involves data aggregation and merging, without tracking the computation process, it is impossible to fully map the data chain and determine whether data loss is due to computational filtering or an anomaly.

[0070] Fifth, data analysis is difficult. In the process of computing with large amounts of data, high concurrency, and multiple stages, some errors usually occur in certain small stages. These errors will not cause the computing task to stop, but they will cause some data to be lost. Without full-chain data tracking and analysis, it is impossible to quickly locate which stage of the real-time process has a problem, let alone provide root cause analysis of the data loss problem.

[0071] As described above, data acquisition, message storage, data computation, and data storage occur at multiple sources. The data storage end utilizes various media, such as FTP and HDFS for file storage, and Hive and MySQL for database storage. The more stages involved, the greater the probability of problems and the higher the chance of data loss. Therefore, tracking the data flow is crucial to quickly analyze the entire data chain and identify the root cause of any issues that arise during data processing, thus addressing the immediate problem. For example, consider the following scenario:

[0072] Step 1: Data collection of 4 records:

[0073] Data collected A: {user:a1,age:23,degree:bachelor,salary:3000,city:hn}

[0074] Data collected B: {user:b1,age:32,degree:master,salary:5000:city:hn}

[0075] Data collected C: {user:c1,age:22,degree:bachelor,salary:4000:city:hn}

[0076] Data collected D: {user:d1,age:24,degree:bachelor,salary:3000:city:gz}

[0077] Step 2: Filtering the data based on age < 30 yields:

[0078] Data collected A: {user:a1,age:23,degree:bachelor,salary:3000,city:hn}

[0079] Data collected C: {user:c1,age:22,degree:bachelor,salary:4000:city:hn}

[0080] Data collected D: {user:d1,age:24,degree:bachelor,salary:3000:city:gz}

[0081] Step 3: Calculation, avg(salary), group by city yields:

[0082] Data: {city:hn,avg:3500},{city:gz,avg:3000}

[0083] Step 4: After data entry (data loss), the following can be obtained:

[0084] Data: {city:hn,avg:3500}

[0085] The above examples illustrate the existence of filtered data, merged data, and missing data. Without end-to-end data tracing and analysis, the current methods for identifying data issues primarily include:

[0086] First, by examining the implementation process of the business logic code and checking the key information "age>=30" in the logs, we found the collected data B:{user:b1,age:32,degree:master,salary:5000:city:hn}, confirming that this data is filtered data.

[0087] Second, after reviewing the log analysis and based on the calculated group conditions, it was found that "Collected Data A" and "Collected Data C" were merged, confirming that these two data points participated in the aggregation calculation and that no data was lost.

[0088] Third, upon checking the logs, it was found that two data entries should have been merged, but only one data entry was stored in the database. Log analysis confirmed that the data "{city:gz,avg:3000}" was missing.

[0089] Fourth, log analysis requires prior recording of logs during the business implementation process. Without detailed log records, it is impossible to analyze the data trajectory through logs, and real-time data verification cannot be carried out. At the same time, log analysis also takes a long time, which is not conducive to quickly realizing data verification for real-time tasks and cannot effectively find the root cause of data problems.

[0090] In summary, during the data acquisition, message storage, computation, and storage stages of a real-time computing task, the data undergoes corresponding changes. Data may be lost, filtered, or merged during computation. For each data occurrence, clear evidence is needed to prove that no data loss occurred during the business logic computation process. This also allows for the rapid identification and analysis of the data computation process, facilitating quick root cause analysis of data problems.

[0091] Based on this, refer to Figure 1 This invention provides a data end-to-end tracing and analysis method, which can be applied to processors, servers, or the cloud corresponding to a big data processing platform. The big data processing platform can be used to process data generated at various stages of network communication, as well as to process interactive data between multiple devices.

[0092] It is understood that the method of this embodiment of the invention, in its application, includes, but is not limited to, the following steps:

[0093] Step S110: Record the first trajectory of the multi-source collected data, and generate the first node data of the graph database based on the first trajectory;

[0094] In this embodiment, multi-source data acquisition refers to data acquisition through multiple methods or at multiple nodes. This embodiment can write multi-source data to a message middleware, then read the multi-source data on the first trajectory from the message middleware using an asynchronous multi-threaded approach, and use the read multi-source data as the first node data of the graph database. The message middleware stores the multi-source data and the first trajectory of the multi-source data. Furthermore, when reading the multi-source data on the first trajectory from the message middleware using an asynchronous multi-threaded approach, messages in the message middleware can be monitored in real time. When a message meeting a first preset requirement is detected entering the message middleware, the message meeting the first preset requirement and its corresponding displacement point are obtained. Then, based on the displacement point, the message meeting the first preset requirement is read asynchronously using a multi-threaded approach. Multiple displacement points constitute the first trajectory. A message meeting the first preset requirement refers to a message stored in the message middleware at the current node.

[0095] Understandably, this embodiment employs a decoupled model of data acquisition and real-time computation to improve the throughput of multi-source data acquisition. The acquired data from multiple sources is first stored in a message middleware, and then messages are read from the message middleware for computation. The message middleware ensures that the stored data can be repeatedly computed within a certain timeframe, providing strong protection for data integrity.

[0096] While the data is being written to the message middleware, a copy is also written to Dgraph (a graph database) via an asynchronous mechanism. The graph database is used to store the trajectory of the collected data. In this embodiment, since this stage represents the initial form of the data, it is not necessary to calculate the relationships between the data; only the node data in the graph database needs to be stored. Each piece of data is stored in a node data format, which is: p(table name:English name{primary key field:value, field name:value,...}).

[0097] Because the amount of data recorded during the acquisition phase is large, it is necessary to ensure that the normal processing of data is not affected while recording the data trajectory. Therefore, this embodiment adopts an asynchronous data writing mechanism. Taking advantage of the characteristic that message middleware can be consumed repeatedly, a specific consumer group is configured, such as g_pc_graph. Data is read asynchronously through multiple threads through this consumer group, and the data is stored as graph data node data in the format of graph data.

[0098] Step S120: Record the second trajectory of the real-time calculated data;

[0099] In this embodiment, the source data in the real-time computing data can be obtained first, then aggregated to obtain aggregated data. Next, the correlation between the source data and the aggregated data is calculated as a second correlation, and a second trajectory of the real-time computing data is generated based on this second correlation. It is understood that when data flows through this point, computational operations such as data merging, filtering, and aggregation occur, creating certain correlations between the data. In this embodiment, aggregation includes data addition, subtraction, averaging, union, intersection, and other processing methods. Consider the following data as an example:

[0100] Data A: {user:a1,age:23,degree:bachelor,salary:3000,city:hn}

[0101] Data B: {user:b1,age:25,degree:master,salary:5000:city:hn}

[0102] like Figure 2 As shown, performing data aggregation calculations on data A and data B using `avg(age)group by city` (i.e., averaging the ages in data A and data B and grouping them by city type) will generate new data C: `{age:24, city:hn}`. During the data calculation, by acquiring the data A and data B involved in the calculation and extracting the `avg` and `group by` operations, the final data C is extracted as: vertex p{aggregate:tt{age:24, city:hn}}. Then, using a fast search algorithm, data A, data B, and data C are located. Based on the data flow of the operators, associations are established between data A and data C, and between data B and data C, with the value of "average aggregation calculation". By establishing the edge relationships between points, the data trajectory of the calculation process is recorded as a second trajectory.

[0103] Step S130: Record the third trajectory of the data entry result, generate the second node data of the graph database based on the third trajectory, and calculate the first association relationship between the first node data and the second node data;

[0104] In this embodiment, after a computation task is completed, the data is written to disk. When the data is written to disk, the final node data of the entire link is also formed, recording the vertex data of the graph database: p(target table name: English name {primary key field: value, field name: value,...}, and then recording the association relationship between the vertex and the upstream node (first association relationship), that is, the edge association relationship v.

[0105] Step S140: Generate a full-link diagram based on the first node data, the second node data, the second trajectory, and the first correlation relationship;

[0106] In this embodiment, after obtaining the full-link graph, the capabilities of a distributed graph database can be used to quickly locate data based on data attributes, primary keys, and type. The full-link graph of the data can be displayed, clearly showing whether the data has been merged or added to the database. It also provides a way to quickly retrieve all records that have not been added to the database within a certain period. For example, using a match query to retrieve data, the query statement is as follows:

[0107] match(x:user) where x.username = 'a1' return x; / / Query and locate data based on the specified conditions

[0108] match(x)-[r:aggregator]-(y)where r.association = 'aggregation' return x,r,y; / / Query link data based on the data and the association between the data.

[0109] In this embodiment, by performing end-to-end data analysis, a query statement can be used to retrieve data that has not reached the final node within a specified time period, thereby enabling the display of a problem dataset. The query statement is shown below:

[0110] match path = ((start{n: collected data}) - [r:all] - (!end{e: inserted data})) / / Query all data from the collected data to the data that was not inserted into the database, where !end represents data that has not reached the last node.

[0111] with start, end, path

[0112] where start_time=>st and end_time<=et and task_id=?

[0113] return collect(start,r,end);

[0114] Step S150: The root end-link diagram analyzes and displays the entire data link.

[0115] In this embodiment, a full-link tracing analysis result graph can be generated based on the correlation of the recorded graph data. Simultaneously, for data links with large data volumes, an asynchronous loading method is used, loading the lower-level data of the link layer by layer. Based on the data link analysis results, the data flow process can be discovered, facilitating data analysis. The generated analysis results are as follows: Figure 3 The data end-to-end diagram shown below illustrates the process from... Figure 3 The link analysis display clearly shows that "Data D" is the filtered data.

[0116] For example, taking a customer relationship management system as an example, the system generates a large amount of order processing data, order completion data, and product instance data every day. A wide table is generated from this real-time data stream, including product instance, order completion date, and product type information. To perform real-time calculations for this business, such as... Figure 4 As shown, the general logic is as follows: filter, complete, and merge data from the source data streams "Accepted Order Data Stream", "Order Completion Data Stream", "Product Instance Data Stream" and dimension table data "Product Type Data" to form "Product Instance Data", and then store the product instance data into the target database.

[0117] In the above business scenarios, there are over a million order processing messages, completion messages, and product instance messages every day. During real-time processing and calculation, various anomalies such as data loss can easily occur. To track data during the calculation and synchronization process and quickly analyze the entire data flow, it is necessary to record the data trajectory. The implementation steps are as follows:

[0118] Step 1: Obtain source data, also known as collected data. By adding a monitor during the collection process, the data stream collected into Kafka can be monitored. The monitor can then be used to parse the data and store it in the graph data database.

[0119] Step 2, Data Calculation Stage: Add interceptors to the filtering and merging operators to monitor in real time.

[0120] (1) The data before the operator, i.e. the input data, is entered into the graph data asynchronously.

[0121] (2) Record the filtered data into the graph data, and at the same time record the relationship between the input data and the filtered data as “filtering”.

[0122] (3) Record the data involved in the calculation and merge into the graph data, and at the same time record the relationship between the input data and the data involved in the calculation as "merge".

[0123] (4) Record the data calculated by the operator into the graph data.

[0124] Step 3, Data Entry: By adding interceptors to the data entry operators, input data is intercepted and recorded, output data is intercepted and recorded into the graph data, and the relationship between the input and data entry is recorded as "data entry".

[0125] Step 4: Through query statements, the entire data chain process can be easily retrieved in seconds from tens of millions of data points, and problematic data that has not been properly entered into the database can be analyzed in seconds. The following statement can realize the entire chain display of data from collection to database entry and whether filtering is performed within seconds.

[0126] match(prod: product instance) where prod.product_id = '1000100' return prod; / / Retrieve data

[0127] `match(prod)-[r:filter / merge]-(y)where r.relation='filter'or'return prod,r,y;` / / Query data relationships.

[0128] match path = ((start{n:collect data})-[r:all]-(!end{e:input data})) / / Query problem data.

[0129] Based on the above, this embodiment addresses the issue of data flow during real-time task computation by tracking and recording data across multiple stages to form a complete data trajectory. This trajectory allows for full-link data analysis, identifying problems in the data execution and computation process. This facilitates analysis of data usage and helps quickly pinpoint data loss or anomalies. Furthermore, due to the large volume and high concurrency of real-time data flow, it's crucial to record data trajectories quickly without impacting computational tasks, supporting efficient real-time writing for high-concurrency tasks. The large data volume also presents a significant challenge to query and display performance. Achieving full-link lineage display for hundreds of millions of data points within seconds effectively improves the efficiency of data analysis and problem identification.

[0130] In some embodiments, the data end-to-end tracing and analysis method of this embodiment is applied to, for example... Figure 5 In the scenario shown, the following steps are included, but are not limited to:

[0131] Step S510: Track recording of collected data.

[0132] In this embodiment, the real-time computing task collects data such as logs, stores it in Kafka (a message queue), and consumes the collected data through an asynchronous, multi-threaded program. During consumption, a specific consumer group, such as `g_pc_graph`, is configured, and the data is then stored in a graph database. Specifically, the asynchronous, multi-threaded program implementation logic is as follows: Figure 6 As shown, including but not limited to the following steps:

[0133] Step S610: Use Flink CDC technology to collect log data from the database into the Kafka message middleware, and distribute it by hash based on the table name to ensure that the data of a table is synchronized to only one partition. The calculation method is as follows:

[0134] Partition number = hash(table name)%total number of partitions

[0135] Step S620: The acquisition program uses asynchronous multi-threading to correspond to the topic partitions. The implementation method is as follows:

[0136] (1) Listen to messages from the message middleware in real time through the Flink distributed computing framework.

[0137] OnAynsConsumerListener{

[0138] / / Consumption will be triggered when there is a new message.

[0139] OnListener(Message msg){

[0140] / / This section enables the implementation of multi-threaded asynchronous processing logic.

[0141] }

[0142] }

[0143] (2) When a new message comes in, the listening program will be triggered, and the program will obtain the message and offset point (offsetid).

[0144] (3) Enable asynchronous message processing to prevent message processing from affecting the data acquisition process. The asynchronous message process is as follows: Figure 7 As shown, a 2^8 circular queue is generated. Writing data retains a pointer, reading data retains the pointer, and reading and writing are isolated. Reading is done according to the ring's sequential position (read position = read position % 2^8), and the data is cleared afterward. Writing data first finds a position on the ring (write position = write position % 2^8), and then writes the data to the area corresponding to that position. Here, "%" refers to the operation of placing other variables into a specific position in the string to generate a new string.

[0145] Step S630: Track recording and storage.

[0146] After the message has been processed in the previous step, and then converted to a new format, the vertex data of the graph database can be directly stored without recording the relationships between the data, in the following format:

[0147] create(dataobjectname1:collecteddata{primarykey:1,attributes:xx,...} / / Creates point data and saves it to the graph database.

[0148] Step S520: Real-time calculation and recording of data trajectory.

[0149] Because real-time computing tasks involve data filtering, aggregation, and other computational operations, data may be merged or lost along the way. Therefore, it's crucial to completely record the data computation process to ensure that each data point is traceable and viewable. The recording of these steps is as follows: Figure 8 As shown, including but not limited to the following steps:

[0150] Step S810: Add a data tracking entry point at the calculation point. When the data flow reaches this operator, the action of recording the data calculation trajectory is triggered simultaneously. Using the concept of AOP (Aspect-Oriented Programming), the recording action is triggered at the time of data input, calculation, and data output, respectively. Figure 9 As shown, including but not limited to the following steps:

[0151] Step S910: The aspect mechanism is used to configure aspects for all operators in the following way: pointcut(*action), which executes the three methods of aspect configuration for all operators ending with action.

[0152] Step S920: Design the implementation of the aspect, add the beforeAction method to trigger the calculation point, which can record the input data; add the actioning method during the calculation, which can record the data involved in the calculation; add the afterAction method after the calculation, which can record the output data.

[0153] Step S820: Modify the implementation method name of the operator. Add "action" to the suffix of the operator method name that needs to be tracked, such as sumAction(...), which indicates the summation operator. To ensure that the front-end usage does not change, modify it to: sum(...){sumAction(...)}. Call sumAction in the original sum method, and then implement the content of the original sum in sumAction.

[0154] Step S830: Record the data before entering the calculation point using the beforeAction method. The recorded data mainly includes: the data body (table, field, primary key metadata and corresponding values), and the association relationship with the parent operator. The implementation method is as follows:

[0155] Step S831: Create input data: create point(data object name 1: input data 1 {primary key: 1, attribute: xx, ...}) / / Create point input data and save it.

[0156] Step S832: Create the association between the input data and the parent operator, as follows:

[0157] create edge(collected data1:rela->collected relationship:output data1) / / Creates the association between the upper-level collected data and the downstream input data, and indicates the type of association.

[0158] Step S840: Record the data in the calculation using the actioning method and label the data involved in the calculation.

[0159] The implementation process of this embodiment includes, but is not limited to, the following steps:

[0160] Step S841, create point(data object name 1: compute data 1 {primary key: 1, attribute: xx..., operator name: sum}) / / Create vertex data and record the corresponding operator;

[0161] Step S842: Create operators and generate a unique identifier for each operator participating in the calculation:

[0162] create point(operator name + unique identifier: compute operator { primary key: identifier, name: xx, compute time: ..., other attributes}) / / Creates a node for the operator and saves it;

[0163] Step S843: Create the association between the operator and the data nodes participating in the computation:

[0164] create vertex(computation data1:rela->computation operator:operator unique identifier) ​​ / / Creates the association between the data involved in the computation and the operator.

[0165] Step S850: Record the calculated data using the afterAction method and save the calculated data to the graph data. The implementation process of this embodiment includes, but is not limited to, the following steps:

[0166] Step S851: create point(data object name 1: output data 1 {primary key: 1, attributes: xx..., record time: xx}) / / Create output data and save it;

[0167] Step S852: Record the relationship between the output and the upstream operator:

[0168] create edge(output data1:rela->calculation operator:operator unique identifier) ​​ / / Creates the association between the output data and the operator.

[0169] Step S860: Data Write Performance Assurance. Due to the high performance and fast response requirements of real-time computing, to ensure that the data recording process does not affect performance, during frequent writes and high concurrency, such as... Figure 10 As shown, the mechanism for adding an asynchronous queue includes, but is not limited to, the following steps:

[0170] Configure the queue. Configure the queue length according to the amount of data required, such as queue:2 (queue length), queue:10 (queue length).

[0171] The data written to the queue includes not only the data itself, but also the statements to be executed. This allows for direct execution during reading, reducing the time spent on interpretation and assembly.

[0172] When processing queued data, the program will automatically increase the concurrency level or configure the maximum and minimum concurrency based on resource availability and the amount of data accumulated in the queue, ensuring efficient writing to the graph database.

[0173] The read data is cached appropriately on the writing end, turning the original single-row write operation into a batch operation to improve writing efficiency.

[0174] Step S530: Record the data trajectory of the data entering the database.

[0175] In this embodiment, data entry is the final step in the real-time computing task, specifically the sink operator. An interceptor called `pointcut(*sink)` needs to be added. When data flows through this stage, only an `afterSink` interceptor needs to be added to record the already entered data. The operation steps include, but are not limited to, the following:

[0176] Step S531: Configure to add interception of sink operations, add the listening configuration pointcut(*Sink), which can listen to all methods with the suffix Sink.

[0177] Step S532: Modify the data entry method. To ensure that the data entry logic and upstream and downstream calls remain unchanged, the original method is retained. Add a method ending with xxxSink and implement the data entry logic in xxxSink, for example, write(...){writeSink(...)}, and implement the write logic in writeSink.

[0178] Step S533: Data writing mainly involves writing the data of the points and edges. The implementation method is as follows:

[0179] create point(data: inbound data {primary key: identifier, name: xx, inbound time: ..., other attributes,}) / / Creates the data for the inbound point;

[0180] create edge(upstream data1:rela->sink:current data1) / / Creates the association between the data entering the database and the result data output from the upstream.

[0181] Step S534: Improve write performance by using the asynchronous queue mechanism in step S860 of step S520 to ensure write performance without affecting the performance of the normal real-time computing process.

[0182] Step S540: Enable fast data query for large amounts of data.

[0183] To achieve fast data retrieval under large data volumes, this embodiment makes appropriate adjustments to the graph data storage. Specifically, it generates a globally unique identifier (globally unique ID) for each data item in the entire graph, then segments the entire graph based on the globally unique identifier, and sets the indexing method according to the data types in the graph. It is understood that the implementation process of this embodiment includes, but is not limited to, the following steps:

[0184] Step S541: Generate a uid for each data. Since all data is stored according to edges and nodes, it forms a directed graph. The graph assigns a globally unique id to each data object, called uid. UID is a 64-bit unsigned integer that monotonically increases from 1.

[0185] Step S542: To quickly query data relationships, the relationships are stored in shards based on the edge data. Identical edges form a shard, and each shard is compressed into a key-value pair. The key is the edge's uid value, and the value is a data structure called a set of points. When an edge corresponds to many relationships, its sharding is automatically split appropriately, primarily splitting the key value into the current edge's uid_n (representing a number incrementing from 1), as shown below:

[0186] key = <edge UID> value <vertex UID, vertex UID, vertex UID, ...>

[0187] key1 = <edge UID_1> value <vertex UID, vertex UID, vertex UID, ...>

[0188] key2 = <edge UID_n> value <vertex UID, vertex UID, vertex UID, ...>

[0189] Step S543: The basis for splitting is that the value in the key-value pair is optimized by integer compression, with each value consisting of 256 uids.

[0190] Step S544: Implement fast relational lookup. The improved storage method is highly beneficial for joining and traversal; traversing one edge only requires one key-value query. For example, to find all associated data of X, simply use <edge X's UID> and <edge X's UID_n> as keys to query and obtain the set of point data, which contains the uids of all associated data. To find common associated data between X and Y, simply query the set of points with <edge X's UID> and <edge X's UID_n>, <edge Y's UID> and <edge Y's UID_n>, then find their intersection. To determine whether the query for UID_n should terminate, simply check if the value can be obtained starting from 1. If the value is empty, the query can terminate.

[0191] Step S545: To facilitate and quickly query large amounts of data, corresponding indexes are established for different data types. For example, for the string type, regular expression, fulltext, term, exact, and hash indexes are supported; for the datetime type, indexes by year, month, day, and hour are supported; for the geo type, indexes by nearby and within are supported. Figure 11 As shown, the steps for building an index are as follows:

[0192] (1) Indexes are stored in key-value pairs, just like data.

[0193] (2) The key of the key index is <edge, token>, and the token is the tokenizer of the index, which is obtained from the value. For example, the token generated by the hash index is the hash value calculated by the hash function.

[0194] (3) One or more tokenizers can be created to generate tokens for the same data. When updating, first delete the corresponding uid from the values ​​of the old tokens, and then add the uid to the values ​​of the newly generated tokens.

[0195] from Figure 11 As we can see, since a fulltext index is created for key1, the fulltext tokenizer is called. From the value "running fast" of key1, two tokens, run and fast, are obtained. These tokens, along with key1, form two keys, and then uid0xa is added to their respective values. In this way, the index of a value is transformed into a key-value pair stored in the backend.

[0196] Step S546: Through the above modifications, a large amount of data is split and stored in a graph structure. Data can be quickly found through indexes, nodes, etc. Related data can be quickly located based on key values, realizing the rapid construction of relationships, reducing relational or NoSQL relationship queries, and improving the performance of fast data query under large data volumes.

[0197] Step S550: Automatic discovery and display of problem data.

[0198] This embodiment can set a collection time label for each collected data point based on the time metric. Then, a calculation process diagram is generated based on the processing sequence number corresponding to the collected data. Finally, abnormal datasets are identified based on the collection time label and the calculation process diagram. It is understood that since all data is stored in a graph database and the relationships between data are preserved, for data that was not ultimately entered into the database or did not participate in the calculation, a search algorithm can automatically discover data issues within a certain time period from the graph database. The time is based on a unified "collection time label." The operation steps are as follows: Figure 12 As shown, including but not limited to the following steps:

[0199] Step S551: Clarify the statistical time standard, using the creation time when the data is collected as the standard. All subsequent data will have the "collection time" added to it. Also, check if there is any abnormal data in the output of the collection operator.

[0200] Step S552: Since each operator has an input dataset, an output dataset, and a dataset involved in the calculation, list the calculation process diagrams of all intermediate operators according to the sequence number of the calculation process.

[0201] Step S553: ​​Except for the first data collection and the last data entry operator, a concurrent and queuing mechanism is used to perform statistical calculations on the problem data for each operator. The formula is: Problem Data = (Input Dataset - Datasets Participating in Calculation) / U = Output Abnormal Dataset. In this step, "U" refers to finding the union of the two sets of data.

[0202] Step S554: The formula for statistically analyzing the problem data of the last input operator is: Problem Data = Input Dataset - Input Dataset.

[0203] Step S555: Finally, merge the problem data calculated by each operator to form the final problem dataset of the calculation task, namely: the problem dataset of operator 1 U, the problem dataset of operator 2 U, ... U the problem dataset of the input operator.

[0204] Step S560: Data full-link tracing and analysis display.

[0205] This embodiment obtains target data by querying the full-link graph based on the query data; then, based on the target data, it retrieves upstream and downstream data from the full-link graph, and obtains the third association between the upstream and downstream data and the target data; finally, it generates a data tracing analysis graph based on the upstream and downstream data, the third association, and the abnormal dataset, and displays the data tracing analysis graph. The display process can be performed through a human-computer interaction interface. It is understood that during the computation and data synchronization of real-time tasks, the data computation process, input, output, and abnormal data are all recorded. To achieve full-link data tracing analysis, the data link can be traced downwards from any data point. The specific operation steps are as follows: Figure 13 As shown, including but not limited to the following steps:

[0206] Step S561: Input the data to be queried. You can use data type, primary key, and other attributes to query the desired data from the graph database. If there are multiple data entries, you can select multiple entries or select one entry.

[0207] Step S562: Selected data can be quickly queried for its associated upstream and downstream data, and the relationship between the data (third relationship) can be retrieved.

[0208] Step S563: Then, based on the upstream and downstream data, continue to query the relevant data and related data.

[0209] Step S564: Repeat step S563 until the downward data is identified as inbound data and the upward data is identified as collected data.

[0210] Step S565: Draw a data tracking analysis chart based on the queried data.

[0211] As can be seen from the above example process, the implementation process and effects of this embodiment can be summarized as follows:

[0212] First, by collecting large amounts of data from multiple sources, a lock-free asynchronous data processing framework is used to record the collected data trajectory and store it in a graph database.

[0213] Secondly, by using aspect-oriented programming, pre-processing, in-processing, and post-processing methods are added to each operator in real-time computation, and the incoming data, processing data, and completed data of each operator are tracked and recorded in the graph database in real time.

[0214] Thirdly, track and record each data inflow operator and outflow operator to record the correlation of the data processing and calculation process, forming a link diagram of the data flow.

[0215] Fourthly, record the data entry process and the relationship between the data entering the warehouse and the data entering the warehouse, as shown in the diagram.

[0216] Fifth, it provides the ability to quickly query and locate data, thereby enabling the rapid creation of data processing diagrams from large amounts of data, facilitating the quick discovery and location of data problems.

[0217] Sixth, when performing real-time computation on multi-source data, the recorded data trajectory can be used to quickly draw a process diagram of data aggregation, merging, filtering, and other operations, which facilitates quick location of data trends and helps in problem analysis.

[0218] This invention provides a method for... Figure 1 The corresponding data end-to-end tracing and analysis device includes:

[0219] The first module is used to record the first trajectory of multi-source collected data and generate the first node data of the graph database based on the first trajectory;

[0220] The second module is used to record the second trajectory of real-time computational data;

[0221] The third module is used to record the third trajectory of the data entering the database, and after generating the second node data of the graph database based on the third trajectory, calculate the first association relationship between the first node data and the second node data.

[0222] The fourth module is used to generate a full-link diagram based on the first node data, the second node data, the second trajectory, and the first correlation.

[0223] The fifth module is used to analyze and display the entire data chain based on the end-to-end diagram.

[0224] The content of the method embodiments of the present invention is applicable to the device embodiments. The specific functions implemented by the device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above methods.

[0225] This invention provides an electronic device, comprising:

[0226] At least one memory for storing programs;

[0227] At least one processor is used to load the program for execution. Figure 1 The data end-to-end tracing and analysis method is shown.

[0228] The content of the method embodiments of the present invention is applicable to the embodiments of the present electronic device. The specific functions implemented by the embodiments of the present electronic device are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above methods.

[0229] This invention provides a computer storage medium storing a computer-executable program, which, when executed by a processor, is used to implement... Figure 1 The data end-to-end tracing and analysis method is shown.

[0230] The content of the method embodiments of the present invention is applicable to the storage medium embodiments. The specific functions implemented by the storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above methods.

[0231] Furthermore, embodiments of the present invention also provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform... Figure 1 The data end-to-end tracing and analysis method is shown.

[0232] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments, and various changes can be made within the scope of knowledge possessed by those skilled in the art without departing from the spirit of the present invention. Furthermore, the embodiments of the present invention and the features thereof can be combined with each other unless otherwise specified.

Claims

1. A data end-to-end tracing and analysis method, characterized in that, Includes the following steps: Record the first trajectory of multi-source collected data, and generate the first node data of the graph database based on the first trajectory; Record the second trajectory of real-time computational data; The third trajectory of the data entry result is recorded. After generating the second node data of the graph database based on the third trajectory, the first association relationship between the first node data and the second node data is calculated. The first association relationship includes the association relationship between the node corresponding to the first node data and the node corresponding to the second node data. A full-link diagram is generated based on the first node data, the second node data, the second trajectory, and the first correlation relationship; The entire data flow is analyzed and displayed based on the aforementioned end-to-end diagram; The first trajectory for recording multi-source collected data, and the generation of first node data for the graph database based on the first trajectory, include: The multi-source collected data is written to a message middleware, which is used to store the multi-source collected data and the first trajectory of the multi-source collected data. The multi-source data collected on the first trajectory is read from the message middleware in an asynchronous multi-threaded manner, and the read multi-source data is used as the first node data of the graph database. The step of reading multi-source collected data on the first trajectory from the message middleware in an asynchronous multi-threaded manner includes: Real-time monitoring of messages from the message middleware; When a message that meets the first preset requirement is detected in the message middleware, the message that meets the first preset requirement and the corresponding displacement point are obtained; wherein, multiple displacement points constitute the first trajectory; Based on the displacement point, messages that meet the first preset requirements are read using an asynchronous multi-threaded approach. The second trajectory that records real-time calculated data includes: Obtain the source data from the real-time computing data; The source data is aggregated to obtain aggregated data; Calculate the second association between the source data and the aggregated data; The second trajectory of the real-time computing data is generated based on the second association.

2. The data end-to-end tracing and analysis method according to claim 1, characterized in that, After the step of generating the full-link graph based on the first node data, the second node data, the second trajectory, and the first correlation relationship, the method further includes: Generate a globally unique identifier for each data item in the entire link graph; The end-to-end graph is segmented and stored according to the globally unique identifier, and the indexing method is set according to the data type in the end-to-end graph.

3. The data end-to-end tracing and analysis method according to claim 1, characterized in that, Before the step of analyzing and displaying the entire data link based on the end-to-end diagram, the method further includes: Set a collection time label for each collected data point according to the time caliber. A calculation process diagram is generated based on the processing sequence number corresponding to the collected data; The abnormal dataset is determined based on the collection time stamp and the calculation process diagram.

4. The data end-to-end tracing and analysis method according to claim 3, characterized in that, The step of analyzing and displaying the entire data link based on the full-link diagram includes: The target data is obtained by querying the full-link graph based on the query data; After obtaining upstream and downstream data from the target data in the full-link diagram, a third association relationship between the upstream and downstream data and the target data is obtained. A data tracking and analysis graph is generated based on the upstream and downstream data, the third correlation, and the abnormal dataset. The data tracking and analysis chart is then displayed.

5. A data end-to-end tracking and analysis device, characterized in that, include: The first module is used to record the first trajectory of multi-source collected data and generate the first node data of the graph database based on the first trajectory. The second module is used to record the second trajectory of real-time computational data; The third module is used to record the third trajectory of the data entry result, generate the second node data of the graph database based on the third trajectory, and calculate the first association relationship between the first node data and the second node data. The first association relationship includes the association relationship between the node corresponding to the first node data and the node corresponding to the second node data. The fourth module is used to generate a full-link diagram based on the first node data, the second node data, the second trajectory, and the first association relationship; The fifth module is used to analyze and display the entire data link based on the full-link diagram; The first trajectory for recording multi-source collected data, and the generation of first node data for the graph database based on the first trajectory, include: The multi-source collected data is written to a message middleware, which is used to store the multi-source collected data and the first trajectory of the multi-source collected data. The multi-source data collected on the first trajectory is read from the message middleware in an asynchronous multi-threaded manner, and the read multi-source data is used as the first node data of the graph database. The step of reading multi-source collected data on the first trajectory from the message middleware in an asynchronous multi-threaded manner includes: Real-time monitoring of messages from the message middleware; When a message that meets the first preset requirement is detected in the message middleware, the message that meets the first preset requirement and the corresponding displacement point are obtained; wherein, multiple displacement points constitute the first trajectory; Based on the displacement point, messages that meet the first preset requirements are read using an asynchronous multi-threaded approach. The second trajectory that records real-time calculated data includes: Obtain the source data from the real-time computing data; The source data is aggregated to obtain aggregated data; Calculate the second association between the source data and the aggregated data; The second trajectory of the real-time computing data is generated based on the second association.

6. An electronic device, characterized in that, include: At least one memory for storing programs; At least one processor is configured to load the program to execute the data end-to-end tracing and analysis method as described in any one of claims 1-4.

7. A computer storage medium, characterized in that, It contains a computer-executable program, which, when executed by a processor, is used to implement the data end-to-end tracing and analysis method as described in any one of claims 1-4.

Images