Data timeliness detection method and device, storage medium and product

Abstract

This application discloses a data timeliness detection method, device, storage medium, and product, relating to the field of electronic digital data processing technology. The data timeliness detection method includes: acquiring raw data from a data source; determining the timestamps corresponding to each process node in the data processing flow based on data embedding points, raw data, and data processing links; obtaining a processing flow tree containing the timestamps; calculating the delay of each process node based on the processing flow tree at preset delay calculation intervals to obtain target timeliness data; wherein intermediate results and historical data generated during the delay calculation are cached in memory. This application calculates the delay of each process node based on the processing link, and caches intermediate results and historical data in memory during calculation, eliminating the need for additional database operations, thereby improving the scope of data timeliness detection and reducing resource consumption.

Description

Technical Field

[0001] This application relates to the field of electronic digital data processing technology, and in particular to methods, devices, storage media and products for detecting data timeliness. Background Technology

[0002] Data timeliness is the core of data and application systems, and its value permeates the entire data lifecycle and business value chain. Current methods mainly assess latency by recording the time difference or link latency of data at a single stage, which requires additional database operations or relies on complex infrastructure, resulting in a limited scope of timeliness detection and high resource consumption. Summary of the Invention

[0003] The main purpose of this application is to provide a data timeliness detection method, device, storage medium and product, which aims to solve the technical problems of limited scope and high resource consumption in timeliness detection.

[0004] To achieve the above objectives, this application proposes a data timeliness detection method, the method comprising: Retrieve raw data from the data source; Based on preset data embedding points, the generation timestamp and reception timestamp of the original data are determined; Based on the data embedding points and data processing links, the processing timestamps corresponding to each process node in the data processing flow of the original data are determined, and a processing flow tree containing the processing timestamps is obtained. The data embedding points are pre-set in the process nodes of the data processing flow, and the data processing links are the processing links of the original data and the dependent data corresponding to the original data. At each preset delay calculation interval, based on the generated timestamp, the received timestamp, and the processing flow tree, the delay of each process node in the data processing flow is calculated to obtain the target timeliness data. The intermediate results and historical data generated during the delay calculation process are cached in memory.

[0005] In one embodiment, the step of determining the processing timestamps corresponding to each process node in the data processing flow of the original data based on the data embedding points and data processing links, and constructing a processing flow tree containing the processing timestamps, includes: The original data stream is copied to a preset processing bypass to obtain copied data, wherein the original data stream includes the original data and the dependent data corresponding to each of the original data; For each currently processing node in progress, determine the current processing timestamp of the replicated data at that current processing node; Based on the current processing timestamp and the data processing link, a processing flow tree containing the processing timestamp is constructed.

[0006] In one embodiment, the step of constructing a processing flow tree containing the processing timestamp based on the current processing timestamp and the data processing link includes: Based on the dependencies between the processing nodes in the processing chain, the upstream node of the current processing node is determined; Obtain the processing flow subtree corresponding to the upstream node, and construct a new processing flow subtree based on the current processing timestamp and the processing flow subtree; Returning to the step of determining the current processing timestamp of the copied data at each currently processing node, until all process nodes are completed, the processing flow tree is obtained.

[0007] In one embodiment, the step of calculating the latency of each process node in the data processing flow based on the generated timestamp, the received timestamp, and the processing flow tree at preset latency calculation intervals to obtain the target timeliness data includes: At each delay calculation interval, based on the generation timestamp and the reception timestamp, the time required for the data to be received from generation to reception is calculated to obtain the reception timeliness data; Determine the target process node that requires latency calculation from the processing flow tree; Calculate the difference between the target timestamps corresponding to the target process nodes to obtain the processing timeliness data; Based on the processing timestamp and the generation timestamp, the time required for the original data to go from generation to completion of processing is calculated to obtain the target timeliness data.

[0008] In one embodiment, before the step of determining the generation timestamp and reception timestamp of the original data based on preset data embedding points, the method further includes: The data source is converted into a unified, preset business database model; The step of calculating the time required for the original data to complete processing from generation to completion based on the processing timestamp and the generation timestamp, and obtaining the data timeliness, further includes: The data in the business database model is combined with the corresponding target timeliness data to obtain business timeliness data.

[0009] In one embodiment, after the step of calculating the latency of each process node in the data processing flow based on the generated timestamp, the received timestamp, and the processing flow tree at preset latency calculation intervals to obtain the target timeliness data, the method further includes: Every preset time period, the target timeliness data corresponding to each of the data sources is summarized, and the maximum receiving timeliness, maximum processing timeliness and maximum data timeliness in each of the data sources are visualized to obtain a database table, wherein the database table includes multiple data items, and each data item includes data from multiple data sources; Based on the database table, determine the root cause of the abnormal data where the timeliness exceeds a preset timeliness threshold.

[0010] In one embodiment, the step of determining the root cause of the abnormal data where the timeliness exceeds a preset timeliness threshold based on the database table includes: Based on the database table, determine the data set associated with the abnormal data; Traverse the data set to determine the unique key of the data item with the largest timeliness data in the data set; The root cause of the anomaly is determined based on the unique key.

[0011] In addition, to achieve the above objectives, this application also proposes a data timeliness detection device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the data timeliness detection method as described above.

[0012] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the data timeliness detection method described above.

[0013] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the data timeliness detection method described above.

[0014] One or more technical solutions proposed in this application have at least the following technical effects: This application obtains raw data from a data source, and based on preset data embedding points, the raw data, and the data processing link, determines the timestamps corresponding to each process node of the data processing flow, thereby obtaining a processing flow tree containing the timestamps. The data embedding points are preset in the process nodes of the data processing flow, and the data processing link is the processing link of the raw data and the dependent data corresponding to the raw data. At preset delay calculation intervals, based on the processing flow tree, the delay of each process node of the data processing flow is calculated to obtain the target timeliness data. The intermediate results and historical data generated during the delay calculation process are cached in memory.

[0015] Current methods primarily assess latency by recording the time difference or link latency of data in a single stage, requiring additional database operations or relying on complex infrastructure, resulting in a limited scope of timeliness detection and high resource consumption. This application reduces reliance on databases and the number of additional operations, thereby reducing resource consumption and expanding the scope of data timeliness detection. Specifically, this application does not assess latency in a single stage based on the timestamp of the original data, but rather determines the timestamps of each process node based on the processing link of the original data and dependent data, obtaining a processing flow tree containing timestamp data, thus enabling full lifecycle timeliness analysis of the original data and dependent data. Furthermore, this application caches intermediate results and historical data in memory when calculating latency, eliminating the need for additional database operations and further reducing resource consumption. Therefore, this application can reduce resource consumption and expand the scope of data timeliness detection. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating an embodiment of the data timeliness detection method of this application. Figure 2 This is a schematic diagram of data embedding points provided for Embodiment 1 of the data timeliness detection method of this application; Figure 3 A schematic diagram of data bypass analysis provided for Embodiment 1 of the data timeliness detection method of this application; Figure 4This is a schematic diagram of the processing flow tree provided in Embodiment 1 of the data timeliness detection method of this application; Figure 5 A schematic diagram of the business database model and time-sensitive data combination provided in Embodiment 1 of the data timeliness detection method of this application; Figure 6 This is a flowchart illustrating Embodiment 2 of the data timeliness detection method of this application. Figure 7 This is a schematic representation of the database provided in Embodiment 2 of the data timeliness detection method of this application; Figure 8 This is a schematic diagram illustrating the determination of a unique key in Embodiment 2 of the data timeliness detection method of this application. Figure 9 This is a schematic diagram of the device structure of the hardware operating environment involved in the data timeliness detection method in this application embodiment; Figure 10 This is a schematic diagram illustrating the data acquisition consent process involved in the data timeliness detection method in this application embodiment.

[0019] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0020] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0021] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0022] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or data timeliness detection device capable of performing the above functions. The following description uses a data timeliness detection device as an example to illustrate this embodiment and the subsequent embodiments.

[0023] Data timeliness is the core lifeline of data systems and application systems, and its value permeates the entire data lifecycle and business value chain. In data systems, timeliness directly determines the freshness of data and the reliability of causal relationships. Delayed data not only distorts the system's perception of the current state, leading to the disruption of resource consistency, but may also cause a chain reaction of deterioration in storage and computing efficiency due to the accumulation of expired data. At the application system level, a lack of timeliness may cause data-based judgments to lag behind real business dynamics. This can range from reducing the accuracy of decisions in scenarios such as recommendations and risk control to missing market opportunities or exacerbating security risks due to response delays.

[0024] Current data timeliness detection methods typically rely on timestamp comparison, heartbeat probes, or distributed tracing. Their core idea is to assess latency by recording the time difference or link latency of data in a single stage (such as transmission or processing). However, they lack timeliness analysis of the entire lifecycle of data in this stage (generation, reception, and processing), and even more so, they lack timeliness analysis of the entire lifecycle of dependent data (generation, reception, and processing) in the entire link, resulting in a limited detection scope.

[0025] Current distributed tracing methods require deep embedding in business code, resulting in strong code intrusion, increased development and maintenance complexity, and limited application in heterogeneous data source environments. Furthermore, when calculating data timeliness, current methods require additional database operations or rely on complex infrastructure, leading to high resource consumption.

[0026] Furthermore, current methods are typically designed for specific types of data sources and are difficult to extend to multiple data formats (such as interface-type data, message queue-type data, etc.), resulting in a limited scope of applicability.

[0027] Based on this, embodiments of this application provide a data timeliness detection method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the data timeliness detection method of this application.

[0028] In this embodiment, the data timeliness detection method includes steps S10 to S40: Step S10: Obtain the raw data from the data source; It should be noted that the data source is the original system, database, message queue, or log file that generates or stores the data to be tested. Raw data refers to the initial data records read directly from the data source without any cleaning, transformation, or aggregation processing, including the timestamp of data generation and business content.

[0029] Step S20: Based on preset data embedding points, determine the generation timestamp and reception timestamp of the original data; It should be noted that pre-set data collection points refer to data collection points set in advance at key stages of the data's entire lifecycle (such as the data start point, receiving node, and processing node) to record precise timestamp information. The generation timestamp refers to the time recorded by the data collection point at the data start point (the moment the data is generated), used to identify the generation time of the original data. The receiving timestamp refers to the time recorded by the data collection point at the data receiving node (the moment the data is received by the system), used to identify the receiving time of the original data.

[0030] It is understood that this embodiment collects the generation timestamp and reception timestamp of the original data through preset data embedding points, which can accurately record the time consumption information from data generation to reception, thereby providing an accurate time reference for the subsequent quantification of reception delay, thus improving the accuracy of timeliness detection.

[0031] Step S30: Based on the data embedding points and data processing links, determine the processing timestamps corresponding to each process node in the data processing flow of the original data, and obtain a processing flow tree containing the processing timestamps. The data embedding points are pre-set in the process nodes of the data processing flow, and the data processing links are the processing links of the original data and the dependent data corresponding to the original data. It's important to note that a data processing chain refers to the complete path that raw data and its dependent data traverse from the start of processing to completion, encompassing multiple processing stages and their interdependencies. A process node refers to a specific execution unit or processing stage within the data processing flow. A processing timestamp refers to the time recorded by embedded points at each process node, used to identify the point in time when the data was processed at that node. A processing flow tree abstracts the processing of each piece of data into a node, and the entire chain of data and its dependent data processing into a tree structure, where nodes represent data processing stages and edges represent data dependencies. Dependent data refers to other auxiliary or reference data that must be referenced during the calculation, transformation, or correlation analysis of the raw data in the data processing process.

[0032] It is understandable that this embodiment obtains processing timestamps by setting data tracking points at each process node, and abstracts the processing link of the original data and its dependent data into a processing flow tree. This allows for the simultaneous recording of the completion time of the data itself and its dependent data at each processing stage, clearly expressing dependencies in a tree structure. This avoids the limitations of focusing only on a single stage or ignoring data dependencies, achieving end-to-end, multi-dimensional timeliness evaluation from data generation to processing completion. The data tracking point placement in this embodiment can be referenced from [reference needed]. Figure 2 , Figure 2 This is a schematic diagram illustrating the embedding of data points at the data generation, reception, and processing nodes.

[0033] Step S40: At each preset delay calculation interval, based on the generated timestamp, the received timestamp, and the processing flow tree, calculate the delay of each process node in the data processing flow to obtain the target timeliness data. The intermediate results and historical data generated during the delay calculation process are cached in memory.

[0034] It should be noted that the preset latency calculation interval refers to the time period for periodically triggering latency calculations, as pre-set by the system, such as performing a calculation at fixed intervals. Latency refers to the time difference consumed in transmitting data between two adjacent process nodes or processing data within a process node. Target timeliness data refers to the specific data indicators or set generated after calculating the latency of each process node, used to characterize the timeliness status of the current data processing process. Intermediate results refer to temporary data generated during latency calculation, while historical data refers to latency records, target timeliness data, and related statistical information that have been calculated within a past period.

[0035] Understandably, since this step caches the intermediate results and historical data generated during the calculation of latency in memory, the number of database query operations can be reduced when processing data items, thereby reducing the dependence on the database and redundant calculations, and reducing resource consumption.

[0036] Furthermore, in this embodiment, calculations are periodically triggered according to a preset delay calculation interval, and the delay of each process node is calculated based on the generation timestamp, receiving timestamp, and processing timestamp of each node in the processing flow tree. Then, target timeliness data reflecting the current link operation status is generated, thereby realizing the full-link timeliness detection of the original data and dependent data.

[0037] In one feasible implementation, the specific implementation of determining the processing timestamps corresponding to each process node in the data processing flow of the original data based on the data embedding points and data processing links, and constructing a processing flow tree containing the processing timestamps, can also be: The original data stream is copied to a preset processing bypass to obtain copied data, wherein the original data stream includes the original data and the dependent data corresponding to each of the original data; For each currently processing node in progress, determine the current processing timestamp of the replicated data at that current processing node; Based on the current processing timestamp and the data processing link, a processing flow tree containing the processing timestamp is constructed.

[0038] It should be noted that a processing bypass refers to a parallel data channel or computing path independently constructed outside the main data processing flow. This path is specifically designed to carry copied data for monitoring, analysis, or detection operations, and its operation does not interfere with the normal business logic of the main flow. The data bypass analysis process in this embodiment can be referred to... Figure 3The raw data stream refers to the complete sequence of data that actually flows in the data processing flow. It includes not only the initial raw data and dependent data, but also intermediate or final data generated after the data has been processed by each process node. Duplicate data refers to a copy of the data cloned from the raw data stream through a splitting mechanism, whose content, structure, and temporal characteristics are consistent with the original data stream.

[0039] The current processing node refers to the process node that is being executed or computed in the data processing flow. The current processing timestamp refers to the precise time recorded at the current processing node when the copied data completes the processing of that node.

[0040] It is understood that this implementation copies the original data stream to an independent processing bypass for latency calculation, so that timeliness analysis and dynamic data caching operations are performed only on the copied data in the bypass without occupying the computing resources of the main process, thereby avoiding the performance loss that the detection mechanism may cause to the production system.

[0041] In one feasible implementation, the specific implementation of constructing a processing flow tree containing the processing timestamp based on the current processing timestamp and the data processing link can also be: Based on the dependencies between the processing nodes in the processing chain, the upstream node of the current processing node is determined; Obtain the processing flow subtree corresponding to the upstream node, and construct a new processing flow subtree based on the current processing timestamp and the processing flow subtree; Returning to the step of determining the current processing timestamp of the copied data at each currently processing node, until all process nodes are completed, the processing flow tree is obtained.

[0042] It should be noted that an upstream node refers to a process node that executes before the current processing node and that the current node depends on; its processing results or data are prerequisites for the current node's processing. A processing flow subtree refers to a branch or local tree structure in the processing flow tree, representing a complete processing chain extending downwards from a certain upstream node and its timestamp information.

[0043] It is understood that, in constructing the processing flow tree, this implementation first determines the upstream node of the current processing node and obtains the processing flow subtree that the upstream node has already constructed. Then, it only aggregates the current processing timestamp of the current processing node with the subtree to generate a new subtree. This avoids retracing the entire link or building the full tree structure from scratch for each node being processed, which greatly reduces the computation time and resource overhead required for full-link analysis and achieves efficient, incremental processing flow tree construction.

[0044] In one feasible implementation, the specific implementation of calculating the latency of each process node in the data processing flow based on the generated timestamp, the received timestamp, and the processing flow tree at preset latency calculation intervals to obtain the target timeliness data can also be: At each delay calculation interval, based on the generation timestamp and the reception timestamp, the time required for the data to be received from generation to reception is calculated to obtain the reception timeliness data; Determine the target process node that requires latency calculation from the processing flow tree; Calculate the difference between the target timestamps corresponding to the target process nodes to obtain the processing timeliness data; Based on the processing timestamp and the generation timestamp, the time required for the original data to go from generation to completion of processing is calculated to obtain the target timeliness data.

[0045] It should be noted that reception timeliness data refers to the quantitative result reflecting the time required from data generation to reception, used to assess the latency of data transmission or access. Target process node refers to the node(s) in the processing flow tree that currently require latency calculation, which can be selected according to preset rules or actual needs. Target timestamp refers to the timestamp associated with the target process node used for latency calculation, such as the processing completion timestamp of that node and its upstream node. Processing timeliness data refers to the quantitative result reflecting the time required to complete the processing stage after data reception, used to assess the latency of the data processing stage.

[0046] It is understood that this implementation calculates the receiving timeliness data based on the generation timestamp and the receiving timestamp respectively, calculates the processing timeliness data based on the target timestamp difference of the target process nodes in the processing flow tree, and calculates the overall target timeliness data based on the processing timestamp and the generation timestamp. Through the aforementioned processing flow tree, this embodiment ensures that timeliness detection is not merely based on recording the time difference or link latency of data in a single stage, but rather combines the processing logic relationship between the original data and dependent data to achieve timeliness detection throughout the entire lifecycle of the original data and dependent data.

[0047] Furthermore, in this embodiment, the child nodes of the processing flow tree have the corresponding timeliness data of their parent nodes. When calculating the timeliness of the data, there is no need to repeatedly perform database queries, thereby reducing resource consumption. Moreover, by calculating the tree height and directly marking abnormal data in the tree structure data, this embodiment enables the system to accurately locate the specific branches and nodes causing the overall delay, thus achieving precise location of timeliness anomalies.

[0048] In one feasible implementation, the specific implementation prior to the step of determining the generation timestamp and reception timestamp of the original data based on preset data embedding points can also be: The data source is converted into a unified, preset business database model; The step of calculating the time required for the original data to complete processing from generation to completion based on the processing timestamp and the generation timestamp, and obtaining the data timeliness, further includes: The data in the business database model is combined with the corresponding target timeliness data to obtain business timeliness data.

[0049] It should be noted that the preset business database model refers to a predefined set of standardized data structures used to uniformly map and transform raw data from different data sources with varying formats into a standardized format with fixed fields, types, and relationships, thus eliminating the differences between heterogeneous data sources. Business timeliness data refers to a composite data object formed by associating and binding the standardized business data content with previously calculated multi-dimensional timeliness indicators (receipt timeliness, processing timeliness, and data timeliness). It contains both business semantic information and the timeliness status of the business data throughout the entire process.

[0050] Understandably, current methods are typically designed for specific types of data sources and are difficult to extend to multiple data formats (such as interface data, message queue data, etc.), thus limiting their applicability. This embodiment introduces a unified, pre-defined business database model at the front end of the detection process for data transformation, and combines standardized business data with multi-dimensional timeliness indicators (reception, processing, data timeliness) at the back end. (See reference...) Figure 5 , Figure 5 This diagram illustrates the combination of a business database model and time-sensitive data in this embodiment. By performing universal modeling of data timeliness, the timeliness detection in this embodiment can support various data formats (such as interface-type data, message queue-type data, etc.), exhibiting good scalability and adaptability, and is suitable for heterogeneous data source environments.

[0051] In summary, this embodiment obtains raw data from the data source, determines the timestamps corresponding to each process node of the data processing flow based on preset data embedding points, the raw data, and the data processing link, and obtains a processing flow tree containing the timestamps. The data embedding points are preset in the process nodes of the data processing flow, and the data processing link is the processing link of the raw data and the dependent data corresponding to the raw data. Every preset delay calculation interval, the delay of each process node of the data processing flow is calculated based on the processing flow tree to obtain the target timeliness data. The intermediate results and historical data generated during the delay calculation are cached in memory.

[0052] Current methods primarily assess latency by recording the time difference or link latency of data in a single stage, requiring additional database operations or relying on complex infrastructure, resulting in a limited scope of timeliness detection and high resource consumption. This embodiment reduces reliance on databases and the number of additional operations, thereby reducing resource consumption and expanding the scope of data timeliness detection. Specifically, this application does not evaluate latency in a single stage based on the timestamp of the original data, but rather determines the timestamps of each process node based on the processing link of the original data and dependent data, obtaining a processing flow tree containing timestamp data, thus enabling full lifecycle timeliness analysis of the original data and dependent data. Furthermore, this embodiment caches intermediate results and historical data in memory when calculating latency, eliminating the need for additional database operations and reducing resource consumption. Therefore, this embodiment can reduce resource consumption and expand the scope of data timeliness detection.

[0053] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 6 After step S40, the data timeliness detection method further includes steps S41-S42: Step S41: At each preset time period, the target timeliness data corresponding to each data source is summarized, and the maximum receiving timeliness, maximum processing timeliness and maximum data timeliness in each data source are visualized to obtain a database table, wherein the database table includes multiple data items, and each data item includes data from multiple data sources. It should be noted that maximum receiving timeliness refers to the maximum value of the receiving timeliness index across all records from a specific data source or multiple data sources within a preset time period. Maximum processing timeliness refers to the maximum value of the processing timeliness index across all records from a specific data source or multiple data sources within a preset time period. Maximum data timeliness refers to the maximum value of the data timeliness index across all records from a specific data source or multiple data sources within a preset time period. (See reference...) Figure 7 , Figure 7 This is a schematic diagram of the periodic indicators of the database table in this embodiment. A database table refers to a structured data collection used to persistently store summary results, and a data item refers to the basic record unit or row in the database table. In this embodiment, each data item not only contains the statistical results of a single data source, but also aggregates relevant data from multiple data sources to form a multi-dimensional statistical record.

[0054] Understandably, after generating the target timeliness data, this embodiment automatically extracts all target timeliness data within a preset time period from the historical cache of each data source. It then filters and calculates the maximum receiving timeliness, maximum processing timeliness, and maximum data timeliness for each data source, storing these extreme values ​​in a database table for display. Each data item in the database table integrates statistical information from multiple data sources. This embodiment automates the aggregation and analysis of end-to-end data, generating monitoring indicators at the database table level, and performing horizontal comparative analysis of timeliness data across data sources. This allows operations and maintenance personnel to easily identify which data source experienced the most severe latency peak within a specific period.

[0055] Step S42: Based on the database table, determine the root cause of the abnormal data where the timeliness exceeds a preset timeliness threshold.

[0056] It should be noted that the preset timeliness threshold refers to the critical numerical standard set by the system in advance to determine whether the timeliness of the data is qualified. When the actual monitored timeliness data exceeds this value, it is marked as abnormal. The root cause of the abnormality refers to the fundamental reason or specific fault point that causes the timeliness data to exceed the preset timeliness threshold.

[0057] In one feasible implementation, the specific implementation of determining the root cause of the abnormal data where the timeliness exceeds a preset timeliness threshold based on the database table can also be: Based on the database table, determine the data set associated with the abnormal data, traverse the data set, determine the unique key of the data item with the largest timeliness data in the data set, and determine the root cause of the anomaly based on the unique key.

[0058] It should be noted that a dataset refers to a collection of data records selected from a database table that have a logical relationship with the currently identified anomaly (such as the same time period, the same business link, or the same fault group). A unique key refers to a field or combination of fields (such as primary key ID, end-to-end tracing ID, TraceID, etc.) that can uniquely identify a specific data record (data item) within the dataset, used to precisely locate the specific virtual machine or host machine causing the problem. See [reference needed]. Figure 8 , Figure 8 This is a schematic diagram illustrating how a unique key for a data item with a time delay in reception is determined in this embodiment.

[0059] It is understood that this embodiment first extracts the data set associated with the current abnormal data based on the database table, then performs a full traversal of the set, and by comparing the timeliness data values ​​in each data item, it locks the data item with the largest timeliness data (i.e. the most severe delay), and extracts the unique key of the data item. Using the unique key, it accurately determines the corresponding virtual machine or host computer, thereby accurately determining the root cause of the overall abnormality.

[0060] In one feasible implementation, the specific method for determining the root cause of an anomaly based on the database table may also be: When the timeliness of a data item exceeds a preset timeliness threshold, the complete path of the data item in the processing flow tree is traced back, and the proportion of the receiving timeliness and processing timeliness of each node on the path to the total timeliness of the data item is calculated. Combined with the fluctuation variance of the node in the historical period, the timeliness contribution weight of each node is dynamically generated. The node with the highest timeliness contribution weight is marked as the primary root cause, and the node with the second highest weight and a direct parent-child dependency relationship with the primary root cause is marked as the secondary root cause. A root cause analysis report containing the relationship between the primary and secondary root causes is generated.

[0061] It should be noted that the timeliness contribution weight refers to the proportion of time consumed by a specific process node relative to the total latency of the entire chain, and is a quantitative indicator after being corrected for the historical stability (variance) of that node. The primary root cause refers to the process node with the highest timeliness contribution weight and the greatest impact on overall latency in the current abnormal event; it is the primary optimization target. The secondary root cause refers to the process node with the second highest timeliness contribution weight after the primary root cause in the current abnormal event, and which has a direct upstream / downstream dependency relationship (parent-child node) with the primary root cause in the processing flow tree; it is usually the trigger or accompanying phenomenon leading to the backlog of the primary root cause. Variance refers to the degree of dispersion of the processing timeliness or receiving timeliness of a process node within a historical preset time period, used to characterize the stability of the node's operation; the larger the variance, the more unstable the node.

[0062] It is understandable that this embodiment combines the topological dependencies of the processing flow tree and the historical fluctuation variance to calculate the timeliness contribution weight, which can eliminate the interference of nodes that are normal and stable (low variance) although they take a long time, and accurately capture the key nodes that have severe fluctuations (high variance) and contribute the most to the total delay as the root cause, thereby improving the accuracy of the determination of abnormal root causes.

[0063] In one feasible implementation, the proportion of the receiving timeliness and processing timeliness of each node on the path to the total data timeliness of the data item is calculated, and the timeliness contribution weight of each node is dynamically generated in combination with the fluctuation variance of the node in the historical period. Another possible implementation is: Calculate the mutation multiple of the current time consumption of each node relative to its historical average time consumption, and calculate the stability of its historical time consumption; based on the mutation multiple, the stability, and the timeliness data of the node, adjust the timeliness contribution weight to obtain the adjusted target timeliness contribution weight.

[0064] It should be noted that the mutation multiple refers to the ratio of the current time consumption of a node to its historical average time consumption, reflecting the degree of deterioration in the node's current performance relative to the normal level. The larger the multiple, the more severe the anomaly. Stability refers to the stability of the node's time consumption over a historical period. If the historical time consumption fluctuates very little, the stability is high, indicating that the node is reliable; if the historical time consumption fluctuates wildly, the stability is low, indicating that the node itself is unstable. The adjusted target timeliness contribution weight refers to the weight value recalculated based on the original proportion weight, after introducing the mutation multiple and stability as correction coefficients.

[0065] Understandably, this embodiment first calculates the mutation multiple of the current time consumption of each node on the path relative to the historical average time consumption, and then calculates the stability of its historical time consumption. Subsequently, using the mutation multiple as an amplification factor and the stability as a confidence coefficient, the original timeliness data proportion of each node is dynamically corrected: for nodes with large mutation multiples and high stability, their weights are significantly increased; for nodes with small mutation multiples (i.e., no different from usual) or extremely low stability (i.e., unstable over the years), their weights are suppressed; finally, the adjusted target timeliness contribution weight is obtained.

[0066] This implementation introduces dynamic adjustments to the weights based on mutation fold and stability, and incorporates historical periodic data stored in this application as a benchmark. This effectively isolates nodes in the link that have long latency periods due to complex business logic but are stable in operation, avoiding misjudging them as root causes of failure. Meanwhile, abnormal mutation nodes that normally run extremely fast but suddenly experience severe delays, even if their absolute latency is not the highest in the long link, can be identified as the primary root cause due to their large mutation fold, thereby improving the accuracy of abnormal root cause identification.

[0067] In summary, this embodiment aggregates and statistically analyzes the target timeliness data from each data source at preset time intervals, extracts the maximum receiving timeliness, maximum processing timeliness, and maximum data timeliness, and integrates these extreme value indicators with information from multiple data sources into a database table for visualization. Subsequently, based on this database table, the statistically obtained timeliness data is compared with preset timeliness thresholds to filter out records that exceed the thresholds. By traversing the associated data set, the data item with the largest timeliness data and its unique key are located, thereby accurately locating the root cause of the anomaly.

[0068] This embodiment automates the aggregation and analysis of end-to-end data, generates monitoring indicators at the database table level, and performs horizontal comparative analysis of timeliness data across data sources. This allows operations and maintenance personnel to instantly identify which data source experienced the most severe latency peak within a specific period. Furthermore, by traversing the dataset, it identifies the data item with the largest timeliness data and extracts its unique key, thereby accurately determining the virtual machine or host machine causing the data timeliness issue.

[0069] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the data timeliness detection method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0070] This application provides a data timeliness detection device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the data timeliness detection method in the first embodiment described above.

[0071] The following is for reference. Figure 9 The diagram illustrates a structural schematic of a data timeliness detection device suitable for implementing embodiments of this application. The data timeliness detection device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, tablets, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 9 The data timeliness detection device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0072] like Figure 9As shown, the data timeliness detection device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the data timeliness detection device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 allows the data timeliness detection device to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows data timeliness detection devices with various systems, it should be understood that it is not required to implement or possess all of the systems shown. More or fewer systems may be implemented alternatively.

[0073] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0074] The data timeliness detection device provided in this application, employing the data timeliness detection method described in the above embodiments, can solve the technical problems of limited detection scope and high resource consumption. Compared with the prior art, the beneficial effects of the data timeliness detection device provided in this application are the same as those of the data timeliness detection method described in the above embodiments, and other technical features of this data timeliness detection device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0075] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0076] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0077] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the data timeliness detection method in the above embodiments.

[0078] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0079] The aforementioned computer-readable storage medium may be included in the data timeliness testing device; or it may exist independently and not assembled into the data timeliness testing device.

[0080] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the data timeliness detection device, cause the data timeliness detection device to perform the aforementioned data timeliness detection method.

[0081] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0082] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0083] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0084] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described data timeliness detection method, thereby solving the technical problems of limited scope and high resource consumption in timeliness detection. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the data timeliness detection method provided in the above embodiments, and will not be repeated here.

[0085] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the data timeliness detection method described above.

[0086] The computer program product provided in this application can solve the technical problems of limited scope and high resource consumption in timeliness detection. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the data timeliness detection method provided in the above embodiments, and will not be repeated here.

[0087] All user-related data involved in this application was obtained with the user's permission or consent, as per [reference]. Figure 10 In other words, when this application is applied to a specific product or technology, user permission is required to acquire and process the relevant data, and the processing of the relevant data must comply with the relevant laws, regulations and regulatory standards of the relevant countries and regions.

[0088] The above description is only a part of the embodiments of this application and does not limit the scope of protection of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for detecting the timeliness of data, characterized in that, The method includes: Retrieve raw data from the data source; Based on preset data embedding points, the generation timestamp and reception timestamp of the original data are determined; Based on the data embedding points and data processing links, the processing timestamps corresponding to each process node in the data processing flow of the original data are determined, and a processing flow tree containing the processing timestamps is obtained. The data embedding points are pre-set in the process nodes of the data processing flow, and the data processing links are the processing links of the original data and the dependent data corresponding to the original data. At each preset delay calculation interval, based on the generated timestamp, the received timestamp, and the processing flow tree, the delay of each process node in the data processing flow is calculated to obtain the target timeliness data. The intermediate results and historical data generated during the delay calculation process are cached in memory.

2. The method as described in claim 1, characterized in that, The step of determining the processing timestamps corresponding to each process node in the data processing flow of the original data based on the data embedding points and data processing links, and constructing a processing flow tree containing the processing timestamps, includes: The original data stream is copied to a preset processing bypass to obtain copied data, wherein the original data stream includes the original data and the dependent data corresponding to each of the original data; For each currently processing node in progress, determine the current processing timestamp of the replicated data at that current processing node; Based on the current processing timestamp and the data processing link, a processing flow tree containing the processing timestamp is constructed.

3. The method as described in claim 2, characterized in that, The step of constructing a processing flow tree containing the processing timestamp based on the current processing timestamp and the data processing link includes: Based on the dependencies between the processing nodes in the processing chain, the upstream node of the current processing node is determined; Obtain the processing flow subtree corresponding to the upstream node, and construct a new processing flow subtree based on the current processing timestamp and the processing flow subtree; Returning to the step of determining the current processing timestamp of the copied data at each currently processing node, until all process nodes are completed, the processing flow tree is obtained.

4. The method as described in claim 1, characterized in that, The step of calculating the latency of each process node in the data processing flow at preset latency calculation intervals, based on the generated timestamp, the received timestamp, and the processing flow tree, to obtain the target timeliness data includes: At each delay calculation interval, based on the generation timestamp and the reception timestamp, the time required for the data to be received from generation to reception is calculated to obtain the reception timeliness data; Determine the target process node that requires latency calculation from the processing flow tree; Calculate the difference between the target timestamps corresponding to the target process nodes to obtain the processing timeliness data; Based on the processing timestamp and the generation timestamp, the time required for the original data to go from generation to completion of processing is calculated to obtain the target timeliness data.

5. The method as described in claim 4, characterized in that, Before the step of determining the generation timestamp and reception timestamp of the original data based on preset data embedding points, the method further includes: The data source is converted into a unified, preset business database model; The step of calculating the time required for the original data to complete processing from generation to completion based on the processing timestamp and the generation timestamp, and obtaining the data timeliness, further includes: The data in the business database model is combined with the corresponding target timeliness data to obtain business timeliness data.

6. The method as described in claim 1, characterized in that, After the step of calculating the latency of each process node in the data processing flow based on the generated timestamp, the received timestamp, and the processing flow tree at preset latency calculation intervals to obtain the target timeliness data, the method further includes: Every preset time period, the target timeliness data corresponding to each of the data sources is summarized, and the maximum receiving timeliness, maximum processing timeliness and maximum data timeliness in each of the data sources are visualized to obtain a database table, wherein the database table includes multiple data items, and each data item includes data from multiple data sources; Based on the database table, determine the root cause of the abnormal data where the timeliness exceeds a preset timeliness threshold.

7. The method as described in claim 6, characterized in that, The step of determining the root cause of abnormal data where the timeliness exceeds a preset timeliness threshold based on the database table includes: Based on the database table, determine the data set associated with the abnormal data; Traverse the data set to determine the unique key of the data item with the largest timeliness data in the data set; The root cause of the anomaly is determined based on the unique key.

8. A data timeliness detection device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the data timeliness detection method as described in any one of claims 1 to 7.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the data timeliness detection method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the data timeliness detection method as described in any one of claims 1 to 7.

Images