A processing method and device for time series data based on time coordination in multiple execution stages

Abstract

The application discloses a processing method and device for time-coordinated processing of time-series data in multiple execution stages. The method defines a cooperative time window constraint for data, and enables the data to be in multiple execution stages while still coordinating the balanced state of data processing. It ensures that downstream processing is not affected by too fast upstream processing or too slow downstream data processing, and both upstream and downstream maintain data consistency. In the case of lack of system resources, the system resources are more focused on the running of delay tasks, the resources of local slow tasks are kept stable, the parallel processing efficiency of time-series data is ensured, and the quality of system indicators is greatly improved. The application is suitable for time-series data processing technology, streaming processing technology, batch processing technology and other fields, and solves the problem that time-series data cannot be processed in coordination in each execution stage while ensuring high parallelism of the system.

Description

Technical Field

[0001] This invention relates to the fields of time-series data processing technology, streaming processing technology, and batch processing technology, and in particular to a method and apparatus for processing time-series data based on time coordination at multiple execution stages. Background Technology

[0002] In the past, time-series data processing often focused on ensuring the availability of the latest intermediate data within a limited timeframe. Intermediate data stored in the indicator queue for too long had to be discarded, making it difficult to guarantee data quality. Furthermore, the lack of coordination between upstream and downstream tasks could lead to unstable consequences, such as extremely fast upstream consumption and slow downstream processing, which is highly unfriendly to streaming and batch processing. The absence of mutual constraints between tasks also significantly hinders time-series data processing. In anti-fraud scenarios, the accuracy of indicators is crucial. From upstream data being pushed into the rule engine for calculation to downstream indicator extraction based on the raw data, inefficient indicator extraction often results in the loss of some indicators, affecting the system's ability to accurately identify transaction risks and potentially leading to erroneous decisions. Existing platforms have failed to coordinate upstream and downstream processing of time-series data based on time, ensuring data quality with minimal impact on overall system parallelism. Summary of the Invention

[0003] The purpose of this invention is to address the shortcomings of existing technologies by proposing a time-coordination-based processing method for time-series data across multiple execution stages. This method utilizes the timestamps carried by the time-series data to coordinate data consistency across multiple execution stages and effectively improves the service node metrics quality of time-series data processing.

[0004] The objective of this invention is achieved through the following technical solution: Firstly, this invention provides a method for processing time-series data across multiple execution stages based on time coordination. The time-series data carries timestamps, and the consistency of the time-series data is maintained across each execution stage based on timestamp coordination. For a time-series data set P = {(t1,m1), (t2,m2), (t3,m3), ..., (tn,mn)}, where tn represents the time of the nth event and m represents the sequence number of the corresponding event ordered by time, the upstream and downstream processing of this time-series data set based on time coordination specifically includes the following steps:

[0005] (1) Define the time window size T to be coordinated according to the characteristics of the data set. The window will slide according to the speed of data processing at each stage, and the upstream and downstream tasks of each element in the time series data set P are constrained by the window size T.

[0006] (2) During the upstream processing stage, the data processing service node will record the timestamp of the data that needs to be processed, record it as the cursor value of the upstream task, and store it in the cache node.

[0007] (3) During the downstream processing stage, the data processing service node records the timestamp of the data currently being processed, which is recorded as the cursor value of the downstream task and stored in the cache node.

[0008] (4) The upstream processing stage in step (2) needs to verify the cursor value of the downstream processing stage in step (3) and determine whether it exceeds the limit of the coordination window by combining the window size T defined in the coordination window. If it does not exceed the limit, the upstream task and the downstream task continue to be executed in parallel. Otherwise, the upstream task needs to wait until the downstream task has processed part of the data and then pushes the window to move, and the upstream task continues to execute.

[0009] Furthermore, in step (1), the data characteristics include streaming data, batch data, incremental data, and full data.

[0010] Furthermore, in step (1), the upstream task and the downstream task are service nodes for different data processing of the same task, or service nodes for different data processing of different tasks.

[0011] Furthermore, in steps (2) and (3), the upstream processing stage and the downstream processing stage are two stages, or coordinated processing between multiple stages.

[0012] Furthermore, time-series data coordination processing is applicable to both single-node and multi-node systems.

[0013] Secondly, the present invention also provides a processing apparatus for time-coordinated processing of time-series data in multiple execution stages, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements the steps of the processing method for time-coordinated processing of time-series data in multiple execution stages.

[0014] Thirdly, the present invention also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of the aforementioned processing method for time-series data based on time coordination at multiple execution stages.

[0015] The beneficial effects of this invention are as follows: This invention provides a time-coordinated processing method for time-series data across multiple execution stages. It defines a coordinated time window constraint on the data, ensuring a balanced data processing state even across multiple execution stages. Most importantly, it coordinates the execution stages of the entire system, guaranteeing efficient parallel processing of time-series data while significantly improving system performance. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of a time-coordinated processing method for time-series data across multiple execution stages according to the present invention.

[0018] Figure 2 This is a comparison chart showing the effects of implementing the method of the present invention and the traditional method.

[0019] Figure 3 This is a structural diagram of a processing device for time-series data based on time coordination in multiple execution stages according to the present invention. Detailed Implementation

[0020] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0021] This invention provides a time-coordinated processing method for time-series data across multiple execution stages. This time-series data coordination processing is applicable to both single-node and multi-node scenarios. The time-series data carries timestamps, and the method coordinates the consistency of the data across different execution stages based on these timestamps. For a time-series data set P = {(t1,m1), (t2,m2), (t3,m3), ..., (tn,mn)}, where tn represents the time of the nth event and m represents the sequence number of the corresponding event sorted by time, the method coordinates upstream and downstream processing of this time-series data set based on time, specifically including the following steps:

[0022] (1) Define the time window size T for the data set to be coordinated based on the data characteristics. This window will shift according to the speed of data processing at each stage, and the upstream and downstream tasks of each element in the time series data set P are constrained by the window size T. The data characteristics include streaming data, batch data, incremental data, and full data. Upstream and downstream tasks are service nodes for different data processing of the same task, or service nodes for different data processing of different tasks. "Upstream" and "downstream" can be execution stages within different service nodes, or different execution stages within the same system, such as pushing data and calculating results into memory (upstream processing stage), and extra querying data results (downstream processing stage). The upstream and downstream processing stages are two stages, or coordinated processing between multiple stages. Multiple stages also need to be constrained by the same coordination window. The fastest executing stage among multiple stages also needs to wait for the slowest executing stage within the window. Therefore, the coordination method between two stages and multiple stages is consistent.

[0023] (2) During the upstream processing stage, the data processing service node will record the timestamp of the data that needs to be processed, record it as the cursor value of the upstream task, and store it in the cache node.

[0024] (3) During the downstream processing stage, the data processing service node records the timestamp of the data currently being processed, which is recorded as the cursor value of the downstream task and stored in the cache node.

[0025] (4) The upstream processing stage in step (2) needs to verify the cursor value of the downstream processing stage in step (3) and determine whether it exceeds the limit of the coordination window by combining the window size T defined in the coordination window. If it does not exceed the limit, the upstream task and the downstream task continue to be executed in parallel. Otherwise, the upstream task needs to wait until the downstream task has processed part of the data and then pushes the window to move, and the upstream task continues to execute.

[0026] Example 1

[0027] like Figure 1 As shown, this embodiment provides a method for processing time-series data based on time coordination at the same or different execution stages. Assume there is a transaction time-series data set P = {(10:02, m1), (11:05, m2), (12:13, m3), (12:22, m4), (14:05, m5), ...

[0028] (14:23, m6), (15:08, m7), (16:23, m8)}, the execution phase is divided into two processes: PUSH (data consumption) and EXTREACT (metric extraction). The specific implementation steps of this method are as follows:

[0029] 1) First, set the size of the time window that the current system needs to coordinate, assuming it is 3 hours, and its window range is as follows: Figure 1 As shown in t1 to t2, the window moves smoothly with the PUSH and EXTRACT operations.

[0030] 2) The upstream task executes the PUSH operation to push data into the engine, calculate the metrics, and temporarily store them in the metric cache queue; the downstream task executes the EXTRACT operation to extract the corresponding metrics from the metric cache queue based on the time series data, and write the obtained metrics into the storage node in the form of data blocks.

[0031] 3) When the PUSH operation reaches the push1 processing point and the EXTRACT operation is reaching the extract1 processing point, the windows of the two types of operations have not exceeded the coordination window size. Therefore, subsequent time series data can continue to be consumed after the push1 processing point.

[0032] 4) When the EXTRACT operation moves from the extract1 processing point to the extract2 processing point and extracts the m2 metrics, the EXTRACT operation starts to process slowly, assuming it stops at t1; at this time, the PUSH operation continues to consume the m5 and m6 data and reaches the push2 processing point, which corresponds to the boundary t2 of the coordination window. At this time, the PUSH operation will wait, and the PUSH operation at the subsequent push3 processing point will also be unable to be executed.

[0033] 5) Until the EXTRACT operation progresses along the time axis from the extract2 processing point, the corresponding PUSH operation also progresses along the time axis, and the two execution phases continue to coordinate with each other.

[0034] Example 2

[0035] By comparing the method of this invention with traditional methods when processing the same data, the method of this invention can improve the quality of system indicators better than the traditional method. This is verified through a series of experiments.

[0036] The experimental scenario involves randomly generating 800,000 transaction records for testing, each record requiring a transaction time field. The test data is evenly distributed over 80 minutes, with 10,000 transaction records per minute. The maximum cache time for indicators in the indicator cache queue is set to 10 minutes.

[0037] The testing method is as follows:

[0038] 1) Using traditional methods to conduct experiments, when consuming time-series data, ensure that the time-series data enters the system evenly, calculate the indicators, extract the indicators, and finally draw charts.

[0039] 2) Experiments were conducted using the method of this invention. Before uniform consumption timing, a timing coordination processing method was added, with the coordination window size set to 10 minutes, and then the same steps as the traditional method were performed.

[0040] Test results are as follows Figure 2 As shown in the results.

[0041] (1) Traditional methods have the problem of losing a large number of metrics, while the method of this invention can ensure that no metrics are lost in offline scenarios, and only a small number of metrics are lost in real-time scenarios due to uncontrollable factors such as network.

[0042] (2) Traditional methods can clearly show data loss and instability when processing time-series data, and the curves in the charts are relatively messy, which shows the instability of its internal processing. In contrast, the method of the present invention remains relatively stable.

[0043] Corresponding to the aforementioned embodiments of the processing method for time-series data based on time coordination in multiple execution stages, the present invention also provides embodiments of the processing apparatus for time-series data based on time coordination in multiple execution stages.

[0044] See Figure 3 The present invention provides a processing apparatus for time-coordinating time-series data across multiple execution stages, comprising a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it implements the processing method for time-coordinating time-series data across multiple execution stages described in the above embodiments.

[0045] The embodiments of this invention concerning a time-coordinated processing device for time-series data across multiple execution stages can be applied to any device with data processing capabilities, such as a computer. The device embodiments can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data-processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 3 The diagram shown is a hardware structure diagram of any device with data processing capabilities, which is a processing device for time-series data based on time coordination in multiple execution stages, except for... Figure 3 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0046] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0047] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0048] This invention also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements the time-coordinated processing method for time-series data across multiple execution stages described in the above embodiments.

[0049] The computer-readable storage medium can be an internal storage unit of any data processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0050] The above embodiments are used to explain and illustrate the present invention, but not to limit the present invention. Any modifications and changes made to the present invention within the spirit and scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A method for processing time-series data based on time coordination across multiple execution stages, characterized in that, Time-series data carries timestamps, and consistency is maintained across different execution stages based on these timestamps. For a time-series data set P = {(t1, m1), (t2, m2), (t3, m3), ..., (tn, mn)}, where tn represents the time of the nth event and m represents the sequence number of the corresponding event sorted by time, the upstream and downstream processing of this time-series data set is coordinated based on time, specifically including the following steps: (1) Data characteristics include streaming data, batch data, incremental data and full data. The time window size T to be coordinated is defined according to the data characteristics. This window will slide according to the speed of data processing at each stage. The upstream and downstream tasks of each element in the time series data set P are constrained by the window size T. (2) During the upstream processing stage, the data processing service node will record the timestamp of the data that needs to be processed, record it as the cursor value of the upstream task, and store it in the cache node; (3) During the downstream processing stage, the data processing service node will record the timestamp of the data currently being processed, which will be recorded as the cursor value of the downstream task and stored in the cache node. (4) The upstream processing stage in step (2) needs to verify the cursor value of the downstream processing stage in step (3) and determine whether it exceeds the limit of the coordination window by combining the window size T defined in the coordination window. If it does not exceed the limit, the upstream task and the downstream task continue to be executed in parallel. Otherwise, the upstream task needs to wait until the downstream task has processed part of the data and then pushes the window to move, and the upstream task continues to execute.

2. The method for processing time-series data based on time coordination across multiple execution stages according to claim 1, characterized in that, In step (1), the upstream task and the downstream task are service nodes for different data processing of the same task, or service nodes for different data processing of different tasks.

3. The method for processing time-series data based on time coordination across multiple execution stages according to claim 1, characterized in that, In steps (2) and (3), the upstream processing stage and the downstream processing stage are two stages, or coordinated processing between multiple stages.

4. The method for processing time-series data based on time coordination across multiple execution stages according to claim 1, characterized in that, Time-series data coordination processing is applicable to both single-node and multi-node systems.

5. A processing apparatus for time-coordinating time-series data across multiple execution stages, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that... When the processor executes the executable code, it implements the steps of the processing method for time-coordinated processing of time-series data in multiple execution stages as described in any one of claims 1-4.

6. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the processing method for time-series data based on time coordination in multiple execution stages as described in any one of claims 1-4.

Images