A method for time-series processing of data in distributed systems

Abstract

This invention discloses a method for time-series data processing in distributed systems. This method reverses the processing order of data falling on different nodes, processing later-generated data later and earlier-generated data earlier. Specifically, the timestamp of data processed by each node in the system is written to an external cache system. During data processing, each node compares its processed data with the timestamps written to the external cache system to ensure that the processed data time is less than the smallest timestamp among all nodes. This invention addresses the problem of ensuring data passes through each stage in a time-series data processing manner in a distributed system architecture while maintaining high system throughput, which is difficult with existing technologies. It is an efficient method for improving data order processing and better supports business needs.

Description

Technical Field

[0001] This invention relates to the field of data analysis using time-series data processing technology, and more particularly to a method for time-series processing of data from distributed systems. Background Technology

[0002] In traditional distributed processing systems, subsystem nodes typically consume data in a best-effort manner, meaning each node operates independently with little interaction. Data is processed immediately upon receipt, and the results are pushed to downstream systems. Because each node operates independently, some nodes execute faster than others, resulting in poor timeliness and significant data misalignment when data is pushed downstream. This is unsuitable for scenarios requiring sequential data delivery. For example, in fraud prevention, high timeliness is crucial. Out-of-order data entering the risk control system can lead to inaccurate calculations of relevant indicators, hindering the system's ability to accurately identify transaction risks. Existing platforms fail to simultaneously guarantee high throughput and correct data misalignment. Meeting both requirements would solve the problem of out-of-order data processing in distributed systems, bringing significant business value. Summary of the Invention

[0003] In view of the above problems, this invention proposes a method for time-series processing of data in distributed systems to improve data timeliness and better support business needs.

[0004] The objective of this invention is achieved through the following technical solution: a method for time-series processing of data in a distributed system, specifically comprising the following steps:

[0005] (1) The message bus of the computer system is used as the data source. At the same time, there are several nodes that subscribe to the message bus. Each node randomly obtains data from the message bus. There are several data processing links in each node, and the data within a certain period of time is processed in the order of the links.

[0006] (2) Creation Stage A i The processing task: After the data obtained from the message bus in step (1) is received, it goes through stage A. i A i This represents the i-th processing step in the node, ending at step A. i The event time of the last data in the current batch is written to the external cache system, and this is recorded as node processing step A. i The vernier value.

[0007] (3) Creation Stage A i+1Processing task: The process will proceed through stage A. i+1 The subsequent data processing step A i+1 The processing, in the final stage A i+1 The event time of the last data in the current batch is written to the external cache system, and this is recorded as node processing step A. i+1 The vernier value.

[0008] (4) Steps (2) and (3) need to cross-check the cursor values ​​of each other to determine whether the data can be processed. The standard for data processing is: when the data enters A i+1 During the process, the time and A in the data will be used. i The minimum cursor value of the data is compared. If the event time of the data is less than this cursor value, it means that the data can be processed; otherwise, the data is not processed, and the processable data enters A. i+1 In this process, data that is not processed will not enter A. i+1 Steps.

[0009] Furthermore, in step (1), the message bus can be any message system with distributed publish-subscribe capability, and the external cache system can be any memory storage system that stores data in a key-value manner.

[0010] Furthermore, in step (1), each piece of data has an event time field.

[0011] Furthermore, in steps (2) and (3), if the data exceeds the preset time and does not meet the processing requirements, the data is directly output to avoid data blocking in the internal process.

[0012] Furthermore, the preset time is set according to the actual data processing scenario.

[0013] Furthermore, each time the current stage takes the cursor written to the external cache system by the next stage to calculate the time baseline, and the last stage takes the cursor written to the external cache system by the first stage to calculate the time baseline, so that the whole process forms a closed loop, thus forming a mechanism for mutual cooperative waiting between each node.

[0014] The beneficial effects of this invention are: This invention provides a method for time-series data processing in distributed systems, significantly improving data timeliness. For example, it can more accurately identify risky transactions in anti-fraud scenarios, reduce user asset losses, and provide support for protecting account security. Attached Figure Description

[0015] Figure 1 This is a schematic diagram illustrating the implementation principle of a method for time-series data processing in a distributed system according to the present invention.

[0016] Figure 2 This is a comparison diagram between the method of the present invention and the conventional method. Detailed Implementation

[0017] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0018] like Figure 1 As shown, this invention provides a method for time-series processing of data in a distributed system. The method sorts the data across different nodes, ensuring that the data is processed sequentially between nodes. This effectively improves data timeliness without significantly reducing system throughput, enhancing data processing quality in scenarios with strict time-series requirements. Specifically, the method includes the following steps:

[0019] (1) The message bus of the computer system is used as the data source. At the same time, there are several nodes that subscribe to this message bus. Each node randomly obtains data from the message bus. The data between nodes is not arranged in chronological order. Each piece of data has an event time field. The message bus can be any message system with distributed publish-subscribe capability. The external caching system can be any memory storage system that uses key-value storage. There are several data processing links in each node. The data within a certain period of time is processed in the order of the links.

[0020] (2) Creation Stage A i The processing task: After the data obtained from the message bus in step (1) is received, it goes through stage A. i A i This represents the i-th processing step in the node, ending at step A. i The event time of the last data in the current batch is written to the external cache system, and this is recorded as node processing step A. i The vernier value.

[0021] (3) Creation Stage A i+1 Processing task: The process will proceed through stage A. i+1 The subsequent data processing step A i+1 The processing, in the final stage A i+1 The event time of the last data in the current batch is written to the external cache system, and this is recorded as node processing step A. i+1 The cursor value is determined by the time baseline. The cursor acquisition mechanism is as follows: each time, the current stage retrieves the cursor of the next stage to calculate the time baseline, and the last stage retrieves the cursor of the first stage to calculate the time baseline, so that the whole process forms a closed loop. This creates a mechanism for mutual coordination and waiting between each node.

[0022] (4) Steps (2) and (3) need to cross-check the cursor values ​​of each other to determine whether the data can be processed. The standard for data processing is: when the data enters A i+1 During the process, the time and A in the data will be used. i The minimum cursor value of the data is compared. If the event time of the data is less than this cursor value, it means that the data can be processed; otherwise, the data is not processed, and the processable data enters A. i+1 In this process, data that is not processed will not enter A. i+1 Steps.

[0023] In steps (2) and (3), if the data fails to meet the processing requirements after a preset time, forced processing logic needs to be executed to directly output the data, thus preventing data from being blocked in the internal process. The preset time is set according to the actual data processing scenario.

[0024] Before each node retrieves data from the message bus, it needs to obtain the cursor information saved by the last step of each node from the external cache system. The minimum value among these cursors is calculated as the time baseline for the current node to retrieve data. The node can then begin retrieving data from the message bus. The incoming data is judged: if the event time of the data is less than the time baseline, it passes smoothly; otherwise, it is not allowed to enter. Finally, the event time of the last piece of data is written to the external cache system as a cursor value for use by subsequent steps. This judgment method applies to all steps within a node. The key is that each step uses the cursor value of the next step as its time baseline, while the last step uses the cursor value of the first step, creating a mechanism of mutual coordination and waiting between steps, thus enabling time-series processing of out-of-order data. Example 1

[0025] like Figure 1 As shown in the figure, this embodiment provides a method for time-series data processing in a distributed system. This method can ensure that out-of-order data improves its timeliness after passing through the system. The specific implementation steps of this method are as follows:

[0026] The computer system's message bus serves as the data source, storing out-of-order data that needs to be processed. Each node contains two processing stages, Stage A and Stage B, each performing different tasks.

[0027] Start nodes 1 and 2 to continuously consume data from the message bus in step one.

[0028] Before the data in the consumer in step two needs to be transferred to task A, step three is executed to find the cursor written by task B in the external cache system. At this time, node 1 will get two cursors of task B, namely its own current cursor and the one of node 2. The smallest cursor value is obtained by comparing the two and applied to node 1 as the time baseline. Only data smaller than this time baseline can be transferred to task A. Similarly, node 2 executes the same logic.

[0029] After the data passes through task A, step four needs to be executed to write the event time of the last piece of data processed to the external cache system. Similarly, node 2 executes the same logic.

[0030] Before the data in step A needs to be transferred to task B in step five, step six is ​​executed to find the cursor written by step A in the external cache system. At this time, node 1 will get two cursors of step A, namely its own current cursor and the one of node 2. The smallest cursor value is compared between the two and applied to node 1 as the time baseline. Only data smaller than this time baseline can be transferred to task B. Similarly, node 2 executes the same logic.

[0031] After the data passes through task B, step seven needs to be executed to write the event time of the last piece of data processed to the external cache system. Similarly, node 2 executes the same logic. Example 2

[0032] like Figure 2 As shown, by comparing the method of the present invention with the traditional method when processing the same data, the method of the present invention achieves better data timeliness than the traditional method. This is verified through a series of experiments.

[0033] The experimental scenario is as follows: A set of data D1D2D3D4 is generated, containing an event time field, so that the experimental data is arranged in ascending order of time. Two processing nodes are started, and these two nodes continuously retrieve data from the message bus. The data is then shuffled into D3D1D4D2 and put into the message bus to simulate the situation where the order of data put into the message bus is inconsistent in actual production. The data is processed using traditional methods and the method of this invention, so that the final output result needs to achieve the desired order of D1D2D3D4.

[0034] The testing method is as follows:

[0035] Using the traditional method, node 1 receives data D1D2 and node 2 receives data D3D4. However, the timeline shows that the input data order is D3D1D4D2. Since there is no coordination mechanism, the final output is that node 1 outputs data D1D2 and node 2 outputs data D3D4. Superficially, the output data of node 1 and node 2 are in order, but overall, the final output is D3D1D4D2, which is still not in the expected order as the input data.

[0036] Experiments using the method of this invention show that node 1 receives data D1D2 and node 2 receives data D3D4. However, the timeline shows that the input data order is D3D1D4D2. After adding the coordination mechanism, the final data output is that node 1 outputs data D1D2 and node 2 outputs data D3D4. Overall, D3D4 will wait for D1D2 to complete before being output, thus ensuring that the data is output in order, which is the desired result.

[0037] The above embodiments are used to explain and illustrate the present invention, but not to limit the present invention. Any modifications and changes made to the present invention within the spirit and scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A method for time series processing of distributed system data, characterized in that, Specifically comprising the following steps: (1) taking the message bus of a computer system as a data source, while having several nodes subscribing to the message bus working, each node randomly obtaining data from the message bus all the time, and having several data processing links in each node, and the data in a certain period of time being processed in turn according to the order of the links; (2) Create link A i The processing task: after receiving the data obtained from the message bus in step (1), it goes through link A i , A i represents the i-th processing link in the node, and the event time of the last data in the current batch of data is written to the external cache system when link A i ends, denoted as the cursor value of the node processing link A i ; (3) Create a link A i+1 processing task: link A i processing of data after link A i+1 , write the event time of the last data in the current batch data to the external cache system when link A i+1 is ended, and mark it as the cursor value of link A i+1 processing node. (4) Step (2), step (3) need to cross check each other link cursor value as time baseline, so as to judge whether the data can be processed, the standard for data processing is: when the data enters into A i+1 link, compare the time in the data with the minimum cursor value of A i link, if the event time of the data is less than the cursor value, it means that the data can be processed, otherwise the data is not processed, the data that can be processed enters A i+1 link, the data that is not processed will not enter A i+1 link.

2. The method for timing processing of data for a distributed system according to claim 1, wherein, In the step (1), the message bus can be all message systems with distributed publishing and subscribing capabilities, and the external cache system can be all memory storage systems with kv storage mode.

3. The method for timing processing of data for a distributed system according to claim 1, wherein, In the step (1), each piece of data has an event time field.

4. The method for timing processing of data for a distributed system according to claim 1, wherein, In the steps (2) and (3), if the data does not meet the processing requirement within a preset time, the data is directly output, so as to avoid data jam in the internal process.

5. The method for timing processing of data for a distributed system according to claim 4, wherein, The preset time is set according to the actual data processing scene.

6. The method for time-series processing of data for a distributed system of claim 1, wherein, Each time the current link takes the next link to write the cursor in the external cache system to calculate the time baseline, and the last link takes the first link to write the cursor in the external cache system to calculate the time baseline, so that the whole process forms a closed loop, and thus a mutual collaborative waiting mechanism is formed between each node.

Images