A distributed time series data interval analysis method and system

Abstract

The application discloses a kind of distributed time series data interval analysis method and system, and point data set is constructed hash table, from this;Carry out data piece division, carry out data piece division;In data piece, for each point, using a queue carries out data storage, and a batch of continuous time series values of point are contained in queue, and the number of continuous time series values defined in data piece level is inherited in the queue of data piece;Each data piece has a unique ID;Interval analysis rule includes a unique rule ID, and binding is done with data piece ID or point ID by the rule ID.Interval analysis rule of point is obtained according to the data piece and data subscript where receiving point is located, and calculation result is obtained by executing interval analysis rule, and output to upstream system.The interval analysis of the application is based on data piece, and data piece can be distributedly deployed, distributed analysis is realized, the timeliness and accuracy of time series data interval analysis are improved, and the pressure of database is also reduced.

Description

Technical Field

[0001] This invention belongs to the technical field of data processing and storage, and specifically relates to a distributed time-series data interval analysis method and system. Background Technology

[0002] In the process of acquiring industrial time-series data, it is often necessary to perform interval analysis on a batch of continuous data. For example, this involves analyzing the average or maximum values ​​of five consecutive voltage intervals. If the voltage exceeds a certain value twice consecutively, timely warnings should be issued. Existing technologies typically load the acquired values ​​into a time-series database, query the database to obtain the interval values, and then perform calculations and analysis. Since all calculations and analyses in existing technologies are performed within the time-series database, the pressure on the database becomes extremely high when there are too many data points to collect and analyze, and the collection frequency is too high. This can lead to problems such as untimely and inaccurate analysis results. The high load of query analysis can also affect the stability of the database, thus affecting the preservation of normal time-series data and potentially causing data loss. Summary of the Invention

[0003] The purpose of this invention is to provide a distributed time series data interval analysis method and system, which aims to solve the above-mentioned problems.

[0004] This invention mainly utilizes the following technical solutions:

[0005] A distributed time series data interval analysis method includes the following steps:

[0006] Step S100: Obtain the location dataset, which includes location ID and address information;

[0007] Step S200: Perform hash calculation on each point ID in the point dataset to construct a hash table, thereby each point will correspond to a piece of data in the hash table;

[0008] Step S300: Perform data segmentation. Based on the size of the point dataset, perform data segmentation, dividing a set number of points into one data segment;

[0009] Step S400: Define the data slice data structure. For each point in the data slice, use a queue to store data. The queue contains a batch of consecutive time-series values ​​of the point. The time-series values ​​are processed according to the first-in-first-out rule. The queue in the data slice inherits the number of consecutive time-series values ​​defined at the data slice level. Each data slice has a unique ID.

[0010] Step S500: Define interval analysis rules. Define analysis rules at the data slice level or at the data point level. Each interval analysis rule contains a unique rule ID, which is then bound to the data slice ID or the data point ID.

[0011] Step S600: Based on the data slice and data index of the receiving point, where the data index is the specific location identifier of the hash table where the point is located, obtain the interval analysis rule of the point, execute the interval analysis rule to obtain the calculation result, and output it to the upstream system.

[0012] The data index is the specific location identifier in the hash table where the point is located, obtained after hashing the point ID. This location identifier is unique.

[0013] To better implement the present invention, step S600 further includes the following steps:

[0014] Step S610: After receiving the data collected from the point, perform a hash operation on the point ID to obtain the data slice and data index where the point is located;

[0015] Step S620: Locate the point queue using the data slice and data index, and enqueue the new data to the queue;

[0016] Step S630: For the point and the data slice to which the point belongs, obtain the interval analysis rule, and use the value in the queue to apply the rule and calculate the analysis result;

[0017] Step S640: Dequeue the old points, along with the result data calculated in step S630, and output the data to the upstream system for further processing or storage. The point interval contains a fixed number of N points. Each newly added point is called a new point, and the points dequeued due to each new addition are called old points. After the points are dequeued, the structure calculated according to the interval rules is stored in the upstream system.

[0018] In step S200, a hash table {hash table name: HA, hash value: [{hash value: H1, index value M1}, {hash value: H2, index value: M2}]} has been constructed. Then, in step S300, the hash table is associated with data slices, resulting in the following data structure: {hash table name: HA, data slice: [{name: DP1, queue: Q1, data index set: {M1, M2, M3}]}. By hashing the point ID: HASH(point ID) => hash value(H1), the index value M1 corresponding to H1 is obtained through the hash table in step S200. Then, through the data structure in step S300, the data slice to which M1 belongs and the queue corresponding to the data slice are obtained.

[0019] To better realize the present invention, further, for scenarios that require edge computing, timely edge computing is performed on the data in step S640 to determine the analysis rule value corresponding to the analysis rule ID.

[0020] To better implement the present invention, the analysis rule value further includes any one or more of warning, discard, and data processing. The warning refers to issuing a warning at the edge when the analysis rule value exceeds a threshold; the discard refers to ignoring the data and not processing it when the analysis rule value meets certain conditions; and the data processing refers to adding other additional values.

[0021] To better implement this invention, furthermore, in step S640, a unified output data is defined in JSON format, with the specific format as follows:

[0022] {Point ID: F001, Outbound Data Timestamp: X, Outbound Data Value: x, Attached Value: [{Analysis Rule ID: R001, Analysis Rule Value: y}, {Analysis Rule ID: R002, Analysis Value: z}]}.

[0023] This invention mainly utilizes the following technical solutions:

[0024] A distributed time-series data interval analysis system includes a hashing unit, a sharding unit, a data structure definition unit, and an interval rule definition unit. The hashing unit is used to perform hash calculations on the point IDs in the point dataset and construct a hash table. The sharding unit is used to divide the point dataset into several data shards. The data structure definition unit is used to define the time-series value of each point in the data shard and give each data shard a unique ID. The interval rule definition unit is used to bind the rule ID with the data shard ID or the point ID.

[0025] To better realize the present invention, it further includes a point query unit, an execution unit, and an output unit. The point query unit is used to obtain the data slice and data index of the point; the execution unit is used to obtain the interval analysis rules of the point and execute them; and the output unit is used to output the analysis results of the execution unit to the upstream system.

[0026] The beneficial effects of this invention are as follows:

[0027] This invention moves the analysis of interval time series data off the database and processes it using a separate function, which improves the timeliness and accuracy of interval time series data analysis while reducing the pressure on the database. Secondly, interval analysis is based on data slices, which can be distributed to achieve distributed analysis or edge computing. Attached Figure Description

[0028] Figure 1 A flowchart defining the data sharding process;

[0029] Figure 2 This is a flowchart for data processing and analysis. Detailed Implementation

[0030] Example 1:

[0031] A distributed time series data interval analysis method, such as Figure 1 As shown, it includes the following steps:

[0032] Step S100: Obtain the location dataset, which includes location ID and address information;

[0033] Step S200: Perform hash calculation on each point ID in the point dataset to construct a hash table, thereby each point will correspond to a piece of data in the hash table;

[0034] Step S300: Perform data segmentation. Based on the size of the point dataset, perform data segmentation, dividing a set number of points into one data segment;

[0035] Step S400: Define the data slice data structure. For each point in the data slice, use a queue to store data. The queue contains a batch of consecutive time-series values ​​of the point. The time-series values ​​are processed according to the first-in-first-out rule. The queue in the data slice inherits the number of consecutive time-series values ​​defined at the data slice level. Each data slice has a unique ID.

[0036] Step S500: Define interval analysis rules. Define analysis rules at the data slice level or at the data point level. Each interval analysis rule contains a unique rule ID, which is then bound to the data slice ID or the data point ID.

[0037] Step S600: Based on the data slice and data index of the receiving point, obtain the interval analysis rule for the point, execute the interval analysis rule to obtain the calculation result, and output it to the upstream system. The data index is the specific location identifier in the hash table of the point obtained after hashing the point ID, and this location identifier is unique.

[0038] Preferably, such as Figure 2 As shown, step S600 includes the following steps:

[0039] Step S610: After receiving the data collected from the point, perform a hash operation on the point ID to obtain the data slice and data index where the point is located;

[0040] Step S620: Locate the point queue using the data slice and data index, and enqueue the new data to the queue;

[0041] Step S630: For the point and the data slice to which the point belongs, obtain the interval analysis rule, and use the value in the queue to apply the rule and calculate the analysis result;

[0042] Step S640: Perform a dequeue operation on the old location, along with the result data calculated in step S630, and output the data to the upstream system for further processing or storage.

[0043] The point interval contains a fixed N points. Each newly added point is called a new point, and the points that are dequeued due to each new addition are called old points. After a point is dequeued, the structure of the interval rule calculation is saved to the upstream system.

[0044] In step S200, a hash table {hash table name: HA, hash value: [{hash value: H1, index value M1}, {hash value: H2, index value: M2}]} has been constructed. Then, in step S300, the hash table is associated with data slices, resulting in the following data structure: {hash table name: HA, data slice: [{name: DP1, queue: Q1, data index set: {M1, M2, M3}]}. By hashing the point ID: HASH(point ID) => hash value(H1), the index value M1 corresponding to H1 is obtained through the hash table in step S200. Then, through the data structure in step S300, the data slice to which M1 belongs and the queue corresponding to the data slice are obtained.

[0045] Preferably, for scenarios requiring edge computing, edge computing is performed on the data in step S640 in a timely manner to determine the analysis rule value corresponding to the analysis rule ID.

[0046] Preferably, the analysis rule value includes any one or more of warning, discard, and data processing. The warning refers to issuing a warning at the edge when the analysis rule value exceeds a threshold; the discard refers to ignoring the data and not processing it when the analysis rule value meets certain conditions; and the data processing refers to adding other additional values.

[0047] Preferably, in step S640, the output data is defined in JSON format, with the following specific format:

[0048] {Point ID: F001, Outbound Data Timestamp: X, Outbound Data Value: x, Attached Value: [{Analysis Rule ID: R001, Analysis Rule Value: y}, {Analysis Rule ID: R002, Analysis Value: z}]}.

[0049] The interval analysis of this invention is based on data slices, which can be deployed in a distributed manner to achieve distributed analysis. This improves the timeliness and accuracy of interval analysis of time series data and reduces the pressure on the database. This invention moves the analysis of interval time series data off the database end, using a separate function for processing, which improves the timeliness and accuracy of interval analysis of time series data and reduces the pressure on the database. Furthermore, since interval analysis is based on data slices, data slices can be deployed in a distributed manner to achieve distributed analysis or edge computing.

[0050] Example 2:

[0051] A distributed time-series data interval analysis system includes a hashing unit, a sharding unit, a data structure definition unit, and an interval rule definition unit. The hashing unit is used to perform hash calculations on the point IDs in the point dataset and construct a hash table. The sharding unit is used to divide the point dataset into several data shards. The data structure definition unit is used to define the time-series value of each point in the data shard and give each data shard a unique ID. The interval rule definition unit is used to bind the rule ID with the data shard ID or the point ID.

[0052] Preferably, it further includes a point query unit, an execution unit, and an output unit. The point query unit is used to obtain the data slice and data index of the point; the execution unit is used to obtain the interval analysis rules of the point and execute them; and the output unit is used to output the analysis results of the execution unit to the upstream system.

[0053] The interval analysis of this invention is based on data slices, which can be deployed in a distributed manner to achieve distributed analysis. This improves the timeliness and accuracy of interval analysis of time series data and reduces the pressure on the database. This invention moves the analysis of interval time series data off the database end, using a separate function for processing, which improves the timeliness and accuracy of interval analysis of time series data and reduces the pressure on the database. Furthermore, since interval analysis is based on data slices, data slices can be deployed in a distributed manner to achieve distributed analysis or edge computing.

[0054] Example 3:

[0055] A distributed time series data interval analysis method, such as Figure 1 , Figure 2 As shown, it includes the following steps:

[0056] 1) Construction Phase: The data preparation and definition work required before the practical application of this invention mainly includes the following processes:

[0057] Step 1: Obtain the location dataset. The location dataset is usually defined by the user in the configuration and is used to specify the location data of certain equipment that needs to be collected during the production process. It mainly contains information such as the location ID and address. This invention mainly uses the location ID for subsequent processing.

[0058] Step 2: Perform hash calculation on each point ID in the point dataset to construct a hash table. Each point will then correspond to a single data entry in the hash table.

[0059] Step 3: Divide the data into data shards. Based on the size of the dataset, divide the data into shards appropriately. Typically, 1000 data points can be divided into one shard. If the dataset has 10,000 data points, then 10 shards should be used. The shard size setting also needs to be adjusted according to the performance requirements of the analysis and the availability of resources.

[0060] Step 4: Define the data slice data structure. For each point in the data slice, a queue is used for data storage. The queue contains a batch of consecutive time-series values ​​for that point. The time-series values ​​are processed according to the First-In-First-Out (FIFO) rule. The number of consecutive time-series values ​​in the batch is defined by the data slice, and the queue in the data slice inherits the number of consecutive time-series values ​​defined at the data slice level. Each element in the queue contains the following attributes: point ID, timestamp, and value. Each data slice has a unique ID.

[0061] Step 5: Define interval analysis rules. This can be done at the data slice level or the data point level. Interval analysis rules can use conventional methods such as maximum, minimum, average, and outlier frequency within the analysis interval, or other customized algorithms. Each interval analysis rule contains a unique rule ID, which is then bound to the data slice ID or data point ID.

[0062] 2) Application stage: How to process interval time series data in practical applications of this invention:

[0063] Step 1: After receiving the data collected from the data points, perform a hash operation on the data point ID to obtain the data slice and data index of the data point.

[0064] Step 2: Locate the point queue using the data slices and data indices, and enqueue the new data to the queue.

[0065] Step 3: For the given location and the data segment it belongs to, obtain the interval analysis rule, and apply the rule using the values ​​in the queue to calculate the analysis result. For example, to calculate the average of the three most recent collected values, you can execute Avg(Qv1,Qv2,Qv3,)=Ravg

[0066] Step 4: Dequeue the old locations, along with the results calculated in Step 3 above. This invention defines the unified output data in JSON format, as follows:

[0067] {Point ID: F001, Outbound Data Timestamp: 1671076679, Outbound Data Value: 0.25, Attached Value: [{Analysis Rule ID: R001, Analysis Rule Value: 0.26}, {Analysis Rule ID: R002, Analysis Value: 0.78}]}

[0068] Step 5: This step is optional. For scenarios requiring edge computing, timely edge computing can be performed on the data from Step 4 to determine the analysis rule value corresponding to the analysis rule ID. The following three different operations can be performed:

[0069] 1) Early Warning: When the analysis rule value exceeds the threshold, an early warning is issued at the edge.

[0070] 2) Discard: When the analysis rule value meets certain conditions, ignore the data and do not proceed with the following steps.

[0071] 3) Data processing: Additional supplementary values ​​can be added.

[0072] Step 6: Output the data from Step 4 to the upstream system for further processing or storage.

[0073] The interval analysis of this invention is based on data slices, which can be deployed in a distributed manner to achieve distributed analysis. This improves the timeliness and accuracy of interval analysis of time series data and reduces the pressure on the database. This invention moves the analysis of interval time series data off the database end, using a separate function for processing, which improves the timeliness and accuracy of interval analysis of time series data and reduces the pressure on the database. Furthermore, since interval analysis is based on data slices, data slices can be deployed in a distributed manner to achieve distributed analysis or edge computing.

[0074] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications or equivalent changes made to the above embodiments based on the technical essence of the present invention shall fall within the protection scope of the present invention.

Claims

1. A distributed time-series data interval analysis method, characterized in that, Includes the following steps: Step S100: Obtain the location dataset, which includes location ID and address information; Step S200: Perform hash calculation on each point ID in the point dataset to construct a hash table, thereby each point will correspond to a piece of data in the hash table; Step S300: Perform data sharding. Based on the size of the point dataset, perform data sharding, dividing a set number of points into one data shard, and deploying the data shards in a distributed manner. Step S400: Define the data slice data structure. For each point in the data slice, use a queue to store data. The queue contains a batch of consecutive time-series values ​​of the point. The time-series values ​​are processed according to the first-in-first-out rule. The queue in the data slice inherits the number of consecutive time-series values ​​defined at the data slice level. Each data slice has a unique ID. Step S500: Define interval analysis rules. Define analysis rules at the data slice level or at the data point level. Each interval analysis rule contains a unique rule ID, which is then bound to the data slice ID or the data point ID. Step S600: Based on the data slice and data index of the receiving point, where the data index is the specific location identifier of the hash table where the point is located, obtain the interval analysis rule of the point, execute the interval analysis rule to obtain the calculation result, and output it to the upstream system.

2. The distributed time-series data interval analysis method according to claim 1, characterized in that, Step S600 includes the following steps: Step S610: After receiving the data collected from the point, perform a hash operation on the point ID to obtain the data slice and data index where the point is located; Step S620: Locate the point queue using the data slice and data index, and enqueue the new data to the queue; Step S630: For the point and the data slice to which the point belongs, obtain the interval analysis rule, and use the value in the queue to apply the rule and calculate the analysis result; Step S640: Perform a dequeue operation on the old location, along with the result data calculated in step S630, and output the data to the upstream system for further processing or storage.

3. The distributed time-series data interval analysis method according to claim 2, characterized in that, For scenarios requiring edge computing, perform timely edge computing on the data from step S640 and determine the analysis rule value corresponding to the analysis rule ID.

4. The distributed time-series data interval analysis method according to claim 3, characterized in that, The analysis rule value includes any one or more of the following: warning, discard, and data processing. The warning refers to issuing a warning at the edge when the analysis rule value exceeds a threshold. The discard refers to ignoring the data and not processing it when the analysis rule value meets certain conditions. The data processing refers to adding other additional values.

5. A distributed time-series data interval analysis method according to claim 2, characterized in that, In step S640, a unified output data is defined in JSON format, with the specific format as follows: {Point ID: F001, Outbound Data Timestamp: X, Outbound Data Value: x, Attached Value: [{Analysis Rule ID: R001, Analysis Rule Value: y}, {Analysis Rule ID: R002, Analysis Value: z}]}.

6. A distributed time-series data interval analysis system, used to implement the distributed time-series data interval analysis method as described in claim 1; characterized in that, It includes a hashing unit, a sharding unit, a data structure definition unit, and an interval rule definition unit. The hashing unit is used to perform hash calculations on the point IDs in the point dataset and construct a hash table. The sharding unit is used to divide the point dataset into several data shards. The data structure definition unit is used to define the time-series value of each point in the data shard and to give each data shard a unique ID. The interval rule definition unit is used to bind the rule ID to the data slice ID or the point ID.

7. A distributed time-series data interval analysis system according to claim 6, characterized in that, It also includes a point query unit, an execution unit, and an output unit. The point query unit is used to obtain the data slice and data index where the point is located; the execution unit is used to obtain the interval analysis rules of the point and execute them; and the output unit is used to output the analysis results of the execution unit to the upstream system.

Images