High-concurrency time series data calculation and pushing method and system, medium and terminal

Abstract

The invention discloses a high-concurrency time series data calculation and pushing method and system, a medium and a terminal, and the method comprises the steps: carrying out the unified receiving of time series data based on MQ, carrying out the data filtering processing of the received time series data, and obtaining the processed time series data; performing slicing processing on the processed time sequence data to obtain sliced time sequence data, and calculating an alarm state of the sliced time sequence data; and assembling the time sequence data and the alarm state to generate a message, pushing the message to a page client connected with the current server instance, broadcasting the message to other server instances through MQ, and synchronously pushing the message to the corresponding page client. According to the method, on the premise that the data correctness is guaranteed, the data processing speed is greatly increased, and the data pushing delay is reduced.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a high-concurrency time series data calculation and push method, system, medium and terminal. Background Art

[0002] With the rapid development of industrial Internet technology, the application of industrial Internet platforms in the industrial field is becoming more and more extensive. The calculation and push of high-concurrency time series data has become an important part of the industrial Internet platform, especially in the data screen and monitoring screen of industrial configuration, which has extremely high requirements for the immediacy of operational technology (OT) data.

[0003] At present, existing data processing technologies often suffer from high latency and insufficient accuracy when facing high-concurrency time series data. This is because when traditional processing architectures process large amounts of data, data needs to be transmitted and processed through multiple links, and each link may become a bottleneck for latency. At the same time, due to the complexity of data processing, it is also easy to cause data errors or loss, thus affecting data accuracy.

[0004] Therefore, in order to meet the industrial Internet platform's requirements for immediacy and accuracy in high-concurrency time series data processing, there is an urgent need for an efficient time series data calculation and push method, system, medium and terminal that can greatly improve the speed of data processing and reduce the delay of data push while ensuring data correctness. Summary of the invention

[0005] In view of this, the purpose of the present invention is to overcome the defects in the prior art and provide a high-concurrency time series data calculation and push method, system, medium and terminal, which can greatly improve the speed of data processing and reduce the delay of data push while ensuring the correctness of the data.

[0006] The high-concurrency time series data calculation and push method of the present invention includes:

[0007] The time series data is uniformly received based on MQ, and the received time series data is filtered to obtain the processed time series data;

[0008] Slice the processed time series data to obtain sliced ​​time series data, and calculate the alarm status of the sliced ​​time series data;

[0009] Assemble time series data and alarm status to generate a message, push the message to the page client connected to the current server instance, broadcast the message to other server instances through MQ, and push it to the corresponding page client synchronously.

[0010] Furthermore, the received time series data is filtered, specifically including:

[0011] Check the timeliness and validity of data, and discard expired, disordered, and illegal data;

[0012] Check whether the data is bound to the page. Unbound data does not need to be pushed.

[0013] Furthermore, the processed time series data is sliced, specifically including:

[0014] Data slicing is performed according to the grouping settings bound to the data. Data in the same group is divided into the same slice. The sliced ​​data is distributed to the clustered server instances based on MQ for processing.

[0015] Further, the alarm status of the sliced ​​time series data is calculated, including:

[0016] Based on the Aviator engine, the alarm expression of the time series data is calculated. If the expression is calculated, the intermediate state data is obtained from the cache to determine whether the alarm triggering rules are met. If so, the alarm-related information is recorded and the life cycle of the alarm record is maintained;

[0017] The intermediate state data includes the time when the expression is continuously established and the number of times the expression is established; the alarm triggering rules include immediate triggering, duration triggering and number triggering.

[0018] Furthermore, based on the GuavaCache local cache framework and Redis distributed cache middleware, a secondary cache is built; the secondary cache is used to store time series data and alarm configuration information;

[0019] The secondary cache includes a local cache and a distributed cache; when performing data query, the data in the local cache is queried first. If a hit is not found, the distributed cache is queried. If a hit is not found again, the database or API interface is queried and the data is loaded into the local cache and the distributed cache.

[0020] A high-concurrency time series data calculation and push system, including a data filtering module, a data slicing module, an alarm calculation module and a data push module;

[0021] The data filtering module is used to perform data filtering processing on the received time series data to obtain processed time series data;

[0022] The data slicing module is used to slice the processed time series data to obtain sliced ​​time series data;

[0023] The alarm calculation module is used to calculate the alarm status of the sliced ​​time series data;

[0024] The data push module is used to assemble time series data and alarm status to generate a message, push the message to the page client connected to the current server instance, broadcast the message to other server instances through MQ, and synchronously push it to the corresponding page client.

[0025] Furthermore, the processed time series data is sliced, specifically including:

[0026] Data slicing is performed according to the grouping settings bound to the data. Data in the same group is divided into the same slice. The sliced ​​data is distributed to the clustered server instances based on MQ for processing.

[0027] Further, it also includes a data cache module;

[0028] The data cache module is used to build a secondary cache based on the GuavaCache local cache framework and the Redis distributed cache middleware; the secondary cache is used to store time series data and alarm configuration information;

[0029] The secondary cache includes a local cache and a distributed cache; when performing data query, the data in the local cache is queried first. If a hit is not found, the distributed cache is queried. If a hit is not found again, the database or API interface is queried and the data is loaded into the local cache and the distributed cache.

[0030] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned high-concurrency time series data calculation and push method.

[0031] An electronic terminal comprises a processor and a memory;

[0032] The memory is used to store computer programs, and the processor is used to execute the computer programs stored in the memory, so that the terminal executes the above-mentioned high-concurrency time series data calculation and push method.

[0033] The beneficial effects of the present invention are as follows: the present invention discloses a high-concurrency time series data calculation and push method, system, medium and terminal, which realizes the unified reception, slicing processing and broadcast distribution of time series data based on MQ, and calculates the alarm status of time series data based on the Aviator expression engine with high performance. For high-frequency data relied on in the process, a secondary cache is built based on Redis and Guava Cache to reduce IO and reduce query time consumption, and finally a persistent connection is established with the page client based on the WebSocket protocol to broadcast and push data to the page. The present invention realizes high-performance calculation and low-latency push of OT time series data, avoids data loss or accumulation caused by target instance failure during slicing, and supports broadcasting and pushing assembled data packets to all connected page clients in the server cluster deployment mode, avoiding repeated processing of each server instance and causing waste of resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The present invention will be further described below in conjunction with the accompanying drawings and embodiments:

[0035] Figure 1 This is a schematic diagram of the data calculation and push principle architecture of the present invention;

[0036] Figure 2 The figure is a schematic diagram of the data calculation and push execution flow of the present invention. DETAILED DESCRIPTION

[0037] The present invention is further described below in conjunction with the accompanying drawings, as shown in the drawings:

[0038] This embodiment discloses a high-concurrency time series data calculation and push method, including the following steps:

[0039] The time series data is uniformly received based on MQ, and the received time series data is filtered to obtain the processed time series data;

[0040] Slice the processed time series data to obtain sliced ​​time series data, and calculate the alarm status of the sliced ​​time series data;

[0041] Assemble time series data and alarm status to generate a message, push the message to the page client connected to the current server instance, broadcast the message to other server instances through MQ, and push it to the corresponding page client synchronously.

[0042] The present invention uses MQ to uniformly receive time series data, realizes centralized management and efficient processing of data, ensures the reliability and asynchrony of data reception, and reduces the risk of data loss. By broadcasting messages to other server instances through MQ, synchronous update and push of data are realized, the consistency and integrity of data are guaranteed, and multiple page clients can obtain the latest data in real time. Among them, MQ (Message Queue) is an asynchronous communication mechanism that allows different applications or processes to transmit messages through queues without directly communicating with each other.

[0043] In this embodiment, the received time series data is subjected to data filtering processing, which specifically includes:

[0044] Check the timeliness and validity of the data, and discard expired, disordered, and illegal data; if the timestamp of the data is too delayed compared to the current time, it is expired data; if data with an older timestamp is received after newer data, it is disordered data; if the data format is incorrect and / or the data is missing, it is illegal data.

[0045] Check whether the data is bound to the page. Unbound data does not need to be pushed. In some applications, data push is associated with a specific page or view configuration. Only data related to the current page or view needs to be pushed. Therefore, before processing the data, check whether the data is bound to the current page configuration. Unbound data does not need to be pushed, thereby reducing unnecessary network transmission and server load.

[0046] After the above filtering, the impact of expired, disordered, and illegal data on subsequent processing is avoided, the amount of data for subsequent processing is reduced, useless data is avoided from occupying server resources and causing interference to the correctness of data push, and the overall performance and response speed of data processing are improved.

[0047] In this embodiment, slicing the processed time series data specifically includes:

[0048] Data slicing is performed according to the grouping settings bound to the data. Data in the same group is divided into the same slice. The sliced ​​data is distributed to the clustered server instances based on MQ for processing. Data slicing refers to splitting a larger message or data into multiple smaller parts for more efficient transmission and processing in the message queue; splitting the message into multiple smaller parts on the message producer side, and adding appropriate identifiers to each part, such as a serial number, so that it can be reassembled into a complete message on the message consumer side. The specific implementation method varies depending on the actual working conditions, and message slicing can be implemented according to the corresponding API or tool.

[0049] Grouping can be based on data attributes or business logic needs, bringing related or similar data together for more efficient processing. Each slice is sent to the message queue as a message. Clustered server instances refer to the same service instances deployed on multiple servers, which together form a distributed service architecture. When the message queue receives slice messages, it distributes these messages to different service instances in the cluster for processing according to the load balancing strategy; thus, the parallel processing capabilities of the cluster can be used to speed up data processing. At the same time, since each slice contains data records of the same group, the integrity and consistency of the data can be guaranteed during the processing process.

[0050] Since subsequent alarm calculations only need to obtain data related to the same group, the data in the same group is resent to the same server instance for processing to reduce cache data redundancy in the server's local memory and improve data accuracy.

[0051] In this embodiment, calculating the alarm status of the sliced ​​time series data specifically includes:

[0052] Based on the Aviator engine, the alarm expression of the time series data is calculated. If the expression is calculated, the intermediate state data is obtained from the cache to determine whether the alarm triggering rules are met. If so, the alarm-related information is recorded and the life cycle of the alarm record is maintained; the intermediate state data includes the time when the expression is continuously established and the number of times the expression is established; the alarm triggering rules include immediate triggering, duration triggering, and number triggering.

[0053] The Aviator engine is a lightweight Java expression engine that can write expressions in string form and dynamically calculate the values ​​of these expressions at runtime. The expressions can be executed by calling the AviatorEvaluator API and the calculation results can be obtained. Among them, the alarm expression can be set according to the type or attribute of the time series data and the business logic under the current actual working conditions. For example, if a certain time series data is temperature, the alarm expression can be to determine whether the temperature exceeds the set threshold. Once the set threshold is exceeded, the expression calculation is established.

[0054] The alarm status is the status of judging whether the current time series data triggers the alarm condition according to the preset rules. Once the alarm is triggered, the relevant information of the alarm needs to be recorded, such as the alarm time, triggering conditions, duration, triggering times, etc.; this information needs to be persistently stored for subsequent query and analysis; and the life cycle of the alarm record, such as the alarm start time and end time, needs to be maintained.

[0055] Through the above processing, the Aviator engine is used to efficiently calculate alarm expressions and quickly identify potential problems. By introducing intermediate state data, the alarm trigger judgment is made more accurate and scientific. At the same time, the combination of multiple alarm trigger rules not only improves the flexibility of the alarm, but also effectively reduces false alarms and missed alarms, ensuring the accuracy and practicality of the alarm information. The maintenance of the alarm record life cycle is conducive to optimizing resource utilization and maintaining efficient operation of the alarm.

[0056] In this embodiment, the time series data and the corresponding alarm status are assembled to generate a message, thereby generating a message containing complete information for subsequent processing and push; the message is usually a structured data format, such as JSON or XML, which is convenient for transmission and parsing between different components.

[0057] In a distributed architecture, there may be multiple server instances running at the same time, and each instance may be connected to different clients. To ensure that all clients receive the latest data, messages can be broadcast to all server instances through message queues (MQ); ensuring that data can be reliably transmitted from one server instance to all other instances without requiring each instance to obtain data directly from the data source.

[0058] That is, based on WebSocket push to the page client connected to the current server instance, real-time data updates and alarm notifications can be achieved, and messages can be broadcast to other server instances through MQ to synchronously push data to all connected clients, so that the same data is processed only once, avoiding repeated processing by each server instance and causing waste of resources.

[0059] By distributing data to multiple server instances, you can balance the load and improve throughput and response speed. If a server instance fails, other instances can still receive data through MQ and continue to provide services to clients.

[0060] In this embodiment, during the processing of the above data, in order to reduce data query IO, improve query performance and support high concurrency, a secondary cache can be built based on the GuavaCache local cache framework and Redis distributed cache middleware, and the secondary cache can be used to store time series data and alarm configuration information; associated configuration information and real-time status data can also be stored, wherein the real-time status data includes real-time intermediate status data; the associated configuration information includes associated configuration information between data and pages.

[0061] The secondary cache includes a local cache for the primary cache and a distributed cache for the secondary cache. The primary cache usually has a smaller capacity but a very fast access speed. The secondary cache has a larger capacity but a relatively slow access speed. When the primary cache fails, data can be obtained from the secondary cache.

[0062] When querying data, the data in the local cache is queried first. If it misses, the distributed cache is queried. If it misses again, the database or API interface is queried and the data is loaded into the local cache and distributed cache. By temporarily storing the data that is relied on in the secondary cache, the access frequency to the database and external interface is reduced, and the time consumption of data query is reduced.

[0063] The present invention also relates to a high-concurrency time series data calculation and push system, which corresponds to the calculation and push method of the above embodiment and can be understood as a system for implementing the above method. The system includes a data filtering module, a data slicing module, an alarm calculation module and a data push module;

[0064] The data filtering module is used to perform data filtering processing on the received time series data to obtain processed time series data;

[0065] The data slicing module is used to slice the processed time series data to obtain sliced ​​time series data;

[0066] The alarm calculation module is used to calculate the alarm status of the sliced ​​time series data;

[0067] The data push module is used to assemble time series data and alarm status to generate a message, push the message to the page client connected to the current server instance, broadcast the message to other server instances through MQ, and synchronously push it to the corresponding page client.

[0068] The push method and system of the present invention solve the problems of local data redundancy and accuracy of multiple instances on the server side through data slicing, enabling the platform to complete the calculation and processing and real-time push of OT time series data with high performance and low latency, thereby improving the speed and accuracy of data processing, thereby meeting the high requirements of the industrial Internet platform for the immediacy and accuracy of OT data, providing strong technical support for the development of the industrial Internet platform, and further promoting the process of intelligent and efficient industrial production.

[0069] This embodiment further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, any one of the push methods in this embodiment is implemented.

[0070] This embodiment also provides an electronic terminal, including a processor and a memory;

[0071] The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the terminal executes any one of the push methods in this embodiment.

[0072] The computer-readable storage medium in this embodiment can be understood by a person skilled in the art that all or part of the steps of the above-mentioned method embodiments can be completed by hardware related to the computer program. The aforementioned computer program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

[0073] The electronic terminal provided in this embodiment includes a processor, a memory, a transceiver and a communication interface. The memory and the communication interface are connected to the processor and the transceiver and complete communication with each other. The memory is used to store computer programs, the communication interface is used to communicate, and the processor and the transceiver are used to run computer programs so that the electronic terminal executes each step of the above method.

[0074] In this embodiment, the memory may include a random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0075] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0076] In the above-described embodiments, although the invention has been described in conjunction with specific embodiments of the invention, many substitutions, modifications, and variations of these embodiments will be apparent to those of ordinary skill in the art based on the foregoing description. For example, other storage structures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed. Embodiments of the invention are intended to encompass all such substitutions, modifications, and variations that fall within the broad scope of the appended claims.

[0077] The same or similar parts between the various embodiments in this specification can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0078] The present invention can be used in many general or special computing system environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like.

[0079] The present invention may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0080] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. A high-concurrency time series data calculation and push method, characterized by: include: The time series data is uniformly received based on MQ, and the received time series data is filtered to obtain the processed time series data; Slice the processed time series data to obtain sliced ​​time series data, and calculate the alarm status of the sliced ​​time series data; Assemble time series data and alarm status to generate a message, push the message to the page client connected to the current server instance, broadcast the message to other server instances through MQ, and push it to the corresponding page client synchronously.

2. The high-concurrency time series data calculation and push method according to claim 1, characterized in that: The received time series data is filtered and processed, including: Check the timeliness and validity of data, and discard expired, disordered, and illegal data; Check whether the data is bound to the page. Unbound data does not need to be pushed.

3. The high-concurrency time series data calculation and push method according to claim 1, characterized in that: Slice the processed time series data, including: Data slicing is performed according to the grouping settings bound to the data. Data in the same group is divided into the same slice. The sliced ​​data is distributed to the clustered server instances based on MQ for processing.

4. The high-concurrency time series data calculation and push method according to claim 1, characterized in that: Calculate the alarm status of the sliced ​​time series data, including: Based on the Aviator engine, the alarm expression of the time series data is calculated. If the expression is calculated, the intermediate state data is obtained from the cache to determine whether the alarm triggering rules are met. If so, the alarm-related information is recorded and the life cycle of the alarm record is maintained; The intermediate state data includes the time when the expression is continuously established and the number of times the expression is established; the alarm triggering rules include immediate triggering, duration triggering and number triggering.

5. The high-concurrency time series data calculation and push method according to claim 1, characterized in that: Build a secondary cache based on the GuavaCache local cache framework and Redis distributed cache middleware; use the secondary cache to store time series data and alarm configuration information; The secondary cache includes a local cache and a distributed cache; when performing data query, the data in the local cache is queried first. If a hit is not found, the distributed cache is queried. If a hit is not found again, the database or API interface is queried and the data is loaded into the local cache and the distributed cache.

6. A high-concurrency time series data calculation and push system, characterized by: It includes data filtering module, data slicing module, alarm calculation module and data push module; The data filtering module is used to perform data filtering processing on the received time series data to obtain processed time series data; The data slicing module is used to slice the processed time series data to obtain sliced ​​time series data; The alarm calculation module is used to calculate the alarm status of the sliced ​​time series data; The data push module is used to assemble time series data and alarm status to generate a message, push the message to the page client connected to the current server instance, broadcast the message to other server instances through MQ, and synchronously push it to the corresponding page client.

7. The high-concurrency time series data calculation and push system according to claim 6, characterized in that: Slice the processed time series data, including: Data slicing is performed according to the grouping settings bound to the data. Data in the same group is divided into the same slice. The sliced ​​data is distributed to the clustered server instances based on MQ for processing.

8. The high-concurrency time series data calculation and push system according to claim 6, characterized in that: Also includes a data cache module; The data cache module is used to build a secondary cache based on the GuavaCache local cache framework and the Redis distributed cache middleware; the secondary cache is used to store time series data and alarm configuration information; The secondary cache includes a local cache and a distributed cache; when performing data query, the data in the local cache is queried first. If a hit is not found, the distributed cache is queried. If a hit is not found again, the database or API interface is queried and the data is loaded into the local cache and the distributed cache.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

10. An electronic terminal, characterized in that: including a processor and a memory; The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the terminal executes the method according to any one of claims 1 to 5.

Images