A high-concurrency time-series data processing method

Abstract

The application discloses a kind of high concurrent time series data processing methods, comprising: for TPC terminal allocation unique identifier, obtain several communication identifications;Edge computing node is deployed in cloud, and according to the number of TPC terminal setting calculation group, thread is opened and thread pool is built;Traversal calculation group, based on thread ID and communication identification constructs hash table, so that thread pool, hash table and calculation group correspond;Determine current calculation group, and the high concurrent time series data of TPC terminal is encrypted and transmitted to current calculation group and is decrypted;Current calculation group thread pool is received with decryption data, and data is transmitted to corresponding thread according to hash table and is handled;Whether the CPU usage of current calculation group reaches preset threshold value is judged;Yes then switch next calculation group and repeat the above steps, otherwise the data after processing is encrypted and transmitted to user end.The application relates to the field of high concurrent time series data processing, and solves the problems that the order of high concurrent time series data processing is difficult to guarantee and the load is unbalanced.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of fire monitoring, and relates to high-concurrency data processing technology, in particular to a high-concurrency time-series data processing method. BACKGROUND

[0002] In the field of fire monitoring, fire monitoring needs to process a large amount of time-series data from different monitoring points in real time and accurately, so as to timely discover fire and make early warning. High-concurrency processing capability can ensure that the system can still operate stably under high load, without missing any important fire monitoring information. This plays a crucial role in improving the efficiency of fire prevention and control and reducing the loss caused by fire. Therefore, in the field of fire monitoring, using an efficient high-concurrency time-series data processing method is the key to ensuring system stability and reliability of detection results.

[0003] Most existing data processing methods rely on single-thread processing. When facing large-scale concurrency scenarios, the processing capability will be severely limited, resulting in that the data processing speed cannot meet the demand. In order to improve the processing capability, some systems use multi-thread processing method. However, in a high-concurrency environment, a large amount of data sources continuously flow into the system, and various threads simultaneously process these data. Since the execution of threads is concurrent, there is a lack of effective coordination mechanism between them to ensure that data is processed in strict time order, so it is difficult to guarantee the data processing sequence. For example, different threads may simultaneously process data with similar time stamps, but due to the scheduling strategy of the operating system and the uncertainty of thread execution, the output sequence of these data after processing may be inconsistent with their original time sequence. This will have a serious impact on application scenarios that require strict time sequence, such as real-time fire monitoring systems, etc.

[0004] Secondly, although multi-thread processing aims to improve the overall processing capability, in actual operation, load imbalance may occur. Some threads may be overloaded due to being allocated a large number of complex tasks or frequent access to shared resources, while other threads may be in a relatively idle state. This load imbalance not only reduces the overall processing efficiency of the system, but also may cause some threads to have performance degradation or even crash due to long-time high-load operation. SUMMARY

[0005] The present application aims to at least solve one of the technical problems existing in the prior art; for this purpose, the present application proposes a high-concurrency time-series data processing method for solving the problems of difficult-to-guarantee sequence and load imbalance in high-concurrency time-series data processing in the field of fire monitoring.

[0006] To achieve the above-mentioned purpose, the present application provides a high-concurrency time-series data processing method, comprising:

[0007] S1, assign a unique identifier to each TPC terminal, obtaining a plurality of communication identifiers;

[0008] S2, deploy a plurality of edge computing nodes in the cloud, set a plurality of initial computing groups according to the number of TPC terminals, and start a plurality of threads on the plurality of edge computing nodes, build a thread pool based on the plurality of threads, and obtain a thread pool of the plurality of initial computing groups;

[0009] S3, traverse the plurality of initial computing groups, obtain the thread ID of each initial computing group, build a hash table based on the thread ID and the plurality of communication identifiers, and obtain a plurality of computing groups corresponding to the thread pool, the hash table and the computing group;

[0010] S4, determine the current computing group, and after encrypting the high-concurrency time sequence data of the plurality of TPC terminals, transmit the decrypted data to the current computing group for decryption to obtain the decrypted data;

[0011] S5, receive the decrypted data using the thread pool of the current computing group, and according to the hash table of the current computing group, transmit the decrypted data to the corresponding thread for processing to obtain the data processed by the plurality of threads;

[0012] S6, determine whether the CPU usage of the plurality of edge computing nodes in the current computing group is equal to the preset threshold; if yes, traverse the next computing group, mark it as the current computing group, and repeat S4-S6; otherwise, encrypt and transmit the data processed by the plurality of threads to the corresponding user terminal.

[0013] Based on the above technical process, by assigning a unique identifier to the TPC terminal and deploying edge computing nodes in the cloud to form a plurality of initial computing groups, starting threads in each computing group to build a thread pool, and one-to-one corresponding the thread pool and the communication identifier through the hash function, a high-concurrency time sequence processing method is obtained, which can guarantee the data processing order and processing efficiency. In addition, this method also uses data encryption technology to ensure the security of data transmission, and dynamically adjusts the processing task of the computing group according to the CPU usage of the edge computing node to realize load balancing, which increases the efficiency, security and reliability of the high-concurrency time sequence data processing method, and improves the utilization rate of resources and the continuity of data processing.

[0014] Further, the method for assigning a unique identifier to each TPC terminal comprises:

[0015] S11, obtain the MAC address of the plurality of TPC terminals through the network management protocol, and obtain the plurality of MAC addresses;

[0016] S12, extract a preset number of hexadecimal numbers of the plurality of MAC addresses and splice them to obtain a plurality of unique identifiers.

[0017] Further, the setting of a plurality of initial calculation groups according to the number of TPC terminals comprises:

[0018] S2-11, initializing an initial calculation group, marked as the current initial calculation group;

[0019] S2-12, traversing a plurality of edge computing nodes, calculating the product of the CPU core number of the current node and the preset unit thread number to obtain the support thread number;

[0020] S2-13, assigning the current calculation node to the current initial calculation group, and accumulating the thread number in the current initial calculation group according to the support thread number;

[0021] S2-14, determining whether the thread number in the current initial calculation group is greater than or equal to the required thread number; if yes, obtaining an initial calculation group and jumping to S2-16; otherwise, jumping to S2-15;

[0022] S2-15, determining whether there is a next calculation node; if yes, marking the next calculation node as the current node and jumping to S2-12; otherwise, closing the current calculation group to obtain a plurality of initial calculation groups;

[0023] S2-16, determining whether there is a next calculation node; if yes, marking the next calculation node as the current node and jumping to S2-11; otherwise, obtaining a plurality of initial calculation groups.

[0024] Further, the construction of a plurality of thread pools based on a plurality of threads comprises:

[0025] S2-21, traversing a plurality of calculation groups, starting a plurality of threads in the current calculation group which is the same as the required thread number to obtain a plurality of threads of the current calculation group;

[0026] S2-22, constructing a thread pool using a thread pool framework and putting the plurality of threads of the current calculation group into the thread pool to obtain a thread pool;

[0027] S2-23, marking the next calculation group as the current calculation group, repeating S2-21 to S2-23 until the traversal is completed to obtain a plurality of thread pools.

[0028] The setting of the initial calculation group provides an important basis for load balancing. By grouping according to the number of TPC terminals and the capacity of edge computing nodes, it can be ensured that each calculation group has sufficient processing capacity to handle the data processing tasks allocated to it, avoiding waste and overload of resources and improving overall processing efficiency.

[0029] Further, the construction of a hash table based on thread ID and a plurality of communication identifiers comprises:

[0030] S3-1, initialize a hash table; wherein the values in the hash table are thread IDs, ranging from [0, N-1], N representing the number of TPC terminals;

[0031] S3-2, perform hash operation on a plurality of communication identifiers to obtain a plurality of initial hash values;

[0032] S3-3, perform AND operation and modulo operation on the plurality of initial hash values to obtain a plurality of hash values with a mapping range of [0, N-1];

[0033] S3-4, mark the plurality of hash values as keys and store them in the hash table to obtain a hash table with one-to-one correspondence between the communication identifiers and the threads.

[0034] The hash table is constructed by using AND operation and modulo operation to establish one-to-one correspondence between the communication identifiers and the threads, because in the case of high concurrency, the data of different TPC terminals will flow into the system at an extremely fast speed, and through the hash table, the data can be quickly determined to which thread it should be allocated for processing according to the communication identifier, so as to ensure that the data of each terminal can accurately find the corresponding processing path.

[0035] Further, the encryption of the high-concurrency time-series data of the plurality of TPC terminals comprises:

[0036] S4-11, traverse the plurality of TPC terminals, and split the data of the current TPC terminal according to a preset number of characters to obtain a plurality of packets with timestamps;

[0037] S4-12, generate a packet ID according to the communication identifier of the current TPC terminal, the timestamps of the plurality of packets, and the positions of the plurality of packets in the data of the current TPC terminal to obtain the packet IDs of the plurality of packets;

[0038] S4-13, arrange the plurality of packets of the current TPC terminal in disorder, and encrypt the plurality of packets arranged in disorder by using an AES algorithm to obtain the encrypted high-concurrency time-series data of the current TPC terminal;

[0039] S4-14, mark the next TPC terminal as the current TPC terminal, and repeat S4-1 to S4-4 until the traversal is completed to obtain a plurality of encrypted high-concurrency time-series data.

[0040] Traverse the TPC terminals, split the data according to a preset number of characters to obtain packets with timestamps; generate a packet ID according to the terminal communication identifier, the packet timestamp, and the position; arrange the packets in disorder and encrypt them by using an AES algorithm to obtain the encrypted data of the current terminal; mark the next terminal and repeat the operation until all the terminals are traversed to obtain a plurality of encrypted high-concurrency time-series data.

[0041] Further, the transmission to the current computing group for decryption comprises:

[0042] S4-21, decrypting the encrypted data using the AES algorithm to obtain a plurality of sub-packets;

[0043] S4-22, obtaining the packet IDs of the plurality of sub-packets, reordering the plurality of sub-packets according to the packet IDs to obtain high-concurrency time sequence data of the plurality of TPC terminals.

[0044] In a high-concurrency environment, a large amount of data is transmitted in the network, and these data may contain sensitive information. Therefore, the high-concurrency time sequence data is segmented according to a preset number of characters to obtain a plurality of data packets, and the plurality of data packets are encrypted using the AES algorithm and then transmitted in random order to the cloud, thereby increasing the security and randomness of data transmission and ensuring the integrity of the time sequence data.

[0045] Further, the transmission of the decrypted data to the corresponding thread for processing comprises:

[0046] S5-11, deploying a fire monitoring model on each edge computing node in the current computing group to obtain a shared model of the plurality of edge computing nodes; wherein the fire monitoring model is constructed based on a machine learning algorithm;

[0047] S5-12, obtaining the high-concurrency time sequence data received by the current thread, including the communication identifier of the TPC terminal, the timestamp, the smoke concentration, the temperature, the flame detection condition, and the gas concentration; wherein the flame detection condition is represented by (0, 1), 0 indicating that no flame is detected and 1 indicating that there is a flame;

[0048] S5-13, inputting the smoke concentration, temperature, flame detection condition, and gas concentration of the same timestamp into the shared model of the current edge computing node to obtain a fire prediction level;

[0049] S5-14, outputting the high-concurrency time sequence data received by the current thread and the fire prediction level together to obtain data processed by the current thread.

[0050] Further, the fire monitoring model is constructed based on a machine learning algorithm, comprising:

[0051] S5-111, obtaining a plurality of smoke concentration, temperature, flame detection condition, gas concentration, and fire level data from a plurality of fire record data, and organizing them into a data set;

[0052] S5-112, constructing a neural network model using a machine learning algorithm, inputting the data set into the neural network model for training and verification, and iteratively optimizing according to the verification result;

[0053] S5-113, save the neural network model with the highest verification accuracy to obtain a fire monitoring model with the input of smoke concentration, temperature, flame detection condition and gas concentration and the output of fire prediction level.

[0054] Further, the encrypted transmission of the data processed by the plurality of threads to the corresponding user terminal comprises:

[0055] S5-21, obtain the communication identifier of the TPC terminal corresponding to the current thread to obtain the current communication identifier;

[0056] S5-22, filter the user ID matched with the current communication identifier from the user database; wherein the user database is deployed in the cloud and contains the user ID, user terminal address and communication identifier;

[0057] S5-23, encrypt and transmit the data processed by the current thread to the corresponding user terminal according to the user ID.

[0058] Compared with the prior art, the beneficial effects of the present application are:

[0059] By allocating a unique communication identifier to the TPC terminal and using a custom hash function to map the communication identifier to a unique number corresponding to the thread, it is ensured that the data of each terminal can be correctly tracked and processed and transmitted in time sequence, ensuring the sequence and timing of data processing;

[0060] By deploying edge computing nodes in the cloud and dividing them into a plurality of computing groups, high-concurrency data is processed in units of computing groups, effectively preventing the problem of task termination and reduced processing efficiency caused by excessive load of a certain computing node, ensuring the continuity and stability of data processing;

[0061] By packet processing high-concurrency time-series data, using AES encryption algorithm and out-of-order transmission of data packets, double encryption is realized to improve the security of data in the network transmission process. BRIEF DESCRIPTION OF DRAWINGS

[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiment or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0063] Figure 1 The technical flow chart of the high-concurrency time-series data processing method provided by the present application;

[0064] Figure 2A method flowchart for setting an initial calculation group is provided for the present application.

[0065] Figure 3 A method flowchart for encrypting high-concurrency time-series data is provided for the present application. DETAILED DESCRIPTION

[0066] The technical solutions of the present application will be described clearly and completely below in conjunction with embodiments. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0067] Please refer to Figure 1 - Figure 3 The embodiment of the present application provides a high-concurrency time-series data processing method, comprising:

[0068] S1, assigning a unique identifier to a plurality of TPC terminals to obtain a plurality of communication identifiers;

[0069] S2, deploying a plurality of edge computing nodes in the cloud, setting a plurality of initial calculation groups according to the number of TPC terminals, and starting a plurality of threads on a plurality of edge computing nodes in each calculation group, constructing a thread pool based on a plurality of threads to obtain a plurality of initial calculation group thread pools;

[0070] S3, traversing a plurality of initial calculation groups, obtaining thread IDs of thread pools of each calculation group, constructing a hash table based on thread IDs and a plurality of communication identifiers, and obtaining a plurality of calculation groups corresponding to the thread pool, the hash table and the calculation group;

[0071] S4, obtaining a current calculation group from a plurality of calculation groups, and encrypting high-concurrency time-series data of a plurality of TPC terminals, transmitting to the current calculation group for decryption to obtain a plurality of decrypted data;

[0072] S5, receiving the plurality of decrypted data using the thread pool of the current calculation group, and transmitting the plurality of decrypted data to the corresponding thread for processing according to the hash table of the current calculation group to obtain a plurality of thread-processed data;

[0073] S6, determining whether the CPU usage of a plurality of edge computing nodes in the current calculation group is equal to a preset threshold; if yes, traversing the next calculation group to mark it as the current calculation group, and repeating S4-S6; if no, transmitting the plurality of thread-processed data to the corresponding user terminal after data encryption.

[0074] In this embodiment, in order to ensure that the data of each terminal can be correctly identified and tracked in the processing process, a unique communication identifier is assigned to each TPC terminal; specifically, the MAC address of a plurality of TPC terminals is obtained through a network management protocol, and then a preset number of hexadecimal numbers of each MAC address are extracted and spliced to obtain a unique identifier. For example, the MAC address of an existing TPC terminal is 00:1A:2B:3C:4D:5E, and 6-bit hexadecimal numbers are extracted and spliced to obtain the unique communication identifier of the terminal: 001A2B.

[0075] Next, in order to reduce the delay of data transmission and improve the real-time performance of data processing, a plurality of edge computing nodes are deployed in the cloud to fully utilize the distributed computing power of the cloud and provide powerful computing power for the processing of high-concurrency time-series data. In this embodiment, a plurality of edge computing nodes are assigned to a plurality of computing groups in the form of computing groups, and high-concurrency data is processed in units of computing groups. When the load of a node in a computing group is too heavy, the data will be quickly transferred to the next computing group for further data processing, preventing the problem of task termination or low processing efficiency caused by excessive load.

[0076] Specifically, an initial computing group is first initialized, and a plurality of edge computing nodes are traversed to calculate the product of the CPU core number of the current node and the preset unit thread number to obtain the support thread number. Then, the current computing node is assigned to the current initial computing group, and the thread number in the current initial computing group is accumulated according to the support thread number. Then, it is judged whether the thread number in the current computing group is greater than or equal to the required thread number. If yes, a computing group is obtained; if no, the next computing node is marked as the current node, and the edge computing nodes are repeatedly added to a computing group until the accumulated thread number is greater than or equal to the ratio of the number of TPC terminals to the preset number, i.e., whether it is greater than or equal to the required thread number, to ensure that each TPC terminal has a thread corresponding to it. Then, the above steps are repeated until all edge computing nodes are assigned to a computing group, thereby obtaining a plurality of initial computing groups. Finally, in each computing group, the same number of threads as the required thread number is started, and the threads are put into a thread pool for management, thereby obtaining a plurality of initial computing groups containing threads. If the last initial computing group has an accumulated thread number less than the number of TPC terminals, the edge computing nodes in the computing group are closed and not used.

[0077] For example, assuming that there are 500 TPC terminals, the preset number is 5, i.e., one thread ID corresponds to 5 terminals, and the required thread number is 100, and the cloud deploys 5 edge computing nodes, and the preset unit thread number is 10.

[0078] First, the first computing group is initialized and marked as the current computing group.

[0079] Then traverse the first edge computing node, assuming that the node has 4 CPU cores, the supported thread number is 4x10=40; the node is classified into the current computing group, and the thread number in the current computing group is 40;

[0080] Determine whether the thread number 40 in the current computing group is greater than the required thread number 100, the result is no, and then further determine that there is a next computing node, and mark the next computing node as the current node;

[0081] Assuming that the second edge computing node has 6 CPU cores, the supported thread number is 6x10=60. The node is classified into the current computing group, and the thread number in the current computing group becomes 40+60=100;

[0082] Determine whether the thread number 100 in the current computing group is greater than or equal to the required thread number 100, the result is yes, and an initial computing group is obtained;

[0083] Further determine that there is a next computing node, then mark the next computing node as the current node, and initialize the second computing group, which is marked as the current computing group;

[0084] Finally, repeat the above process until all edge computing nodes are traversed, obtain a plurality of initial computing groups, and start a plurality of threads in the terminal number in the plurality of computing groups, and construct a thread pool according to the threads to obtain a plurality of initial computing groups containing threads.

[0085] In order to ensure the order of data processing, a custom hash function is provided in this embodiment, which can map the communication identifier to a unique number as a key value stored in the hash table, realizing one-to-one correspondence between the terminal and the thread, thereby improving the efficiency of data processing;

[0086] Specifically, first, initialize a hash table for storing thread objects in a computing group; the value in the hash table is thread ID, ranging from 0 to N-1, N representing the number of TPC terminals; then hash a plurality of communication identifiers to obtain a plurality of initial hash values; then use the custom hash function to perform AND operation and modulo operation on the plurality of initial hash values, map the initial hash values to [0, N-1] to obtain a plurality of hash values; finally, mark the plurality of hash values as keys and store them in the hash table to obtain a hash table with one-to-one correspondence between the communication identifier and the thread;

[0087] In this embodiment, the code of the exemplary custom hash function is as follows:

[0088] private static int customHash(String str){

[0089] long hash = 5381;

[0090] for (int i = 0; i < str.length(); i++) {

[0091] hash = ((hash << 5) + hash) + str.charAt(i);

[0092] / / limit hash to 32 bits

[0093] hash = hash & 0xFFFFFFFFL;

[0094] hash = hash % (1000000L * 8);

[0095] hash = hash % 8;

[0096] return (int) hash;

[0097] In a high-concurrency environment, a large amount of data is transmitted in the network, and if encryption is not performed, the security of the data will face great risks. In the embodiment, the high-concurrency data is processed by packeting, and the packeted data is encrypted by AES, so as to realize double encryption guarantee and improve the security of the data.

[0098] Specifically, the encryption process in the embodiment includes the following steps:

[0099] Step one, traverse a plurality of TPC terminals, and divide the data of the current TPC terminal according to a preset number of characters (for example, 50 characters) to obtain a plurality of sub-tables with timestamps;

[0100] Step two, generate a packet ID according to the identifier of the current TPC terminal, the timestamps of the plurality of packets, and the positions of the plurality of packets in the data of the current TPC terminal, to obtain the packet ID of the plurality of packets; for example, the communication identifier of a terminal is 001A2B, the timestamp is: 2024-10-14T10:00:00.000, and the position of a packet is 1, so the obtained packet ID is: 001A2B20241014T100000001;

[0101] Step three, arrange the plurality of packets of the current TPC terminal in disorder, and encrypt the plurality of packets arranged in disorder by using an AES algorithm, to obtain high-concurrency time sequence data of the current TPC terminal after encryption;

[0102] Step four, mark the next TPC terminal as the current TPC terminal, repeat steps one to four, until the traversal is completed, to obtain the high-concurrency time sequence data after encryption.

[0103] Similarly, the corresponding decryption process is to decrypt the encrypted data according to the package ID and the AES decryption algorithm.

[0104] Through the above steps, high-concurrency data from the terminal can be safely transmitted to the computer in the cloud for data processing. Next, the decrypted data is received by the thread pool of the current computing group, and then the decrypted data is transmitted to the corresponding thread for processing according to the hash table of the computing group;

[0105] In this embodiment, high-concurrency time-series data for processing fire monitoring is taken as an example; first, a fire monitoring model is deployed on each edge computing node in the current computing group to obtain a shared model; then, high-concurrency time-series data received by the current thread is obtained, including the communication identifier of the TPC terminal, the timestamp, the smoke concentration, the temperature, the flame detection condition, and the gas concentration; the flame detection condition is represented by (0, 1), 0 indicating that no flame is detected, and 1 indicating that there is flame; then, the smoke concentration, the temperature, the flame detection condition, and the gas concentration of the same timestamp are input into the shared model to obtain the fire prediction level; finally, the high-concurrency time-series data received by the current thread and the fire prediction level are output together to obtain the data processed by the current thread;

[0106] Here, the fire monitoring model is constructed based on a machine learning algorithm; first, a number of fire monitoring data are collected from a historical fire event database, including smoke concentration, temperature, flame detection condition, gas concentration, and fire level data, and are arranged into a data set; then, the data set is divided, usually into a training set, a validation set, and a test set. The training set is used to train the neural network model, the validation set is used to adjust the model parameters and evaluate the model performance during the training process, and the test set is used to finally evaluate the generalization ability of the model. Generally, the data can be divided according to the proportions of 70%, 20%, and 10%. For example, assuming that there are 1000 fire record data, 700 of which can be used as the training set, 200 as the validation set, and 100 as the test set;

[0107] Then according to the task characteristics and data characteristics of fire monitoring, a suitable machine learning algorithm is selected to build a neural network model, such as a multi-layer perceptron model; then the number of layers of the neural network, the number of neurons in each layer and the activation function are determined to build a neural network model; then the training set data is input into the built neural network model for training, the back propagation algorithm is used to calculate the error of the model, and the stochastic gradient descent algorithm is used to update the parameters of the model to reduce the error; during the training process, the validation set data is used to validate the model periodically, and the accuracy, recall rate, F1 value and other performance indicators of the model on the validation set are calculated to evaluate the performance of the model, for example, after training for a certain number of rounds, the validation set data is used to validate once, and the performance change of the model is observed; then the validation result is iteratively optimized, and finally when the model reaches the highest validation accuracy on the validation set, the model is saved as the final fire monitoring model. Finally, the test set data is used to evaluate the saved fire monitoring model, calculate the performance indicators of the model on the test set, evaluate the generalization ability of the model, and then continuously debug according to the evaluation result to obtain a fire monitoring model with strong generalization ability and high model precision.

[0108] When high-concurrency time-series data is processed in each thread, the CPU usage of the edge computing node is also monitored in real time. When the CPU usage of an edge computing node in the current computing group is equal to a preset threshold, it indicates that the processing capacity of the current computing group may be close to saturation, and load adjustment is needed in a timely manner. At this time, the next computing group can be marked as the current computing group, and the high-concurrency time-series data can be continuously processed to ensure the efficiency and stability of data processing. Since each computing group has a hash table that can correspond threads to terminals, even if the computing group is switched, the high-concurrency time-series data can still be accurately and correctly allocated to the corresponding thread for processing.

[0109] Next, the data output by each thread is re-packaged and encrypted using the AES encryption method, and the re-packaged data is sent to the corresponding user terminal in a disordered manner. Before displaying on the user terminal, the data is decrypted in real time to allow the user terminal to display the current fire monitoring data and fire warning level information in real time, ensuring that the user can obtain the fire monitoring information in a timely, accurate and secure manner, so as to take appropriate measures in a timely manner, improve the efficiency and accuracy of fire prevention and control, and protect the safety of personnel and property.

[0110] Among them, the data is transmitted from the cloud to the corresponding user terminal by screening the user ID matching the terminal communication identifier from the user database, and then encrypting the data processed by the current thread according to the user ID and transmitting it to the corresponding user terminal.

[0111] Part of the data in the above formula is the dimensionless value calculated by removing the dimension, and the formula is obtained by software simulation of a large amount of collected data to obtain a formula closest to the real situation; the preset parameters and the preset threshold in the formula are set by the person skilled in the art according to the actual situation or obtained by a large amount of data simulation.

[0112] Working principle of the present application:

[0113] First, a unique identifier is assigned to each TPC terminal, and the identifiers are obtained as communication identifiers; then, a plurality of edge computing nodes are set in the cloud, and the nodes are organized to form an initial computing group according to the number of TPC terminals;

[0114] In each computing group, a plurality of threads are started and a thread pool is established; then, the threads in the thread pool are corresponded to the communication identifiers using a hash table, so that the data of each TPC terminal can be correctly processed by the thread;

[0115] When processing data, the high-concurrency time-series data in the current computing group is encrypted and transmitted to the computing group for decryption, and the decrypted data is received by the thread pool and distributed to the corresponding thread according to the hash table for processing;

[0116] During the processing, it is judged whether the CPU usage of the current computing group reaches a preset threshold, if so, the next computing group is turned to continue processing, otherwise, the processed data is encrypted and transmitted to the corresponding user terminal.

[0117] The above embodiments are only used to illustrate the technical method of the present application and are not limiting, although the present application has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical method of the present application can be modified or replaced equivalently without departing from the spirit and scope of the technical method of the present application.

Claims

1. A high concurrency time series data processing method, characterized in that, The method comprises the following steps: S1, assigning unique identifiers to a plurality of TPC terminals to obtain a plurality of communication identifiers; S2, deploying a plurality of edge computing nodes in the cloud, setting a plurality of initial computing groups according to the number of TPC terminals, and starting a plurality of threads on the plurality of edge computing nodes, constructing a thread pool based on the plurality of threads to obtain a thread pool of the plurality of initial computing groups; S3, traversing the plurality of initial computing groups, obtaining the thread ID of each initial computing group, constructing a hash table based on the thread ID and the plurality of communication identifiers, and obtaining a plurality of computing groups corresponding to the thread pool and the hash table; S4, determining the current computing group, and after encrypting the high-concurrency time sequence data of the plurality of TPC terminals, transmitting the encrypted data to the current computing group for decryption to obtain decrypted data; S5, receiving the decrypted data using the thread pool of the current computing group, and according to the hash table of the current computing group, transmitting the decrypted data to the corresponding thread for processing to obtain a plurality of thread-processed data; S6, determining whether the CPU usage rate of the plurality of edge computing nodes in the current computing group is equal to a preset threshold value; Yes, traversing the next computing group, marking it as the current computing group, and repeating S4-S6; No, encrypting and transmitting the plurality of thread-processed data to the corresponding user terminal.

2. The high-concurrency time-series data processing method of claim 1, wherein, The method for assigning unique identifiers to a plurality of TPC terminals comprises the following steps: S11, obtaining the MAC addresses of the plurality of TPC terminals through a network management protocol to obtain a plurality of MAC addresses; S12, extracting a preset number of hexadecimal numbers of the plurality of MAC addresses and splicing them to obtain a plurality of unique identifiers.

3. The high-concurrency time-series data processing method of claim 1, wherein, The method for setting a plurality of initial computing groups according to the number of TPC terminals comprises the following steps: S2-11, initializing an initial computing group and marking it as the current initial computing group; S2-12, traversing the plurality of edge computing nodes, calculating the product of the CPU core number of the current node and a preset unit thread number to obtain the support thread number; S2-13, assigning the current computing node to the current initial computing group, and accumulating the thread number in the current initial computing group according to the support thread number; S2-14, calculating the ratio of the number of TPC terminals to a preset number to obtain the required thread number, and determining whether the thread number in the current initial computing group is greater than or equal to the required thread number; Yes, obtaining an initial computing group and jumping to S2-16; No, jumping to S2-15; S2-15, determining whether there is a next computing node; Yes, marking the next computing node as the current node and jumping to S2-12; No, closing the current computing group to obtain a plurality of initial computing groups; S2-16, determining whether there is a next computing node; Yes, marking the next computing node as the current node and jumping to S2-11; No, obtaining a plurality of initial computing groups.

4. The high-concurrency time-series data processing method of claim 1, wherein, The method for constructing a plurality of thread pools based on a plurality of threads comprises the following steps: S2-21, traversing the plurality of computing groups, starting a plurality of threads in the current computing group, and obtaining a plurality of threads in the current computing group; S2-22, constructing a thread pool using a thread pool framework, and putting the plurality of threads in the current computing group into the thread pool to obtain a thread pool. S2-23, mark the next calculation group as the current calculation group, repeat S2-21 to S2-23 until the traversal is completed, and obtain a plurality of thread pools.

5. The high-concurrency time-series data processing method of claim 1, wherein, The hash table is constructed based on the thread ID and the plurality of communication identifiers, and includes: S3-1, initializing a hash table; wherein the values in the hash table are thread IDs, ranging from 0 to N-1, N representing the number of TPC terminals; S3-2, performing hash operation on the plurality of communication identifiers to obtain a plurality of initial hash values; S3-3, performing AND operation and modulo operation on the plurality of initial hash values to obtain a plurality of hash values with a mapping range of [0, N-1]; S3-4, marking the plurality of hash values as keys and storing them in the hash table to obtain a hash table in which the communication identifiers and threads are one-to-one.

6. The high-concurrency time-series data processing method of claim 1, wherein, The high-concurrency time sequence data of the plurality of TPC terminals is encrypted, including: S4-11, traversing the plurality of TPC terminals, and dividing the data of the current TPC terminal according to a preset character number to obtain a plurality of packets with timestamps; S4-12, generating a packet ID according to the communication identifier of the current TPC terminal, the timestamps of the plurality of packets, and the positions of the plurality of packets in the data of the current TPC terminal to obtain the packet ID of the plurality of packets; S4-13, arranging the plurality of packets of the current TPC terminal in disorder, and encrypting the plurality of packets arranged in disorder by using the AES algorithm to obtain the high-concurrency time sequence data of the current TPC terminal after encryption; S4-14, marking the next TPC terminal as the current TPC terminal, and repeating S4-1 to S4-4 until the traversal is completed to obtain a plurality of high-concurrency time sequence data after encryption.

7. The high-concurrency time-series data processing method of claim 1, wherein, The high-concurrency time sequence data of the plurality of TPC terminals is encrypted, including: S4-21, decrypting the plurality of data after encryption by using the AES algorithm to obtain a plurality of packets; S4-22, obtaining the packet ID of the plurality of packets, and reordering the plurality of packets according to the packet ID to obtain the high-concurrency time sequence data of the plurality of TPC terminals.

8. The high-concurrency time-series data processing method of claim 1, wherein, The decrypted data is transmitted to the corresponding thread for processing, including: S5-11, deploying a fire monitoring model on each of the edge computing nodes in the current calculation group to obtain a shared model of the plurality of edge computing nodes; wherein the fire monitoring model is constructed based on a machine learning algorithm; S5-12, obtaining the high-concurrency time sequence data received by the current thread, including the communication identifier of the TPC terminal, the timestamp, the smoke concentration, the temperature, the flame detection condition, and the gas concentration; S5-13, inputting the smoke concentration, the temperature, the flame detection condition, and the gas concentration of the same timestamp into the shared model of the current edge computing node to obtain a fire prediction level; S5-14, outputting the high-concurrency time sequence data received by the current thread and the fire prediction level together to obtain the data processed by the current thread.

9. The high concurrency time series data processing method of claim 8, wherein, The fire monitoring model is constructed based on a machine learning algorithm, including: S5-111, obtaining a plurality of smoke concentration, temperature, flame detection condition, gas concentration, and fire level data from a plurality of fire record data, and arranging them into a data set; S5-112, a neural network model is constructed by using a machine learning algorithm, the data set is input into the neural network model for training and verification, and iterative optimization is performed according to the verification result; S5-113, save the neural network model with the highest verification accuracy to obtain a fire monitoring model with input of smoke concentration, temperature, flame detection condition and gas concentration and output of fire prediction grade.

10. The high-concurrency time-series data processing method of claim 1, wherein, The encrypted transmission of the data processed by the plurality of threads to the corresponding user end comprises: S5-21, obtaining a communication identifier of a TPC terminal corresponding to the current thread to obtain a current communication identifier; S5-22, filtering a user ID matched with the current communication identifier from a user database; wherein the user database is deployed in the cloud and contains the user ID, the user end address and the communication identifier; S5-23, encrypting and transmitting the data processed by the current thread to the corresponding user end according to the user ID.

Images