High-concurrency time series data processing method

Abstract

The invention discloses a high-concurrency time sequence data processing method, which comprises the following steps of: distributing a unique identifier for a TPC terminal to obtain a plurality of communication identifiers; edge computing nodes are deployed at the cloud, computing groups are set according to the number of TPC terminals, threads are started, and a thread pool is constructed; traversing the calculation group, constructing a hash table based on the thread ID and the communication identifier, and enabling the thread pool and the hash table to correspond to the calculation group; determining a current computing group, encrypting high-concurrency time sequence data of the TPC terminal and transmitting the high-concurrency time sequence data to the current computing group for decryption; receiving the decrypted data by using the current calculation group thread pool, and transmitting the data to a corresponding thread for processing according to the hash table; judging whether the CPU utilization rate of the current calculation group reaches a preset threshold value or not; and if so, switching to the next calculation group to repeat the steps, otherwise, encrypting and transmitting the processed data to the user side. The invention relates to the field of high-concurrency time series data processing, and solves the problems that the high-concurrency time series data processing sequence is difficult to guarantee and the load is unbalanced.

Description

Technical Field

[0001] The invention belongs to the field of fire monitoring, relates to a high-concurrency data processing technology, and specifically is a high-concurrency time series data processing method. Background Art

[0002] In the field of fire monitoring, fire monitoring requires real-time and accurate processing of a large amount of time series data from different monitoring points in order to detect fires in a timely manner and issue early warnings. High concurrency processing capabilities can ensure that the system can still operate stably under high load conditions without missing any important fire monitoring information. This plays a vital role in improving the efficiency of fire prevention and control and reducing the losses caused by fires. Therefore, in the field of fire monitoring, the use of efficient high-concurrency time series data processing methods is the key to ensuring system stability and the reliability of detection results.

[0003] Most existing data processing methods rely on single-threaded processing. When faced with large-scale concurrent scenarios, the processing capacity will be severely limited, resulting in data processing speed that cannot meet the demand. In order to improve processing capabilities, some systems have adopted multi-threaded processing methods. However, in a high-concurrency environment, a large amount of data continuously flows into the system, and each thread processes this data at the same time. Since the execution of threads is concurrent, there is a lack of effective coordination mechanism between them to ensure that the data is processed in strict time order, which makes it difficult to ensure the order of data processing. For example, different threads may process data with similar timestamps at the same time, but due to the scheduling strategy of the operating system and the uncertainty of thread execution, the output order of these data after processing may be inconsistent with their original time order. This will have a serious impact on those application scenarios that have strict requirements on timing, such as real-time fire monitoring systems.

[0004] Secondly, although multithreading is intended to improve overall processing capabilities, in actual operation, load imbalance may occur. Some threads may be overloaded due to being assigned a large number of complex tasks or frequently accessing shared resources, while other threads may be relatively idle. This load imbalance will not only reduce the overall processing efficiency of the system, but may also cause some threads to experience performance degradation or even crash due to long-term high-load operation. Summary of the invention

[0005] The present invention aims to solve at least one of the technical problems existing in the prior art; to this end, the present invention proposes a high-concurrency time series data processing method, which is used to solve the problems of difficulty in ensuring the sequentiality of high-concurrency time series data processing and load imbalance in the field of fire monitoring.

[0006] To achieve the above object, the present invention provides a high-concurrency time series data processing method, comprising:

[0007] S1, assigning unique identifiers to a number of TPC terminals to obtain a number of communication identifiers;

[0008] S2, deploying several edge computing nodes in the cloud, setting several initial computing groups according to the number of TPC terminals, and starting several threads on several edge computing nodes, building thread pools based on the several threads, and obtaining thread pools of several initial computing groups;

[0009] S3, traverse several initial computing groups, obtain the thread ID of each initial computing group, build a hash table based on the thread ID and several communication identifiers, and obtain several computing groups with one-to-one correspondence between thread pools, hash tables and computing groups;

[0010] S4, determining the current computing group, encrypting the high-concurrency time series data of several TPC terminals, and transmitting the encrypted data to the current computing group for decryption to obtain decrypted data;

[0011] S5, using the thread pool of the current computing group to receive the decrypted data, and according to the hash table of the current computing group, transferring the decrypted data to the corresponding thread for processing, to obtain data processed by several threads;

[0012] S6, determine whether the CPU usage of several edge computing nodes in the current computing group is equal to the preset threshold; if yes, traverse the next computing group, mark it as the current computing group, and repeat S4-S6; if no, encrypt and transmit the data processed by several threads to the corresponding user end.

[0013] Based on the above technical process, several initial computing groups are formed by assigning unique identifiers to TPC terminals and deploying edge computing nodes in the cloud. At the same time, threads are started in each computing group to build a thread pool, and the thread pool and communication identifier are matched one by one through a hash function, thereby obtaining a high-concurrency time series processing method that can ensure the data processing order and processing efficiency. In addition, the method also uses data encryption technology to ensure the security of data transmission, and dynamically adjusts the processing tasks of the computing group according to the CPU usage of the edge computing node to achieve load balancing, thereby increasing the efficiency, security and reliability of the high-concurrency time series data processing method, and improving resource utilization and data processing continuity.

[0014] Furthermore, allocating unique identifiers to the plurality of TPC terminals includes:

[0015] S11, obtaining MAC addresses of several TPC terminals through a network management protocol to obtain several MAC addresses;

[0016] S12, extracting a preset number of hexadecimal numbers of a plurality of MAC addresses and concatenating them to obtain a plurality of unique identifiers.

[0017] Furthermore, the step of setting a plurality of initial calculation groups according to the number of TPC terminals includes:

[0018] S2-11, initialize an initial calculation group, marked as the current initial calculation group;

[0019] S2-12, traverse several edge computing nodes, calculate the product of the number of CPU cores of the current node and the preset number of unit threads, and obtain the number of supported threads;

[0020] S2-13, classifying the current computing node into the current initial computing group, and accumulating the number of threads in the current initial computing group according to the number of supported threads;

[0021] S2-14, judging whether the number of threads in the current initial computing group is greater than or equal to the required number of threads; if yes, obtaining an initial computing group and jumping to S2-16; if no, jumping to S2-15;

[0022] S2-15, determine whether there is a next computing node; if yes, mark the next computing node as the current node and jump to S2-12; if no, close the current computing group and obtain several initial computing groups;

[0023] S2-16, determine whether there is a next computing node; if yes, mark the next computing node as the current node and jump to S2-11; if no, obtain several initial computing groups.

[0024] Furthermore, constructing a plurality of thread pools based on a plurality of threads includes:

[0025] S2-21, traverse several computing groups, start several threads in the current computing group that are equal to the required number of threads, and obtain several threads of the current computing group;

[0026] S2-22, using the thread pool framework to build a thread pool, and putting several threads of the current computing group into the thread pool to obtain a thread pool;

[0027] S2-23, mark the next computing group as the current computing group, repeat S2-21 to S2-23 until the traversal is completed, and obtain several thread pools.

[0028] The initial computing group setting provides an important basis for achieving load balancing. By grouping according to the number of TPC terminals and the capabilities of edge computing nodes, it can be ensured that each computing group has sufficient processing power to handle the data processing tasks assigned to it, avoiding resource waste and overload, and improving overall processing efficiency.

[0029] Furthermore, the constructing of a hash table based on the thread ID and a plurality of communication identifiers includes:

[0030] S3-1, initialize a hash table; the value in the hash table is thread ID, ranging from [0, N-1], where N represents the number of TPC terminals;

[0031] S3-2, performing hash operations on a number of communication identifiers to obtain a number of initial hash values;

[0032] S3-3, performing AND operation and modulo operation on several initial hash values ​​to obtain several hash values ​​with a mapping range of [0, N-1];

[0033] S3-4, mark several hash values ​​as keys and store them in a hash table, to obtain a hash table in which communication identifiers correspond to threads one by one.

[0034] The hash table is constructed using AND operations and modulo operations to match communication identifiers with threads one by one. This is because in high concurrency situations, data from different TPC terminals will flow into the system at an extremely fast speed. Through the hash table, it is possible to quickly determine which thread the data should be assigned to for processing based on the communication identifier, ensuring that the data from each terminal can accurately find the corresponding processing path.

[0035] Furthermore, the encrypting of the high-concurrency time series data of a plurality of TPC terminals includes:

[0036] S4-11, traverse a number of TPC terminals, divide the data of the current TPC terminal according to a preset number of characters, and obtain a number of sub-packets with time stamps;

[0037] S4-12, generating a packet ID according to the communication identifier of the current TPC terminal, the timestamps of the plurality of subpackets, and the positions of the plurality of subpackets in the data of the current TPC terminal, and obtaining the packet IDs of the plurality of subpackets;

[0038] S4-13, shuffle the subpackets of the current TPC terminal, and encrypt the shuffled subpackets using the AES algorithm to obtain the encrypted high-concurrency time series data of the current TPC terminal;

[0039] S4-14, mark the next TPC terminal as the current TPC terminal, repeat S4-1 to S4-4 until the traversal is completed, and obtain a number of encrypted high-concurrency time series data.

[0040] Traverse the TPC terminals, split the data according to the preset number of characters to obtain sub-packets with timestamps; generate a packet ID based on the terminal communication identifier, sub-packet timestamp and position; sub-packetize in random order and encrypt with the AES algorithm to obtain the encrypted data of the current terminal; mark the next terminal and repeat the operation until all terminals are traversed to obtain a number of encrypted high-concurrency time series data.

[0041] Furthermore, the transmitting to the current computing group for decryption includes:

[0042] S4-21, decrypting the encrypted data using the AES algorithm to obtain a number of sub-packets;

[0043] S4-22, obtain packet IDs of several subpackets, reorder the several subpackets according to the packet IDs, and obtain high-concurrency timing data of several TPC terminals.

[0044] In a high-concurrency environment, a large amount of data is transmitted in the network, and this data may contain sensitive information. Therefore, the high-concurrency time series data is segmented according to the preset number of characters to obtain several data packets, which are then encrypted with AES and then transmitted to the cloud in a random order, which increases the security and randomness of data transmission and ensures the integrity of the time series data.

[0045] Furthermore, the step of transmitting the decrypted data to a corresponding thread for processing includes:

[0046] S5-11, deploying a fire monitoring model on each edge computing node in the current computing group to obtain a shared model of several edge computing nodes; wherein the fire monitoring model is constructed based on a machine learning algorithm;

[0047] S5-12, obtaining the high-concurrency time series data received by the current thread, including the communication identifier, timestamp, smoke density, temperature, flame detection status and gas concentration of the TPC terminal; the flame detection status is represented by (0, 1), 0 means no flame is detected, and 1 means there is a flame;

[0048] S5-13, inputting the smoke density, temperature, flame detection status and gas concentration of the same timestamp into the shared model of the current edge computing node to obtain the fire prediction level;

[0049] S5-14, output the high-concurrency time series data and the fire prediction level received by the current thread together to obtain the data processed by the current thread.

[0050] Furthermore, the fire monitoring model is constructed based on a machine learning algorithm, including:

[0051] S5-111, based on several fire record data, obtain several smoke concentration, temperature, flame detection conditions, gas concentration and fire grade data and organize them into a data set;

[0052] S5-112, use machine learning algorithms to build a neural network model, input the data set into the neural network model for training and verification, and perform iterative optimization based on the verification results;

[0053] S5-113, save the neural network model with the highest verification accuracy, and obtain a fire monitoring model whose input is smoke concentration, temperature, flame detection status, and gas concentration, and whose output is the fire prediction level.

[0054] Furthermore, encrypting and transmitting the data processed by the plurality of threads to the corresponding user end includes:

[0055] S5-21, obtaining a communication identifier of the TPC terminal corresponding to the current thread, and obtaining a current communication identifier;

[0056] S5-22, filtering and obtaining a user ID that matches the current communication identifier from a user database; wherein the user database is deployed in the cloud and includes a user ID, a user terminal address, and a communication identifier;

[0057] S5-23, encrypt the data processed by the current thread according to the user ID and transmit it to the corresponding user end.

[0058] Compared with the prior art, the present invention has the following beneficial effects:

[0059] By assigning a unique communication identifier to each TPC terminal and using a custom hash function to map the communication identifier to a unique number corresponding to each thread, the data of each terminal can be correctly tracked and processed and transmitted in chronological order, ensuring the sequentiality and timing of data processing;

[0060] By deploying edge computing nodes in the cloud and dividing them into several computing groups, high-concurrency data can be processed in computing groups, effectively preventing the task termination and processing efficiency reduction caused by excessive load on a computing node, thus ensuring the continuity and stability of data processing;

[0061] By subpacketizing high-concurrency time-series data, using the AES encryption algorithm and out-of-order transmission of data packets, double encryption protection is achieved, improving the security of data during network transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0063] Figure 1 A technical flow chart of a high-concurrency time series data processing method provided by the present invention;

[0064] Figure 2A flow chart of a method for setting an initial calculation group provided by the present invention;

[0065] Figure 3 A flow chart of a method for encrypting high-concurrency time series data provided by the present invention. DETAILED DESCRIPTION

[0066] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0067] See also Figure 1 - Figure 3 , an embodiment of the present invention provides a high-concurrency time series data processing method, including:

[0068] S1, assigning unique identifiers to a number of TPC terminals to obtain a number of communication identifiers;

[0069] S2, deploying several edge computing nodes in the cloud, setting several initial computing groups according to the number of TPC terminals, and starting several threads on several edge computing nodes in each computing group, building a thread pool based on the several threads, and obtaining thread pools of several initial computing groups;

[0070] S3, traverse several initial computing groups, obtain the thread ID of the thread pool of each computing group, build a hash table based on the thread ID and several communication identifiers, and obtain several computing groups with one-to-one correspondence between thread pools, hash tables and computing groups;

[0071] S4, obtaining a current computing group from a plurality of computing groups, encrypting high-concurrency time series data of a plurality of TPC terminals, transmitting the data to the current computing group for decryption, and obtaining a plurality of decrypted data;

[0072] S5, using the thread pool of the current computing group to receive the decrypted data, and transferring the decrypted data to the corresponding thread for processing according to the hash table of the current computing group, to obtain the data processed by the thread;

[0073] S6, determine whether the CPU usage of several edge computing nodes in the current computing group is equal to the preset threshold; if yes, traverse the next computing group and mark it as the current computing group, and repeat S4-S6; if no, encrypt the data processed by several threads and transmit it to the corresponding user end.

[0074] In this embodiment, in order to ensure that the data of each terminal can be correctly identified and tracked during the processing, a unique communication identifier is assigned to each TPC terminal; specifically, the MAC addresses of several TPC terminals are first obtained through the network management protocol, and then the preset number of hexadecimal numbers of each MAC address are extracted and spliced ​​to obtain a unique identifier. For example, the MAC address of an existing TPC terminal is 00:1A:2B:3C:4D:5E, and the 6-digit hexadecimal number is extracted and spliced ​​to obtain the unique communication identifier of the terminal: 001A2B.

[0075] Next, in order to reduce the delay of data transmission and improve the real-time performance of data processing, several edge computing nodes are deployed in the cloud, making full use of the distributed computing capabilities of the cloud to provide powerful computing capabilities for the processing of high-concurrency time series data. In this embodiment, by allocating several edge computing nodes into several computing groups, high-concurrency data is processed in units of computing groups. When a node in a computing group is overloaded, the data will be quickly transferred to the next computing group to continue data processing, preventing the task termination or low processing efficiency caused by overload.

[0076] Specifically, first, an initial computing group is initialized, and several edge computing nodes are traversed to calculate the product of the number of CPU cores of the current node and the preset number of unit threads to obtain the number of supported threads; then the current computing node is classified into the current initial computing group, and the number of threads in the current initial computing group is accumulated according to the number of supported threads; then it is determined whether the number of threads in the current computing group is greater than or equal to the required number of threads; if yes, a computing group is obtained; if not, the next computing node is marked as the current node, and the edge computing nodes are added to a computing group repeatedly until the cumulative number of threads is greater than or equal to the ratio of the number of TPC terminals to the preset number, that is, whether it is greater than or equal to the required number of threads, so as to ensure that each TPC terminal can have a thread corresponding to it; then the above steps are repeated until all edge computing nodes are assigned to a computing group, thereby obtaining several initial computing groups; finally, in each computing group, threads with the same number of required threads are opened, and the threads are put into a thread pool for management to obtain several initial computing groups containing threads; if the cumulative number of threads in the last initial computing group is less than the number of TPC terminals, the edge computing nodes in the computing group are closed and not used.

[0077] For example, assuming that there are 500 TPC terminals and the preset number is 5, that is, one thread ID corresponds to 5 terminals one by one, then the required number of threads is 100, and 5 edge computing nodes are deployed in the cloud, and the preset number of unit threads is 10;

[0078] First, initialize the first calculation group and mark it as the current calculation group;

[0079] Then traverse the first edge computing node. Assuming that the node has 4 CPU cores, the number of supported threads is 4×10=40. The node is assigned to the current computing group, and the number of threads in the current computing group is 40.

[0080] It is determined whether the number of threads in the current computing group, 40, is greater than the required number of threads, 100. If not, it is further determined that there is a next computing node, and the next computing node is marked as the current node.

[0081] Assuming that the second edge computing node has 6 CPU cores, the number of threads supported is 6×10=60. The node is added to the current computing group, and the number of threads in the current computing group becomes 40+60=100;

[0082] Determine whether the number of threads in the current computing group 100 is greater than or equal to the required number of threads 100. If yes, get an initial computing group;

[0083] If it is further determined that there is a next computing node, the next computing node is marked as the current node, and the second computing group is initialized and marked as the current computing group;

[0084] Finally, the above process is repeated until all edge computing nodes are traversed to obtain several initial computing groups, and threads of the number of terminals are opened in several computing groups, and thread pools are built according to the threads to obtain several initial computing groups containing threads.

[0085] In order to ensure the order of data processing, a custom hash function is provided in this embodiment, which can map the communication identifier to a unique number and store it in the hash table as a key value, so as to achieve a one-to-one correspondence between the terminal and the thread, thereby improving the efficiency of data processing;

[0086] Specifically, first, a hash table is initialized to store thread objects in a computing group; the values ​​in the hash table are thread IDs in the range of [0, N-1], where N represents the number of TPC terminals; then, a number of communication identifiers are hashed to obtain a number of initial hash values; then, a custom hash function is used to perform AND operations and modulo operations on the initial hash values, and the initial hash values ​​are mapped to [0, N-1] to obtain a number of hash values; finally, the hash values ​​are marked as keys and stored in the hash table to obtain a hash table in which communication identifiers correspond to threads one by one;

[0087] In this embodiment, the code of the exemplary custom hash function is:

[0088] private static int customHash(String str){

[0089] long hash = 5381;

[0090] for(int i=0; i <str.length();i++){

[0091] hash=((hash<<5)+hash)+str.charAt(i);}

[0092] / / Limit the hash value to the range of 32-bit integers

[0093] hash = hash & 0xFFFFFFFFL;

[0094] hash = hash% (1000000L*8);

[0095] hash = hash%8;

[0096] return (int) hash}

[0097] In a high-concurrency environment, a large amount of data is transmitted in the network. If it is not encrypted, the security of the data will face great risks. In this embodiment, by sub-packetizing the high-concurrency data and then encrypting the sub-packetized data with AES, double encryption protection is achieved to improve data security;

[0098] Specifically, the encryption process in this embodiment includes the following steps:

[0099] Step 1: traverse a number of TPC terminals, divide the data of the current TPC terminal according to a preset number of characters (for example, 50 characters), and obtain a number of sub-tables with time stamps;

[0100] Step 2: Generate a packet ID according to the identifier of the current TPC terminal, the timestamps of the multiple packets, and the positions of the multiple packets in the data of the current TPC terminal to obtain the packet IDs of the multiple packets; for example, if the communication identifier of a terminal is 001A2B, the timestamp is: 2024-10-14T10:00:00.000, and the position of a certain packet is 1, the obtained packet ID is: 001A2B20241014T1000000001;

[0101] Step 3: shuffle several subpackets of the current TPC terminal, and encrypt the shuffled subpackets using the AES algorithm to obtain the encrypted high-concurrency time series data of the current TPC terminal;

[0102] Step 4: Mark the next TPC terminal as the current TPC terminal, and repeat steps 1 to 4 until the traversal is completed to obtain the encrypted high-concurrency time series data.

[0103] Similarly, the corresponding decryption process is to decrypt the encrypted data according to the packet ID and the AES decryption algorithm.

[0104] Through the above steps, the high-concurrency data from the terminal can be safely transmitted to the cloud computer for data processing. Next, the thread pool of the current computing group is used to receive the decrypted data, and then the decrypted data is transmitted to the corresponding thread for processing according to the hash table of the computing group;

[0105] In this embodiment, the processing of high-concurrency time series data for fire monitoring is taken as an example; first, a fire monitoring model is deployed on each edge computing node in the current computing group to obtain a shared model; then the high-concurrency time series data received by the current thread is obtained, including the communication identifier, timestamp, smoke density, temperature, flame detection status and gas concentration of the TPC terminal; the flame detection status is represented by (0, 1), 0 means no flame is detected, and 1 means there is a flame; then the smoke density, temperature, flame detection status and gas concentration of the same timestamp are input into the shared model to obtain the fire prediction level; finally, the high-concurrency time series data received by the current thread and the fire prediction level are output together to obtain the data processed by the current thread;

[0106] Here, the fire monitoring model is built based on a machine learning algorithm; first, a number of fire monitoring data are collected from the historical fire event database, including smoke concentration, temperature, flame detection conditions, gas concentration, and fire level data, and organized into a data set; then the data set is divided, usually into a training set, a validation set, and a test set. The training set is used to train the neural network model, the validation set is used to adjust the model parameters and evaluate the model performance during the training process, and the test set is used to finally evaluate the generalization ability of the model. Generally speaking, it can be divided according to the ratio of 70%, 20%, and 10%. For example, assuming there are 1,000 fire record data, 700 of them can be used as training sets, 200 as validation sets, and 100 as test sets;

[0107] Then, according to the task characteristics and data characteristics of fire monitoring, select a suitable machine learning algorithm to build a neural network model, such as a multi-layer perceptron model; then determine the number of neural network layers, the number of neurons in each layer, and the activation function to build a neural network model; then input the training set data into the constructed neural network model for training, use the back propagation algorithm to calculate the model error, and update the model parameters through the stochastic gradient descent algorithm to reduce the error; during the training process, regularly use the validation set data to verify the model, calculate the accuracy, recall rate, F1 value and other performance indicators of the model on the validation set, and evaluate the performance of the model. For example, after a certain number of training rounds, use the validation set data for a validation and observe the performance changes of the model; then iterate and optimize according to the validation results, and finally save the model as the final fire monitoring model when the model reaches the highest validation accuracy on the validation set. Finally, use the test set data to make a final evaluation of the saved fire monitoring model, calculate the performance indicators of the model on the test set, evaluate the generalization ability of the model, and then continuously debug according to the evaluation results to obtain a fire monitoring model with strong generalization ability and high model accuracy.

[0108] When high-concurrency time series data is processed in each thread, the CPU usage of the edge computing node is also monitored in real time. When the CPU usage of an edge computing node in the current computing group is equal to the preset threshold, it means that the processing capacity of the current computing group may be close to saturation and load adjustment is required in time. At this time, you can traverse the next computing group and mark it as the current computing group, and continue to process high-concurrency time series data to ensure the efficiency and stability of data processing. Since each computing group has a hash table that can correspond threads to terminals, even if the computing group is switched, high-concurrency time series data can still be accurately assigned to the corresponding thread for processing.

[0109] Next, the data output by each thread is divided into packets and then encrypted by AES in the same way, and the packets are sent to the corresponding user end in random order. The packets are decrypted in real time before being displayed on the user end, so that the user end can view the current fire monitoring data and fire warning level information in real time, ensuring that the user can obtain the fire monitoring information in a timely, accurate and safe manner, so as to take corresponding measures in time, improve the efficiency and accuracy of fire prevention and control, and ensure the safety of personnel and property;

[0110] The data is transmitted from the cloud to the corresponding user terminal by screening the user ID that matches the terminal communication identifier from the user database, and then encrypting the data processed by the current thread according to the user ID and transmitting it to the corresponding user terminal.

[0111] Part of the data in the above formula is calculated by removing the dimension and taking its numerical value. The formula is a formula closest to the actual situation obtained by software simulation of a large amount of collected data; the preset parameters and preset thresholds in the formula are set by technical personnel in this field according to actual conditions or obtained through simulation of a large amount of data.

[0112] Working principle of the present invention:

[0113] First, a unique identifier is assigned to each TPC terminal, and these identifiers are obtained as communication identifiers; then, multiple edge computing nodes are set up in the cloud, and these nodes are organized into an initial computing group according to the number of TPC terminals;

[0114] In each computing group, multiple threads are started and a thread pool is established. Then, a hash table is used to match the threads in the thread pool with the communication identifiers to ensure that the data of each TPC terminal can be processed by the correct thread.

[0115] When processing data, the high-concurrency time series data in the current computing group is encrypted and transmitted to the computing group for decryption. The decrypted data is received by the thread pool and assigned to the corresponding thread for processing according to the hash table;

[0116] During the processing, it is determined whether the CPU usage of the current computing group has reached the preset threshold. If so, the processing is transferred to the next computing group. Otherwise, the processed data is encrypted and transmitted to the corresponding user end.

[0117] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A high-concurrency time series data processing method, characterized in that: include: S1, assigning unique identifiers to a number of TPC terminals to obtain a number of communication identifiers; S2, deploying several edge computing nodes in the cloud, setting several initial computing groups according to the number of TPC terminals, and starting several threads on several edge computing nodes, building thread pools based on the several threads, and obtaining thread pools of several initial computing groups; S3, traverse several initial computing groups, obtain the thread ID of each initial computing group, build a hash table based on the thread ID and several communication identifiers, and obtain several computing groups with one-to-one correspondence between thread pools, hash tables and computing groups; S4, determining the current computing group, encrypting the high-concurrency time series data of several TPC terminals, and transmitting the encrypted data to the current computing group for decryption to obtain decrypted data; S5, using the thread pool of the current computing group to receive the decrypted data, and according to the hash table of the current computing group, transferring the decrypted data to the corresponding thread for processing, to obtain data processed by several threads; S6, determining whether the CPU usage of several edge computing nodes in the current computing group is equal to a preset threshold; If yes, traverse the next calculation group, mark it as the current calculation group, and repeat S4-S6; If not, the data processed by the several threads will be encrypted and transmitted to the corresponding user end.

2. A high-concurrency time series data processing method according to claim 1, characterized in that: The allocating unique identifiers to the plurality of TPC terminals comprises: S11, obtaining MAC addresses of several TPC terminals through a network management protocol to obtain several MAC addresses; S12, extracting a preset number of hexadecimal numbers of a plurality of MAC addresses and concatenating them to obtain a plurality of unique identifiers.

3. The high-concurrency time series data processing method according to claim 1, characterized in that: The step of setting a plurality of initial calculation groups according to the number of TPC terminals includes: S2-11, initialize an initial calculation group, marked as the current initial calculation group; S2-12, traverse several edge computing nodes, calculate the product of the number of CPU cores of the current node and the preset number of unit threads, and obtain the number of supported threads; S2-13, classifying the current computing node into the current initial computing group, and accumulating the number of threads in the current initial computing group according to the number of supported threads; S2-14, calculate the ratio of the number of TPC terminals to the preset number, obtain the required number of threads, and determine whether the number of threads in the current initial calculation group is greater than or equal to the required number of threads; if yes, obtain an initial calculation group and jump to S2-16; if no, jump to S2-15; S2-15, determine whether there is a next computing node; if yes, mark the next computing node as the current node and jump to S2-12; if no, close the current computing group and obtain several initial computing groups; S2-16, determine whether there is a next computing node; if yes, mark the next computing node as the current node and jump to S2-11; if no, obtain several initial computing groups.

4. The high-concurrency time series data processing method according to claim 1, characterized in that: The method of constructing a plurality of thread pools based on a plurality of threads includes: S2-21, traverse several computing groups, start several threads in the current computing group that are equal to the required number of threads, and obtain several threads of the current computing group; S2-22, using the thread pool framework to build a thread pool, and putting several threads of the current computing group into the thread pool to obtain a thread pool; S2-23, mark the next computing group as the current computing group, repeat S2-21 to S2-23 until the traversal is completed, and obtain several thread pools.

5. The high-concurrency time series data processing method according to claim 1, characterized in that: The step of constructing a hash table based on the thread ID and a plurality of communication identifiers includes: S3-1, initialize a hash table; the value in the hash table is thread ID, ranging from [0, N-1], where N represents the number of TPC terminals; S3-2, performing hash operations on a number of communication identifiers to obtain a number of initial hash values; S3-3, performing AND operation and modulo operation on several initial hash values ​​to obtain several hash values ​​with a mapping range of [0, N-1]; S3-4, mark several hash values ​​as keys and store them in a hash table, to obtain a hash table in which communication identifiers correspond to threads one by one.

6. A high-concurrency time series data processing method according to claim 1, characterized in that: The encrypting of high-concurrency time series data of a plurality of TPC terminals includes: S4-11, traverse a number of TPC terminals, divide the data of the current TPC terminal according to a preset number of characters, and obtain a number of sub-packets with time stamps; S4-12, generating a packet ID according to the communication identifier of the current TPC terminal, the timestamps of the plurality of subpackets, and the positions of the plurality of subpackets in the data of the current TPC terminal, and obtaining the packet IDs of the plurality of subpackets; S4-13, shuffle the subpackets of the current TPC terminal, and encrypt the shuffled subpackets using the AES algorithm to obtain the encrypted high-concurrency time series data of the current TPC terminal; S4-14, mark the next TPC terminal as the current TPC terminal, repeat S4-1 to S4-4 until the traversal is completed, and obtain a number of encrypted high-concurrency time series data.

7. The high-concurrency time series data processing method according to claim 1, characterized in that: The transmission to the current computer for decryption includes: S4-21, decrypting the encrypted data using the AES algorithm to obtain a number of sub-packets; S4-22, obtain packet IDs of several subpackets, reorder the several subpackets according to the packet IDs, and obtain high-concurrency timing data of several TPC terminals.

8. The high-concurrency time series data processing method according to claim 1, characterized in that: The transferring of the decrypted data to the corresponding thread for processing comprises: S5-11, deploying a fire monitoring model on each edge computing node in the current computing group to obtain a shared model of several edge computing nodes; wherein the fire monitoring model is constructed based on a machine learning algorithm; S5-12, obtaining the high-concurrency time series data received by the current thread, including the communication identifier, timestamp, smoke density, temperature, flame detection status and gas concentration of the TPC terminal; S5-13, inputting the smoke density, temperature, flame detection status and gas concentration of the same timestamp into the shared model of the current edge computing node to obtain the fire prediction level; S5-14, output the high-concurrency time series data and the fire prediction level received by the current thread together to obtain the data processed by the current thread.

9. A high-concurrency time series data processing method according to claim 8, characterized in that: The fire monitoring model is built based on a machine learning algorithm and includes: S5-111, based on several fire record data, obtain several smoke concentration, temperature, flame detection conditions, gas concentration and fire grade data and organize them into a data set; S5-112, use machine learning algorithms to build a neural network model, input the data set into the neural network model for training and verification, and perform iterative optimization based on the verification results; S5-113, save the neural network model with the highest verification accuracy, and obtain a fire monitoring model whose input is smoke concentration, temperature, flame detection status, and gas concentration, and whose output is the fire prediction level.

10. The high-concurrency time series data processing method according to claim 1, characterized in that: The encrypted data processed by the plurality of threads is transmitted to the corresponding user end, including: S5-21, obtaining a communication identifier of the TPC terminal corresponding to the current thread, and obtaining a current communication identifier; S5-22, filtering and obtaining a user ID that matches the current communication identifier from a user database; wherein the user database is deployed in the cloud and includes a user ID, a user terminal address, and a communication identifier; S5-23, encrypt the data processed by the current thread according to the user ID and transmit it to the corresponding user end.

Images