Parallel computing method supporting real-time data stream processing in parallel programming

Abstract

The invention provides a parallel computing method for supporting real-time data stream processing in parallel programming, which comprises the following steps of: dividing a real-time data stream into fragments through a parallel pipeline, and filtering specified fields by different threads to improve the efficiency; after filtering is completed, performing local aggregation on a result by using a local aggregation function of MapReduce to reduce the data volume; a distributed memory database based on Redis Cluster is established, and data are stored in a fragmented mode according to a time window; constructing a long-short-term memory recurrent neural network model, and performing parallel training through a parameter server architecture; if the long short-term memory recurrent neural network model needs incremental learning, a parameter interface is reserved for iterative updating, and repeated training is avoided; after training is completed, the model is deployed to a GPU server, and real-time data flow is predicted; through a task scheduler of a distributed memory computing framework, cooperation between threads and processes is automatically managed, and programming work is simplified.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to a parallel computing method supporting real-time data stream processing in parallel programming. Background Art

[0002] Data stream refers to a series of dynamic data collections that are infinite in time distribution and quantity. The value of data decreases with the passage of time, so real-time calculations must be performed to give a second-level response. Data stream processing of large amounts of data often involves a lot of tasks to be processed. If all tasks are executed in a serial manner, the efficiency of the entire system will be very low, and the user experience of the application will be very poor. With the continuous increase in the scale of real-time data and the complexity of business processes, existing data processing systems face some challenges and problems. First, the traditional serial data processing method cannot meet the real-time requirements, the processing speed is slow, and it cannot respond to changing data in time. Secondly, due to the huge amount of data, the traditional centralized storage cannot meet the storage and access requirements, resulting in performance degradation. While parallel computing methods can improve processing capabilities to a certain extent, there are problems such as low data processing efficiency, low prediction accuracy, and complex programming. In traditional parallel computing methods, the processing efficiency is low due to the insufficient parallelism of data processing tasks. At the same time, due to the complexity of distributed data storage and processing, the delay in the data processing process is high, which cannot meet the real-time requirements. In addition, for the computing requirements of real-time data streams, traditional serial computing methods cannot meet the requirements of efficient and accurate computing. In addition, due to the high programming complexity of parallel computing methods, developers need to have a high degree of professional knowledge and experience, which reduces development efficiency. In summary, there is an urgent need for a parallel computing method that can perform efficient and low-latency computing on real-time data streams. Summary of the invention

[0003] The present invention provides a parallel computing method supporting real-time data stream processing in parallel programming, which mainly includes:

[0004] The real-time data stream is divided into shards through parallel pipelines, and different threads filter the specified fields to improve efficiency; after filtering, the local aggregation function of MapReduce is used to locally aggregate the results to reduce the amount of data; a distributed memory database based on RedisCluster is established to store data in shards according to time windows; a unique ID is set as the association key in the database, and Spark Streaming is used to build a distributed memory computing framework for real-time data; a long short-term memory recursive neural network model is built, and parallel training is performed through the parameter server architecture; if the long short-term memory recursive neural network model requires incremental learning, a parameter interface is reserved for iterative updates to avoid repeated training; after training, the model is deployed to the GPU server to predict the real-time data stream; the task scheduler of the distributed memory computing framework is used to automatically manage the collaboration between threads and processes to simplify programming.

[0005] Further optionally, the real-time data stream is divided into shards through parallel pipelines, and different threads filter the specified fields to improve efficiency, including:

[0006] Determine the number of shards that need to be divided; create a corresponding number of threads based on the number of shards, and associate each thread with the corresponding shard. Each thread processes a shard independently; in each thread, filter the data in the shard according to the conditions of the specified field, and only retain the data that meets the conditions; in the parallel pipeline, monitor the processing status and performance indicators of each thread.

[0007] Further optionally, after the filtering is completed, the local aggregation function of MapReduce is used to locally aggregate the results to reduce the amount of data, including:

[0008] According to the filtering conditions, the input data is mapped into key-value pairs, and data with the same key is merged; according to the key of the key-value pair, the compressed output results are grouped according to the partitioning rules to ensure that data with the same key is sent to the same folder for processing; the folder locally aggregates the input data, and merges and compresses the output results to reduce the amount of data transmission; the output results of multiple folders are merged to form the final aggregation result, and the final aggregation result is output.

[0009] Further optionally, the establishment of a distributed memory database based on RedisCluster to store data in slices according to time windows includes:

[0010] Determine the time window size and data sharding granularity; store data in shards according to the time window, and store the data in each time window on an independent Redis node; write data to the master node in RedisCluster, which is responsible for writing and replicating data, and backing up and reading data from the slave node; regularly persist the data in the memory through RDB snapshots; combine multiple commands into one transaction for execution through the transaction operations provided by RedisCluster; and monitor the system's operating status in real time through the monitoring and management tools provided by RedisCluster.

[0011] Further optionally, the unique ID is set in the database as an association key, and Spark Streaming is used to build a distributed memory computing framework for real-time data, including:

[0012] Get metadata information for each data source, including field name, data type, and data range, and determine the properties of each data source; set a unique ID for each data source in the database to ensure that the value of a record can be uniquely identified in different data sources; divide the real-time data stream into batches and use Spark Streaming to process each batch of data; use Spark Streaming to build a distributed memory computing framework and load real-time data into the distributed memory computing framework; use Spark Streaming's connection operation to find matching records in different data sources based on the unique ID; set the processing window size and data refresh frequency to reduce data latency; use data synchronization methods to ensure data consistency before data association.

[0013] Further optionally, the constructing of the long short-term memory recursive neural network model and performing parallel training through a parameter server architecture includes:

[0014] Obtain historical data streams and construct data stream data sets; construct a long short-term memory recursive neural network model, set the number of layers of the long short-term memory recursive neural network model, the number of hidden units in each layer, and the dimensions of input and output; construct the number and distribution strategy of parameter servers, and select a combination of a main parameter server and several working nodes; the main parameter server is responsible for synchronization and update of model parameters, and the working nodes are responsible for calculation and communication; input the data stream data set to the main parameter server, the parameter server divides the data stream data set into batches, and distributes each batch of data to different working nodes; each working node calculates the distributed data, and each working node uses the allocated data to train the long short-term memory neural network model. After all nodes complete the calculation, the calculation results are sent to the main parameter server, and the main parameter server updates the parameters of the long short-term memory neural network model according to the received calculation results; determine whether the model training is completed according to the preset number of training iterations or convergence conditions; if the preset convergence conditions are not met, change the gradient of each training and train again; if the preset convergence conditions have been met, end the training and obtain a trained long short-term memory recursive neural network model.

[0015] Further optionally, if the long short-term memory recursive neural network model requires incremental learning, a parameter interface is reserved for iterative updating to avoid repeated training, including:

[0016] Define a parameter interface to receive parameter update requests for new data; obtain new training data and input the new training data into the parameter server; determine the step size for updating the model parameters each time, and use gradient descent to iteratively update the existing model using the new data; end the training until the specified number of iterations is reached, and obtain the incremental model parameters; complete the incremental learning process, and update the long short-term memory recursive neural network model parameters according to the incremental model parameters.

[0017] Further optionally, after the training is completed, the model is deployed on a GPU server to predict the real-time data stream, including:

[0018] Export the trained LSTM RNN model and load it to the GPU server, and load the model parameters and structure into the memory of the GPU device; build a real-time data pipeline and preprocess the received data to match the input format of the LSTM RNN model; use the LSTM RNN model loaded on the GPU server to perform real-time inference on the preprocessed streaming data, and obtain prediction results from the LSTM RNN model.

[0019] Further optionally, the task scheduler of the distributed memory computing framework automatically manages the collaboration between threads and processes to simplify programming work, including:

[0020] Set up a unified task scheduling component to manage the execution collaboration between threads and processes; create a thread pool, create several threads and reuse them to execute tasks; create a process pool, create several processes and reuse them; use mutex locks to achieve collaboration and concurrency between threads or processes; set up monitoring components to monitor application deployment status, task execution, and resource consumption.

[0021] The technical solution provided by the embodiment of the present invention may have the following beneficial effects:

[0022] The present invention discloses a parallel computing method that supports real-time data stream processing in parallel programming. The system realizes the ability to efficiently process large-scale real-time data through parallel processing and distributed storage. The efficiency and accuracy of real-time data stream prediction are improved through parallel training and low-latency prediction data stream of GPU server. At the same time, the programming work is simplified and the development efficiency is improved through the automatic management of the task scheduler. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 The present invention is a flow chart of a parallel computing method supporting real-time data stream processing in parallel programming.

[0024] Figure 2 A schematic diagram of a parallel computing method supporting real-time data stream processing in parallel programming according to the present invention. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0026] In this embodiment, a parallel computing method supporting real-time data stream processing in parallel programming may specifically include:

[0027] S101. The real-time data stream is divided into shards through parallel pipelines, and different threads filter the specified fields to improve efficiency.

[0028] Determine the number of shards that need to be divided; create a corresponding number of threads based on the number of shards, and associate each thread with the corresponding shard. Each thread processes a shard independently; in each thread, filter the data in the shard according to the conditions of the specified field, and only retain the data that meets the conditions; in the parallel pipeline, monitor the processing status and performance indicators of each thread.

[0029] For example, there is a real-time data stream that needs to be processed, and the data stream contains 1,000 data items. According to the processing requirements, the data stream is divided into 10 shards, and each shard is responsible for processing 100 data items. 10 threads are created, and each thread is assigned a shard for processing. Each thread processes a shard independently and filters the data by the conditions of the specified field. Each thread traverses the data in its shard and retains the data that meets the conditions. In a parallel pipeline, the processing status and performance indicators of each thread can be monitored, including processing speed and memory usage.

[0030] S102. After filtering is completed, the local aggregation function of MapReduce is used to locally aggregate the results to reduce the amount of data.

[0031] According to the filtering conditions, the input data is mapped into key-value pairs, and data with the same key is merged; according to the key of the key-value pair, the compressed output results are grouped according to the partitioning rules to ensure that data with the same key is sent to the same folder for processing; the folder locally aggregates the input data, and merges and compresses the output results to reduce the amount of data transmission; the output results of multiple folders are merged to form the final aggregation result, and the final aggregation result is output.

[0032] For example, there are 100 million records of data to be processed, including user ID, advertisement ID, and user click object fields. The data is mapped into key-value pairs, and data with the same key is merged to obtain user ID key-value pairs, advertisement ID key-value pairs, and user click object key-value pairs. The MapReduce task sets 200 Map tasks to count user clicks in parallel, and sets 100 Map tasks to count advertisement clicks. After grouping, there are 10 million user IDs, 10,000 advertisement IDs, and 89.99 million user click objects; local aggregation is performed according to user ID grouping to count the clicks of each user; local aggregation is performed according to advertisement ID grouping to count the clicks of each advertisement. Before the output results of each Map task, the key-value pairs are locally aggregated to obtain 69 million user clicks and 20.99 million advertisement clicks. Finally, the output results of multiple folders are merged to form the final aggregation result.

[0033] S103. Establish a distributed memory database based on RedisCluster and store the data in shards according to time windows.

[0034] Determine the time window size and data sharding granularity; store data in shards according to the time window, and store the data in each time window on an independent Redis node; write data to the master node in RedisCluster, which is responsible for writing and replicating data, and backing up and reading data from the slave node; regularly persist the data in the memory through RDB snapshots; combine multiple commands into one transaction for execution through the transaction operations provided by RedisCluster; and monitor the system's operating status in real time through the monitoring and management tools provided by RedisCluster.

[0035] For example, store the user's behavior data within each hour, and perform statistics and analysis on the behavior. First, the time window size can be determined to be 1 hour, that is, the behavior data within one hour is stored in each time window. Then, the data in each time window can be stored on a separate Redis node. There are 3 Redis nodes, namely Redis1, Redis2 and Redis3. Next, write the data to the master node in RedisCluster. The master node is responsible for writing and replicating data, and the slave node is responsible for backing up and reading data. The master node is Redis1, and the slave nodes are Redis2 and Redis3. In order to persist the data, the data in the memory is regularly persisted through RDB snapshots. An RDB snapshot is taken every 24 hours to persist the data of the previous 24 hours to the disk. When multiple commands need to be executed, use the transaction operation provided by RedisCluster to combine multiple commands into one transaction for execution. Use the monitoring and management tools provided by RedisCluster to monitor the running status of the system in real time. Use the Redis command monitor to view the running status of the system in real time.

[0036] S104. Set a unique ID as an association key in the database and use Spark Streaming to build a distributed memory computing framework for real-time data.

[0037] Get metadata information for each data source, including field name, data type, and data range, and determine the properties of each data source; set a unique ID for each data source in the database to ensure that the value of a record can be uniquely identified in different data sources; divide the real-time data stream into batches and use Spark Streaming to process each batch of data; use Spark Streaming to build a distributed memory computing framework and load real-time data into the distributed memory computing framework; use Spark Streaming's connection operation to find matching records in different data sources based on the unique ID; set the processing window size and data refresh frequency to reduce data latency; use data synchronization methods to ensure data consistency before data association.

[0038] For example, there are two data sources: data source A and data source B. Data source A contains order information, and the fields include order ID, order amount, and order date; data source B contains customer information, and the fields include customer ID, customer name, and customer address. Get the metadata information of each data source. For data source A, the field names are order ID, order amount, and order date, and the data types are integer, floating point, and date respectively. The data range is that the order ID ranges from 1000 to 9999, the order amount ranges from 0 to 10000, and the order date ranges from January 1, 2010 to December 31, 2022. For data source B, the field names are customer ID, customer name, and customer address, and the data types are integer, string, and string respectively. The data range is that the customer ID ranges from 100 to 999, the customer name is a string with a length of no more than 50, and the customer address is a string with a length of no more than 100. Next, set a unique ID for each data source in the database, set the unique ID for data source A to 1, and set the unique ID for data source B to 2. In this way, the value of a record can be uniquely identified in different data sources through a unique ID. Divide the real-time data stream into batches and process it using Spark Streaming. Set each batch of data to contain 100 records, and perform data processing and analysis in Spark Streaming. Use Spark Streaming to build a distributed memory computing framework and load real-time data into the distributed memory computing framework; associate the order information of data source A with the customer information of data source B, associate them according to the order ID and customer ID, and load the associated results into memory for subsequent data query and analysis. Use Spark Streaming's join operation to find matching records in different data sources based on unique IDs. Find order information in data source A by order ID, and find customer information in data source B by customer ID. Set the processing window size and data refresh frequency to reduce data latency. Set the processing window size to 5 seconds and the data refresh frequency to 1 second to ensure timely processing and updating of data. Use data synchronization to ensure data consistency before data association. Use the data synchronization tool to ensure that the data in data source A and data source B are consistent before association to avoid data inconsistency.

[0039] S105. Build a long short-term memory recursive neural network model and perform parallel training through a parameter server architecture.

[0040] Obtain historical data streams and construct data stream data sets; construct a long short-term memory recursive neural network model, set the number of layers of the long short-term memory recursive neural network model, the number of hidden units in each layer, and the dimensions of input and output; construct the number and distribution strategy of parameter servers, and select a combination of a main parameter server and several working nodes; the main parameter server is responsible for synchronization and update of model parameters, and the working nodes are responsible for calculation and communication; input the data stream data set to the main parameter server, the parameter server divides the data stream data set into batches, and distributes each batch of data to different working nodes; each working node calculates the distributed data, and each working node uses the allocated data to train the long short-term memory neural network model. After all nodes complete the calculation, the calculation results are sent to the main parameter server, and the main parameter server updates the parameters of the long short-term memory neural network model according to the received calculation results; determine whether the model training is completed according to the preset number of training iterations or convergence conditions; if the preset convergence conditions are not met, change the gradient of each training and train again; if the preset convergence conditions have been met, end the training and obtain a trained long short-term memory recursive neural network model.

[0041] For example, a two-layer LSTM recurrent neural network model is constructed. Each layer has 10 hidden units. The input dimension is 50 and the output dimension is 10. A combination of 1 parameter server and 2 worker nodes is selected. One of the parameter servers is used as the main parameter server, which is responsible for synchronizing and updating the model parameters, and the two worker nodes are responsible for calculation and communication. Now, the training data is divided into batches, each batch contains 100 sample data, and the data of each batch is distributed to different worker nodes. Worker node 1 obtains the data of batches 1-50, and worker node 2 obtains the data of batches 51-100. Each worker node uses the allocated data to train the LSTM neural network model. Worker node 1 and worker node 2 send the calculation results to the main parameter server. After receiving the calculation results of worker node 1 and worker node 2, the main parameter server updates the parameters of the LSTM neural network model according to the received calculation results. After the main parameter server updates the model parameters according to the calculation results, the new model parameters obtained are W1=5 and W2=3. According to the preset number of training iterations or convergence conditions, it is judged whether the model training is completed. The number of training iterations is set to 2000. After the 2000th iteration, it is found that the loss function value has converged and no longer changes. Therefore, it is judged that the model has been trained. If the preset convergence condition is not reached, the gradient of each training will be changed and training will be performed again. Adjust the learning rate or increase the number of training iterations to obtain better model performance. When the model training is completed, a trained long short-term memory recurrent neural network model is obtained, in which the model parameters are W1=8 and W2=6.

[0042] S106. If the long short-term memory recursive neural network model requires incremental learning, a parameter interface is reserved for iterative updating to avoid repeated training.

[0043] Define a parameter interface to receive parameter update requests for new data; obtain new training data and input the new training data into the parameter server; determine the step size for updating the model parameters each time, and use gradient descent to iteratively update the existing model using the new data; end the training until the specified number of iterations is reached, and obtain the incremental model parameters; complete the incremental learning process, and update the long short-term memory recursive neural network model parameters according to the incremental model parameters.

[0044] For example, incremental learning is required for newly added user data. After model training is completed, the parameters are saved to a file, including the Lyr1 weight matrix, the Lyr2 bias vector, etc. Subsequently, 2 million user data are added every week to update the model. Define a parameter interface to receive parameter update requests for new data. The interface contains the following parameters: learning rate, which is used to control the step size of each update of model parameters; number of iterations, which is used to specify the number of training iterations; new training data, which is new data for training. According to the learning rate set in the parameter interface, the step size of each update of model parameters is determined. If the learning rate is 0.03, the parameters of the model will be updated at a ratio of the current gradient multiplied by 0.03 at each update. Use the gradient descent algorithm to iteratively update the existing model with new data. Calculate the error between the new data and the prediction result of the current model, and calculate the gradient of the parameters through the back propagation algorithm. Then, update the parameters of the model according to the product of the step size and the gradient, so that the error gradually decreases. Repeat the iterative training until the specified number of iterations. At each iteration, the model will be updated according to the new data and the model parameters will be gradually optimized. After the specified number of iterations, the incremental model parameters are obtained, and the original LSTM model parameters can be updated according to the incremental model parameters. The incremental parameters are updated by adding the weights to the original parameters, where the weight of the incremental parameters is 0.2 and the weight of the original parameters is 0.8, thereby obtaining the updated LSTM model parameters.

[0045] S107. After training is completed, the model is deployed on the GPU server to predict the real-time data stream.

[0046] Export the trained LSTM RNN model and load it to the GPU server, and load the model parameters and structure into the memory of the GPU device; build a real-time data pipeline and preprocess the received data to match the input format of the LSTM RNN model; use the LSTM RNN model loaded on the GPU server to perform real-time inference on the preprocessed streaming data, and obtain prediction results from the LSTM RNN model.

[0047] For example, export the LSTM recurrent neural network model as a file named "model.pt" containing model parameters and structure. Load the model to a GPU server for inference. The model can accept data of size [3,224,224] as input. Next, build a real-time data pipeline to receive and preprocess streaming data. Use Socket programming to receive real-time data and use OpenCV to convert it to a format acceptable to the model. Resize the data to 224x224 and normalize the pixel values ​​to the range [0,1]. Load the preprocessed data input to the LSTM recurrent neural network model on the GPU server for real-time inference and obtain the prediction results.

[0048] S108. Through the task scheduler of the distributed memory computing framework, the collaboration between threads and processes is automatically managed to simplify programming work.

[0049] Set up a unified task scheduling component to manage the execution collaboration between threads and processes; create a thread pool, create several threads and reuse them to execute tasks; create a process pool, create several processes and reuse them; use mutex locks to achieve collaboration and concurrency between threads or processes; set up monitoring components to monitor application deployment status, task execution, and resource consumption.

[0050] For example, a task scheduling component needs to manage 20 threads and 10 processes, where the execution time of a thread is 1 second and the execution time of a process is 2 seconds. Use thread pools and process pools to execute these tasks. Create a thread pool and set the maximum number of threads to 10. Assign 20 tasks to the thread pool for execution. After each task is executed, output the execution time of the task and the thread number. Create a process pool and set the maximum number of processes to 5. Assign 10 tasks to the process pool for execution. After each task is executed, output the execution time of the task and the process number. During the execution of threads and processes, use mutex locks to ensure the mutual exclusivity of threads and processes when accessing tasks, thereby achieving cooperation and concurrency between threads or processes and avoiding concurrency conflicts. Finally, use the monitoring component to monitor the deployment status, task execution, and resource consumption of the application. The monitoring component periodically outputs the number of threads and processes in the thread pool and process pool, as well as the execution time of each task. Analyze the performance of the task scheduling component through the data obtained by the monitoring component.

Claims

1. A parallel computing method supporting real-time data stream processing in parallel programming, characterized in that: The method comprises: The real-time data stream is divided into shards through parallel pipelines, and different threads filter the specified fields to improve efficiency; after filtering, the local aggregation function of MapReduce is used to locally aggregate the results to reduce the amount of data; a distributed memory database based on RedisCluster is established to store data in shards according to time windows; a unique ID is set as the association key in the database, and Spark Streaming is used to build a distributed memory computing framework for real-time data; a long short-term memory recursive neural network model is built, and parallel training is performed through the parameter server architecture; if the long short-term memory recursive neural network model requires incremental learning, a parameter interface is reserved for iterative updates to avoid repeated training; after training, the model is deployed to the GPU server to predict the real-time data stream; the task scheduler of the distributed memory computing framework is used to automatically manage the collaboration between threads and processes to simplify programming.

2. The method according to claim 1, wherein: The real-time data stream is divided into shards through parallel pipelines, and different threads filter the specified fields to improve efficiency, including: Determine the number of shards to be divided; Create a corresponding number of threads based on the number of shards, and associate each thread with the corresponding shard. Each thread processes a shard independently. In each thread, filter the data in the shard based on the conditions of the specified field, and only retain the data that meets the conditions. In a parallel pipeline, monitor the processing status and performance indicators of each thread.

3. The method according to claim 1, wherein: After the filtering is completed, the local aggregation function of MapReduce is used to locally aggregate the results to reduce the amount of data, including: According to the filtering conditions, the input data is mapped into key-value pairs and the data with the same key is merged; according to the key of the key-value pair, the compressed output results are grouped according to the partitioning rules to ensure that the data with the same key is sent to the same folder for processing; The folder locally aggregates the input data, and merges and compresses the output results to reduce the amount of data transmission; the output results of multiple folders are merged to form the final aggregation result, and the final aggregation result is output.

4. The method according to claim 1, wherein: The distributed memory database based on RedisCluster is established to store data in time window segments, including: Determine the time window size and data sharding granularity; shard and store data according to the time window, and store the data in each time window on an independent Redis node; write the data to the master node in the RedisCluster, which is responsible for writing and replicating data, and backing up and reading data from the slave node; Periodically persist the data in memory through RDB snapshots; Through the transaction operation provided by RedisCluster, multiple commands are combined into one transaction for execution; The monitoring and management tools provided by RedisCluster can be used to monitor the system's operating status in real time.

5. The method according to claim 1, wherein: The unique ID is set as the association key in the database, and Spark Streaming is used to build a distributed memory computing framework for real-time data, including: Obtain metadata information for each data source, including field name, data type, and data range, and determine the properties of each data source; Set a unique ID for each data source in the database to ensure that the value of a record can be uniquely identified in different data sources; Divide the real-time data stream into batches and use Spark Streaming to process each batch of data; use Spark Streaming to build a distributed memory computing framework and load the real-time data into the distributed memory computing framework; Use Spark Streaming's join operation to find matching records in different data sources based on unique IDs; set the processing window size and data refresh frequency to reduce data latency; use data synchronization methods to ensure data consistency before data association.

6. The method according to claim 1, wherein: The long short-term memory recursive neural network model is constructed and trained in parallel through a parameter server architecture, including: Obtain historical data streams and construct data stream datasets; construct a long short-term memory recursive neural network model, set the number of layers of the long short-term memory recursive neural network model, the number of hidden units in each layer, and the dimensions of input and output; construct the number and distribution strategy of parameter servers, and select a combination of a main parameter server and several working nodes; the main parameter server is responsible for synchronization and updating of model parameters, and the working nodes are responsible for calculation and communication; Input the data stream data set to the main parameter server, the parameter server divides the data stream data set into batches, and distributes each batch of data to different working nodes; each working node calculates the distributed data, and each working node uses the allocated data to train the long short-term memory neural network model. After all nodes complete the calculation, the calculation results are sent to the main parameter server, and the main parameter server updates the parameters of the long short-term memory neural network model according to the received calculation results; Determine whether the model training is complete based on the preset number of training iterations or convergence conditions; If the preset convergence condition is not reached, the gradient of each training is changed and training is performed again; If the preset convergence condition has been reached, the training is terminated and a trained long short-term memory recurrent neural network model is obtained.

7. The method according to claim 1, wherein: If the long short-term memory recursive neural network model requires incremental learning, a parameter interface is reserved for iterative updating to avoid repeated training, including: Define a parameter interface to receive parameter update requests for new data; Get new training data and input the new training data into the parameter server; Determine the step size for each update of the model parameters, and use gradient descent to iteratively update the existing model using new data; until the training reaches the specified number of iterations, end the training and obtain the incremental model parameters; complete the incremental learning process, and update the long short-term memory recursive neural network model parameters according to the incremental model parameters.

8. The method according to claim 1, wherein: After the training is completed, the model is deployed on the GPU server to predict the real-time data stream, including: Export the trained LSTM recurrent neural network model and load it to the GPU server, and load the model parameters and structure into the memory of the GPU device; Build a real-time data pipeline and preprocess the received data to match the input format of the LSTM RNN model; use the LSTM RNN model loaded on the GPU server to perform real-time inference on the preprocessed streaming data and obtain prediction results from the LSTM RNN model.

9. The method according to claim 1, wherein: The task scheduler of the distributed memory computing framework automatically manages the collaboration between threads and processes, and simplifies programming work, including: Set up a unified task scheduling component to manage the execution collaboration between threads and processes; Create a thread pool, create several threads and reuse them to execute tasks; create a process pool, create several processes and reuse them; use mutex locks to achieve collaboration and concurrency between threads or processes; set up monitoring components to monitor application deployment status, task execution and resource consumption.

Images