Real-time data stream processing method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of data processing, and discloses a real-time data stream processing method and device, electronic equipment and a storage medium. The method comprises the following steps: distributing a received to-be-processed data stream to a first distributed node through a preset load balancer; fragmenting the received distribution data stream at the first distributed node according to a preset fragmentation rule, then writing the fragmented data stream into a message queue, and performing parallel processing on fragmented data in the message queue through multiple threads; storing the intermediate state data in the fragmented data processing process, generating snapshot data according to the fragmented data, and synchronizing the snapshot data to the distributed storage system; acquiring load data of the first distributed node, and dynamically updating a data flow distribution strategy of the load balancer according to the load data through the resource use prediction model; and writing a processing result of the to-be-processed data stream into persistent storage through an asynchronous batch mechanism, and returning the processing result through a preset real-time communication protocol. The method improves the data stream processing efficiency and stability.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and in particular, to a real-time data stream processing method, apparatus, electronic device, and storage medium. Background Art

[0002] In application fields such as finance and healthcare that have high requirements for the real-time performance and reliability of data processing, traditional data stream processing technologies face severe challenges. For example, high-frequency financial trading needs to quickly complete risk assessment and fraud detection, and medical intensive care requires real-time processing of multi-parameter vital sign data of patients. However, traditional stream data processing technologies are difficult to meet these requirements. Although Apache Kafka and Apache Flink respectively provide a reliable messaging system and high-throughput and low-latency processing capabilities, the former has complex deployment management and high resource requirements, and the latter has a complex architecture and high integration costs, which are limited in the application scenarios of resource-constrained financial branches and grass-roots medical care. And Node.js Streams, as a stream processing mechanism suitable for simple tasks, also has prominent performance and scalability problems when processing high-throughput data. In the high-frequency financial trading scenario, a large number of transaction data per second will cause the processing speed to drop sharply due to the processing bottleneck of Node.js, affecting the timeliness of decision-making; in the medical field, when multiple devices concurrently upload data, it will also cause delays in patient data processing and delays in critical value alarms. Moreover, the existing stream processing capabilities of Node.js are limited under high concurrency and large data volumes, and its scalability depends on external tools and increases system complexity. When the data traffic is large and the logic is complex, delays are likely to occur, affecting real-time performance, and it also lacks built-in fault tolerance and data recovery mechanisms, resulting in a high risk of data loss. Summary of the Invention

[0003] The main technical problem to be solved by the embodiments of this application is the deficiencies in high concurrency, low latency, and scalability of some traditional data stream processing technologies.

[0004] To solve the above technical problems, the first technical solution adopted in the embodiments of the present application is: to provide a real-time data stream processing method, including: distributing the received data stream to be processed to the deployed first distributed node through a preset load balancer; sharding the received distributed data stream according to a preset sharding rule in the first distributed node and writing it into a message queue, and processing the sharded data in the message queue in parallel through independent threads in a multi-thread; dynamically storing the intermediate state data in the process of processing the sharded data through an in-memory database, generating snapshot data according to the sharded data, and synchronizing the snapshot data to a preset distributed storage system using an incremental snapshot mechanism; collecting the load data of the first distributed node in real time, and dynamically updating the data stream distribution policy of the load balancer according to the load data through a trained resource usage prediction model; writing the processing result corresponding to the data stream to be processed into a persistent storage through an asynchronous batch mechanism, and returning the processing result through a preset real-time communication protocol.

[0005] Optionally, after the step of dynamically storing the intermediate state data in the process of processing the sharded data through the in-memory database, it further includes: recording the processing state data and intermediate result data when the sharded data is processed in the independent thread; generating intermediate state data according to the processing state data and intermediate result data, and storing the intermediate state data in the preset in-memory database; when the processing task corresponding to the sharded data is restarted, or the first distributed node corresponding to the sharded data resumes operation, obtaining the latest intermediate state data from the in-memory database; restoring the execution progress of processing the sharded data according to the latest intermediate state data.

[0006] Optionally, the step of generating snapshot data based on the sharded data and synchronizing the snapshot data to a preset distributed storage system using an incremental snapshot mechanism includes: when a preset snapshot generation event mechanism is triggered, or when the current time reaches a preset time period switching point, determining whether the hash value of the current sharded data processing status is the same as the hash value of the previous snapshot data; if the hash values are different, obtaining the change log or incremental record of the in-memory database as incremental data, and generating snapshot data based on the incremental data and the metadata of the current sharded data; performing consistent hashing calculation according to the unique encoding of the current sharded data, and mapping the corresponding calculation result to the target node of the distributed storage system; storing the snapshot data in a hierarchical manner to the target node of the distributed storage system according to the generation timestamp of the snapshot data; if the first distributed node detected to have failed is detected, obtaining the corresponding snapshot data from the target node of the distributed storage system according to the unique encoding of the sharded data and the generation timestamp; restoring the snapshot data to the new fault-free first distributed node through the load balancer, and continuing to execute the data stream processing task according to the data stream processing progress in the snapshot data.

[0007] Optionally, the step of real-time collecting the load data of the first distributed node and dynamically adjusting the data stream distribution policy of the load balancer through a trained resource usage prediction model includes: real-time collecting the processor usage rate, memory usage rate, network bandwidth occupancy rate, disk read / write rate, and task processing delay time of the first distributed node as the load data; normalizing the load data and converting the normalized load data into time series data acceptable to the resource usage prediction model, where the resource usage prediction model is constructed based on a long short-term memory network; sequentially processing the time series data through the forget gate, input gate, memory unit, and output gate of the resource usage prediction model to obtain the resource usage hidden state at each time step; inputting the resource usage hidden state data of the last time step into a fully connected layer for processing to obtain the resource usage trend prediction result of the first distributed node; updating the data stream distribution policy of the load balancer according to the resource usage trend prediction result.

[0008] Optionally, after the step of parallelly processing the shard data in the message queue by independent threads in a multi-threaded manner, the method further includes: continuously monitoring the response status and current performance data of each of the first distributed nodes using a monitoring tool; when it is detected that a first distributed node loses response or has abnormal performance, marking the first distributed node as a faulty state; according to the load data of the first distributed nodes in the current non-faulty state, redistributing the new to-be-processed data stream to the first distributed nodes in the non-faulty state; obtaining the first distributed nodes in the non-faulty state as transfer nodes; and resuming the processing task of the corresponding shard data on the transfer nodes according to the intermediate state data and the snapshot data of the first distributed node in the faulty state.

[0009] Optionally, after the step of parallelly processing the shard data in the message queue by independent threads in a multi-threaded manner, the method further includes: matching the target data processing type corresponding to the to-be-processed data stream in a preset data type library, and generating different target data processing retry policies according to the target data processing type; when the independent thread processing the shard data triggers a preset exception capture mechanism, determining whether the retry execution times of the target data processing retry policy are greater than a preset retry times threshold; if not greater than the retry times threshold, executing the target data processing retry policy and accumulating the retry execution times; if greater than the retry times threshold, sending a data stream processing warning to a preset warning recipient.

[0010] Optionally, after the step of sharding the received distributed data stream according to a preset sharding rule by the first distributed node and then writing it into the message queue, the method further includes: monitoring the data volume and processing time of the shard data; if the processing time of the shard data is greater than a preset first processing time threshold, reducing the preset value of the single shard data volume in the sharding rule according to the data volume; if the processing time of the shard data is less than the preset first processing time threshold, increasing the preset value of the single shard data volume in the sharding rule according to the data volume.

[0011] To solve the above technical problems, the second technical solution adopted in the embodiments of the present application is: to provide a real-time data stream processing device, including: a data stream distribution module, configured to distribute the received data stream to be processed to the deployed first distributed node through a preset load balancer; a data stream sharding module, configured to shard the received distributed data stream according to a preset sharding rule in the first distributed node and then write it into a message queue, and process the sharded data in the message queue in parallel through independent threads in a multi-threaded manner; a data processing snapshot module, configured to dynamically store intermediate state data during the processing of the sharded data through an in-memory database, generate snapshot data according to the sharded data, and synchronize the snapshot data to a preset distributed storage system using an incremental snapshot mechanism; a distribution policy optimization module, configured to collect the load data of the first distributed node in real time, and dynamically update the data stream distribution policy of the load balancer according to the load data through a trained resource usage prediction model; a data processing result module, configured to write the processing result corresponding to the data stream to be processed into persistent storage through an asynchronous batch mechanism, and return the processing result through a preset real-time communication protocol.

[0012] To solve the above technical problems, the third technical solution adopted in the embodiments of the present application is: to provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor is enabled to execute the real-time data stream processing method as described above.

[0013] To solve the above technical problems, the fourth technical solution adopted in the embodiments of the present application is: to provide a non-volatile computer-readable storage medium, which stores computer-executable instructions, and when the computer-executable instructions are executed by an electronic device, the electronic device is enabled to execute the real-time data stream processing method as described above.

[0014] Different from the related art, the present application realizes seamless recovery of data stream processing by dynamically distributing data streams through a load balancer, sharding and parallel processing in a multi-threaded manner, storing intermediate states and incremental snapshots in memory, optimizing the scheduling policy with a resource prediction model, and combining asynchronous batch writing and a real-time communication protocol, solves the core problems of high-concurrency processing, uneven resources, and delayed fault recovery in traditional distributed systems, achieves the effects of improving processing efficiency, shortening fault recovery time, and optimizing resource utilization rate, and provides reliable technical support for scenarios with high real-time requirements and large amounts of data. Description of the Drawings

[0015] One or more embodiments are exemplarily illustrated by corresponding drawings. These exemplary illustrations do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings are represented as similar elements. Unless otherwise stated, the drawings in the figures do not constitute a scale limitation.

[0016] Figure 1 It is a schematic diagram of the operating environment of the real-time data stream processing method provided by an embodiment of the present application.

[0017] Figure 2 It is a schematic diagram of the execution flow of the real-time data stream processing method provided by an embodiment of the present application.

[0018] Figure 3 It is a schematic diagram of the execution flow of restoring sharded data in the real-time data stream processing method provided by an embodiment of the present application.

[0019] Figure 4 It is a schematic diagram of the execution flow of restoring snapshot data in the real-time data stream processing method provided by an embodiment of the present application.

[0020] Figure 5 It is a schematic diagram of the system structure of the real-time data stream processing device provided by an embodiment of the present application.

[0021] Figure 6 It is a schematic diagram of the hardware structure of an electronic device for executing the real-time data stream processing method provided by an embodiment of the present application. Detailed Embodiments

[0022] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0023] It should be noted that if there is no conflict, the various features in the embodiments of the present application can be combined with each other, and all are within the protection scope of the present application. In addition, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order from the module division in the device schematic diagram or the flowchart.

[0024] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as commonly understood by those skilled in the technical field to which this application belongs. The terms used in the specification of this application are only for the purpose of describing specific embodiments and are not used to limit this application. The term "and / or" used in this specification includes any and all combinations of one or more of the related listed items.

[0025] To facilitate the understanding of this embodiment, first, a real-time data stream processing method disclosed in the embodiments of the present application will be introduced in detail. Please refer to Figure 1 , Figure 1 which is a schematic diagram of the operating environment of the real-time data stream processing method provided by the embodiments of the present application. As Figure 1 shown, the execution subject of the real-time data stream processing method provided by the embodiments of the present application is generally an electronic device with certain computing capabilities, such as a computer device. In some possible implementation manners, the real-time data stream processing method can be implemented by a processor calling computer-readable instructions stored in a memory. Among them, Figure 1 the computer device in Figure 1 can be a server. The server can be an independent server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms. It can be understood that

[0026] Please continue to refer to Figure 2 , Figure 2 which is a schematic diagram of the execution flow of the real-time data stream processing method provided by the embodiments of the present application. As Figure 2 shown, it includes the following steps:

[0027] S1. Distribute the received data stream to be processed to the deployed first distributed node through a preset load balancer.

[0028] Among them, in step S1, a load balancer can reasonably allocate data streams to avoid excessive or insufficient load on a certain node, achieve load balancing among nodes, and improve the processing capacity, reliability, and scalability of the entire system. The load balancer will analyze and judge the received data stream to be processed according to preset rules and algorithms, and then evenly distribute it to multiple deployed first distributed nodes. For example, in the medical information system of a large hospital, a large amount of patient data is generated every day, such as medical record information, inspection reports, imaging materials, etc. These data need to be processed and stored in a timely manner to support doctors' diagnosis and treatment work. The preset load balancer will distribute the received patient data according to factors such as the load situation and processing capacity of each first distributed node. Another example is in the financial transaction system, where there are a large number of transaction requests every day, such as stock trading, fund subscription and redemption, transfer and remittance, etc. These transaction requests need to be processed in a timely manner to ensure the smooth progress of transactions and the safety of funds. For some small transfer and remittance requests, the load balancer may distribute them to nodes with faster response speeds to improve the user experience; while for large stock trading requests, the load balancer will distribute them to nodes with stronger processing capabilities and higher security levels to ensure the accuracy of transactions and the safety of funds.

[0029] S2. After the first distributed node slices the received distributed data stream according to the preset slicing rules, write it into the message queue, and parallelly process the sliced data in the message queue through independent threads in multiple threads.

[0030] Among them, the real-time data stream solution based on Node.js Streams allows the processing of data streams through pipelines by using the built-in stream processing mechanism of Node.js. It is suitable for simple stream processing tasks. However, when dealing with high-throughput data, there are bottlenecks in performance and scalability. By using the Worker Threads module of Node.js to design a multi-threaded real-time stream processing architecture, the data stream tasks are assigned to multiple threads for execution, thus breaking through the single-thread bottleneck. For example, first, according to the preset sharding rules (such as user ID hash, time window, or business type), the entire batch of data stream is cut into multiple independent shards to ensure that the data of each shard has logical relevance and parallelizability. Then, the sharded data is written into a message queue (such as Kafka, RabbitMQ, etc.), and the data is ensured not to be lost and the traffic peak is smoothed through the persistence mechanism of the queue. Finally, the multi-threaded architecture is started, and each shard is processed in parallel by an independent thread. The threads coordinate the state through memory sharing or distributed locks to achieve parallel computing at the shard level. This design improves the fault tolerance through shard isolation (if there is an exception in a shard, it does not affect the processing of other shards), significantly increases the data throughput by using multi-thread parallelism (for example, a data stream of 100,000 TPS can be increased to nearly linear growth through parallel processing of 10 shards), and at the same time, the introduction of the message queue ensures the orderliness of data stream processing and the elastic scaling ability of the system.

[0031] As an alternative implementation, after the step of parallelly processing the sharded data in the message queue by independent threads in step S2, the following steps S201 to S205 may further be included.

[0032] S201. Continuously monitor the response status and current performance data of each first distributed node by using a monitoring tool.

[0033] As an example, by deploying a monitoring tool (such as Prometheus+Grafana, Zabbix, etc.), the response status of the first distributed node (such as node survival status, request latency, error rate) and performance data (CPU utilization, memory occupancy, disk I / O, network throughput) are collected and analyzed in real time. The monitoring system continuously polls or obtains data through the active reporting of nodes, and displays the trends of key indicators through a visualization interface. For example, when the CPU utilization of a certain node exceeds the threshold, the monitoring tool will trigger an alarm; if a node does not respond for 30 consecutive seconds, it is marked as a faulty node.

[0034] S202. When it is detected that the first distributed node loses response or has performance anomalies, mark the first distributed node as a faulty state.

[0035] S203. Redistribute the new data stream to be processed to the first distributed nodes in a non-fault state according to the load data of the first distributed nodes in the current non-fault state.

[0036] Among them, the above steps S202 and S203 constitute a dynamic adjustment mechanism for dealing with node failures in the real-time data stream processing system. This mechanism can timely detect and mark the faulty nodes, quickly redistribute the new data stream to be processed to the non-faulty nodes, ensure the stable operation of the system when some nodes fail, reduce service interruptions, achieve load balancing, and improve the overall availability and processing efficiency of the system.

[0037] S204. Obtain the first distributed nodes in a non-fault state as transfer nodes.

[0038] S205. According to the intermediate state data and snapshot data of the first distributed nodes in a fault state, resume the processing tasks of the corresponding shard data on the transfer nodes.

[0039] Among them, steps S204 and S205 jointly construct an efficient task recovery mechanism for the real-time data stream processing system in the face of node failures, ensuring that the selected transfer nodes have sufficient resources and capabilities to undertake the tasks left by the faulty nodes. For example, in a medical information system, each node is responsible for processing important data such as patient medical records, examination reports, and imaging materials. If a node suddenly fails, through the above task recovery mechanism, a suitable transfer node can be quickly found, and based on the intermediate state data and snapshot data of the faulty node, the processing of patient data can be resumed, ensuring that doctors can timely obtain the latest information of patients and will not delay diagnosis and treatment due to node failures. In the financial field, taking the stock trading system as an example, a large number of trading orders need to be processed every day, and each node undertakes important trading processing tasks. Once a node fails, it may lead to trading interruptions and cause huge losses to investors. Through the above task recovery mechanism, the system can quickly transfer the tasks of the faulty node to a suitable transfer node to continue execution, ensuring the smooth progress of trading, not only maintaining the stability of the financial market but also protecting the interests of investors.

[0040] As another alternative implementation, after the step of parallel processing of the shard data in the message queue by independent threads in the multi-threading in step S2, the following steps S211 to S214 may further be included.

[0041] S211. Match the target data processing type corresponding to the data stream to be processed in the preset data type library, and generate different target data processing retry policies according to the target data processing type.

[0042] Among them, during the data stream processing, different data types face different situations and impacts when errors occur. Therefore, different retry strategies need to be adopted. For example, transaction data in the financial field has high requirements for timeliness and accuracy, and usually adopts a more aggressive retry strategy, such as exponential backoff retry. That is, the first retry interval is relatively short (such as 1 second), and then the retry interval increases exponentially each time (such as 2 seconds, 4 seconds, 8 seconds, etc.). This can attempt to recover the processing in the shortest possible time while avoiding excessive pressure on the system due to frequent retries. In the medical field, the volume of imaging diagnosis data is large and the processing is complex, which may involve multiple steps such as image recognition and analysis. Processing errors may be due to insufficient computing resources or algorithm problems. Before retrying, the computing resource usage of the system can be checked first. If the resources are insufficient, resource allocation can be performed first, and then the retry can be carried out. For another example, if the image recognition algorithm makes an error, the algorithm version can be tried to be updated before retrying.

[0043] S212. When the independent thread processes the sharded data and triggers the preset exception capture mechanism, determine whether the retry execution count of the target data processing retry strategy is greater than the preset retry count threshold.

[0044] S213. If it is not greater than the retry count threshold, execute the target data processing retry strategy and accumulate the retry execution count.

[0045] S214. If it is greater than the retry count threshold, send a data stream processing warning to the preset warning recipient.

[0046] Among them, the above steps S211 to S214 construct a dynamic retry strategy management mechanism based on data types. By matching the processing requirements of different data streams through the preset data type library, differential fault tolerance solutions such as exponential backoff retry for financial transaction data and resource allocation retry for medical imaging data are realized. This mechanism dynamically determines the retry count threshold during exception capture, which not only ensures the rapid recovery of high-timeliness data (such as financial transactions), but also avoids infinite retries for complex data processing (such as medical imaging). Ultimately, while improving the system's fault tolerance ability, it reduces resource consumption through intelligent retry control, and triggers the warning mechanism after exceeding the limit to achieve an operation and maintenance closed-loop.

[0047] As another alternative implementation, after the step of fragmenting the received distribution data stream according to a preset fragmentation rule and writing it into the message queue at the first distributed node in step S2, it is also possible to monitor the data volume and processing time consumption of the fragmented data; if the processing time consumption of the fragmented data is greater than a preset first processing time consumption threshold, the preset value of the single fragmented data volume in the fragmentation rule is reduced according to the data volume; if the processing time consumption of the fragmented data is less than the preset first processing time consumption threshold, the preset value of the single fragmented data volume in the fragmentation rule is increased according to the data volume. On the one hand, reducing the single fragmented data volume when the processing time consumption is too long can effectively improve the parallelism of data processing, reduce the processing pressure of each independent thread, avoid processing bottlenecks caused by excessive data volume, and improve the overall processing efficiency and response speed of the system. On the other hand, if the processing time consumption is short, increasing the single fragmented data volume can reduce the switching overhead and management cost between threads, make more full use of system resources, and improve resource utilization. In fields such as medical and finance where extremely high requirements are placed on data processing timeliness, it is possible to flexibly adapt to the data processing requirements in different business scenarios, ensuring that the system can operate stably and efficiently under different data volumes and processing complexities.

[0048] S3. Dynamically store the intermediate state data during the processing of the fragmented data through an in-memory database, generate snapshot data according to the fragmented data, and use an incremental snapshot mechanism to synchronize the snapshot data to a preset distributed storage system.

[0049] Among them, an in-memory database (such as Redis) is a high-speed database that stores data in memory, with characteristics of low latency and high concurrent access, suitable for storing temporary data that needs to be read and written quickly. During the processing of fragmented data, the in-memory database is used to store intermediate state data in real time, that is, the temporary results or intermediate calculation states generated during the data stream processing (such as partial settlement records of financial transaction fragments). These data need to be quickly restored in case of processing failure. Snapshot data refers to a complete mirror of the intermediate state data at a specific time point, used to save the system state for subsequent recovery. The incremental snapshot mechanism only synchronizes the data that has changed since the last snapshot (such as newly added annotation information in the processing of medical image fragments), improving the synchronization efficiency by reducing the data transmission volume. The distributed storage system (such as HDFS) is an infrastructure for persistently storing snapshot data, with high scalability and fault tolerance. By synchronizing the snapshot data in the in-memory database to the distributed storage, it can ensure that even if the in-memory database fails, the data can be restored from the persistent storage, guaranteeing the high availability of the system.

[0050] As an alternative implementation, please continue to refer to Figure 3 , Figure 3 which is a schematic diagram of the execution process for restoring and processing fragmented data in the real-time data stream processing method provided by the embodiments of the present application. As shown in Figure 3As shown in the figure, it specifically includes the following steps S301 to S304.

[0051] S301. Record the processing status data and intermediate result data when the sharded data is processed in an independent thread.

[0052] Among them, the processing status data is mainly used to describe information such as the current stage and status of the sharded data during the processing in the independent thread, which can reflect the progress of the processing flow and whether abnormal situations are encountered, helping the system comprehensively understand the real-time situation of data processing. For example, the processing stage identifier, processing progress, error information, processing time, etc. The intermediate result data refers to the phased calculation results or data conversion results generated by each processing step during the process of processing the sharded data in the independent thread. For example, when performing complex calculations on the sharded data, it may need to be completed in multiple steps, and the calculation results generated by each step are the intermediate result data.

[0053] S302. Generate intermediate status data based on the processing status data and intermediate result data, and store the intermediate status data in a preset in-memory database.

[0054] S303. When the processing task corresponding to the sharded data is restarted, or when the first distributed node corresponding to the sharded data resumes operation, obtain the latest intermediate status data from the in-memory database.

[0055] S304. Restore the execution progress of processing the sharded data according to the latest intermediate status data.

[0056] Among them, the above steps S301 to S304 construct an efficient data processing and recovery mechanism based on intermediate status data. By recording the status and intermediate results of sharded data processing in real time and constructing intermediate status data, accurate breakpoint resumption is achieved when the task is restarted or the node is restored. Compared with the overall migration of failed node recovery, its advantage lies in focusing on the continuity of the data processing process. It can avoid repeated calculations in financial transactions and ensure the uninterrupted diagnostic process in medical image processing. At the same time, it provides a basis for processing progress monitoring and resource optimization for the system, improves the fault tolerance ability by dynamically adjusting strategies, and forms a synergistic complement with node fault recovery to jointly ensure the efficient and stable processing of real-time data streams.

[0057] As an alternative implementation, please continue to refer to Figure 4 , Figure 4 is a schematic diagram of the execution process of restoring snapshot data in the real-time data stream processing method provided by the embodiment of the present application. As Figure 4 shown, it specifically includes the following steps S311 to S316.

[0058] S311. When the preset snapshot generation event mechanism is triggered, or when the current time reaches the preset time period switching point, determine whether the hash value of the current shard data processing status is the same as the hash value of the previous snapshot data.

[0059] Among them, the snapshot generation event mechanism is a set of preset rules or conditions. When these rules or conditions are met, the system will trigger the snapshot generation operation. For example, the data volume reaches a threshold, a specific operation is completed, or an error or exception occurs. The preset time period switching point refers to a preset specific time point or time period. When the current time reaches these time points, the system will trigger the snapshot generation operation.

[0060] S312. If the hash values are different, obtain the change log or incremental record of the in-memory database as incremental data, and generate snapshot data based on the incremental data and the metadata of the current shard data.

[0061] S313. Perform consistent hashing calculation according to the unique encoding of the current shard data, and map the corresponding calculation result to the target node of the distributed storage system.

[0062] S314. Store the snapshot data in a hierarchical manner to the target node of the distributed storage system according to the generation timestamp of the snapshot data.

[0063] Among them, the above steps S311 to S314 construct an efficient, intelligent and reliable snapshot data management and storage mechanism. In the financial trading system, this mechanism can quickly restore the trading processing status by capturing data changes in real time and generating incremental snapshots. For example, when high-frequency trading is interrupted due to a node failure, the system can accurately resume transmission according to the latest snapshot data (including metadata such as transaction amount and timestamp), avoiding repeated calculations or order losses. In the medical image processing system, this mechanism ensures the integrity of patient diagnosis data through periodic snapshots and event trigger mechanisms. For example, when the system suddenly fails while a doctor is processing a CT image, the latest snapshot can retain the intermediate states such as the marked lesion location and diagnosis opinion, and there is no need to re-analyze after recovery.

[0064] S315. If the first distributed node where a failure is detected, obtain the corresponding snapshot data from the target node of the distributed storage system according to the unique encoding and generation timestamp of the shard data.

[0065] S316. Restore the snapshot data to the new first distributed node without failure through the load balancer, and continue to execute the data stream processing task according to the data stream processing progress in the snapshot data.

[0066] Among them, the above steps S311 to S316 are based on the snapshot data recovery mechanism. With the help of the persistent feature of distributed storage, they can provide a comprehensive system state backtracking. Different from the intermediate state recovery that depends on memory, they can completely retain the processing context. Even if the memory data is lost, they can accurately locate and migrate the data to a new node to achieve seamless connection of the business process.

[0067] S4. Real-time collect the load data of the first distributed node, and dynamically update the data flow distribution strategy of the load balancer according to the load data through the trained resource usage prediction model.

[0068] As an optional implementation manner, the process of using the resource usage prediction model to optimize the data flow distribution strategy in the above step S4 may specifically include the following steps S41 to S45.

[0069] S41. Real-time collect the processor utilization rate, memory utilization rate, network bandwidth occupancy rate, disk read and write speed, and task processing delay time of the first distributed node as load data.

[0070] S42. Normalize the load data and convert the normalized load data into time series data that can be received by the resource usage prediction model, where the resource usage prediction model is constructed based on a long short-term memory network.

[0071] S43. Process the time series data through the forget gate, input gate, memory unit, and output gate of the resource usage prediction model in sequence to obtain the resource usage hidden state at each time step.

[0072] Among them, the processing process based on the long short-term memory network realizes the dynamic analysis of time series data through four core components. The forget gate determines which outdated information (such as the overnight trading load pattern in the financial system) to discard from the long-term memory through the sigmoid function. The input gate uses the sigmoid and tanh functions to screen and store the key data at the current time step (such as the sudden surge in AI model inference tasks during medical image processing). The memory unit integrates new and old information and updates the long-term state (such as recording the continuously increasing CPU utilization trend in high-frequency financial trading); the output gate generates hidden features in combination with the current state (such as predicting the memory peak value in the next 5 minutes), and finally outputs to adjust the load balancing strategy. In the financial field, it can predict the resource requirements during peak trading hours in advance and dynamically allocate computing power. In the medical scenario, it can optimize the task scheduling of distributed nodes according to the resource consumption pattern of real-time image analysis tasks to ensure the diagnostic efficiency and system stability.

[0073] S44. Input the resource usage hidden state data of the last time step into the fully connected layer for processing to obtain the resource usage trend prediction result of the first distributed node.

[0074] S45. Update the data stream distribution policy of the load balancer according to the resource usage trend prediction result.

[0075] Among them, in the above steps S41 to S45, a resource usage prediction model constructed based on a long short-term memory network is used. This model can effectively capture the long-term dependencies of the distributed node loads, accurately filter out outdated information, screen key data, integrate new and old states, and combine the resource usage trend prediction result obtained from the fully connected layer to dynamically update the data stream distribution policy of the load balancer. In the financial field, it can anticipate the resource requirements during trading peaks in advance; in the medical scenario, it can optimize the node scheduling for image analysis tasks to ensure system stability and business efficiency.

[0076] S5. Write the processing result corresponding to the data stream to be processed into persistent storage through an asynchronous batch mechanism, and return the processing result through a preset real-time communication protocol.

[0077] Among them, the asynchronous batch mechanism means that the data to be processed is first temporarily stored in the memory queue and then written into persistent storage in batches when a preset threshold (such as the data volume, time interval) is reached. This method reduces frequent I / O operations and improves the writing efficiency. For example, when the financial system processes user transfer requests, 100 transactions are asynchronously collected and then written into the database in batches to avoid network latency for single operations. The real-time communication protocol immediately returns the result through a low-latency protocol (such as WebSocket) after the data processing is completed. For example, after the medical imaging system completes the CT scan analysis, it pushes the lesion annotation result to the doctor's terminal through the real-time protocol without waiting for the batch data processing to be completed.

[0078] The real-time data stream processing method provided by the embodiments of this application realizes the efficient processing and intelligent scheduling of real-time data streams through a distributed load balancing and multi-thread parallel architecture, combined with dynamic sharding, incremental snapshots, and a resource usage prediction model. In the financial field, the prediction model anticipates the resource requirements during trading peaks and dynamically allocates computing power, combined with the hierarchical fault tolerance mechanism of the in-memory intermediate state and distributed snapshot storage, to ensure the breakpoint continuation and fund security of high-frequency transactions. In the medical scenario, the node scheduling is optimized according to the resource consumption pattern of the image analysis task, and the diagnostic progress is retained through periodic snapshots to ensure seamless recovery after the image processing is interrupted. The dynamic sharding mechanism automatically adjusts the shard size according to the processing time, and the differential retry strategies (such as exponential backoff for financial transactions and resource allocation for medical images) reduce the ineffective resource consumption while improving the fault tolerance, providing a solution with high performance, high availability, and low cost for scenarios with high timeliness and high reliability requirements.

[0079] Please continue to refer to Figure 5 , Figure 5 which is the system structure schematic diagram of the real-time data stream processing device provided by the embodiments of this application, as Figure 5As shown in the figure, the real-time data stream processing device 50 includes: a data stream distribution module 51, a data stream sharding module 52, a data processing snapshot module 53, a distribution policy optimization module 54, and a data processing result module 55.

[0080] The data stream distribution module 51 is specifically configured to distribute the to-be-processed data stream received through a preset load balancer to the deployed first distributed node.

[0081] The data stream sharding module 52 is specifically configured to, in the first distributed node, shard the received distributed data stream according to a preset sharding rule, write the sharded data stream into a message queue, and process the sharded data in the message queue in parallel through independent threads in a multi-threaded manner.

[0082] The data processing snapshot module 53 is specifically configured to dynamically store the intermediate state data during the processing of the sharded data through an in-memory database, generate snapshot data according to the sharded data, and synchronize the snapshot data to a preset distributed storage system using an incremental snapshot mechanism.

[0083] The distribution policy optimization module 54 is specifically configured to collect the load data of the first distributed node in real time, and dynamically update the data stream distribution policy of the load balancer according to the load data through a trained resource usage prediction model.

[0084] The data processing result module 55 is specifically configured to write the processing result corresponding to the to-be-processed data stream into persistent storage through an asynchronous batching mechanism, and return the processing result through a preset real-time communication protocol.

[0085] As an optional implementation manner, the data processing snapshot module 53 is specifically configured to record the processing state data and intermediate result data when the sharded data is processed in the independent thread; generate intermediate state data according to the processing state data and intermediate result data, and store the intermediate state data in the preset in-memory database; when the processing task corresponding to the sharded data is restarted, or when the first distributed node corresponding to the sharded data resumes operation, obtain the latest intermediate state data from the in-memory database; and restore the execution progress of processing the sharded data according to the latest intermediate state data.

[0086] As an alternative implementation, the data processing snapshot module 53 is further specifically configured to, when a preset snapshot generation event mechanism is triggered, or when the current time reaches a preset time period switching point, determine whether the hash value of the current shard data processing status is the same as the hash value of the previous snapshot data; if the hash values are different, obtain the change log or incremental record of the in-memory database as incremental data, and generate snapshot data according to the incremental data and the metadata of the current shard data; perform consistent hashing calculation according to the unique encoding of the current shard data, and map the corresponding calculation result to the target node of the distributed storage system; store the snapshot data in the target node of the distributed storage system in a hierarchical manner according to the generation timestamp of the snapshot data; if the first distributed node detected to have failed is found, obtain the corresponding snapshot data from the target node of the distributed storage system according to the unique encoding of the shard data and the generation timestamp; restore the snapshot data to the new first distributed node without failure through the load balancer, and continue to execute the data stream processing task according to the data stream processing progress in the snapshot data.

[0087] As an alternative implementation, the distribution strategy optimization module 54 is specifically configured to collect the processor utilization rate, memory utilization rate, network bandwidth occupancy rate, disk read and write speed, and task processing delay time of the first distributed node in real time as the load data; perform normalization processing on the load data, and convert the normalized load data into time series data acceptable to the resource usage prediction model, where the resource usage prediction model is constructed based on a long short-term memory network; process the time series data through the forget gate, input gate, memory unit, and output gate of the resource usage prediction model in sequence to obtain the resource usage hidden state at each time step; input the resource usage hidden state data of the last time step into the fully connected layer for processing to obtain the resource usage trend prediction result of the first distributed node; update the data stream distribution strategy of the load balancer according to the resource usage trend prediction result.

[0088] As an alternative implementation, the data stream sharding module 52 is further specifically configured to continuously monitor the response status and current performance data of each first distributed node using a monitoring tool; when it is detected that the first distributed node loses response or has abnormal performance, mark the first distributed node as a failed state; redistribute the new to-be-processed data stream to the first distributed nodes in a non-failed state according to the load data of the first distributed nodes in the current non-failed state; obtain the first distributed nodes in a non-failed state as transfer nodes; restore and execute the processing task of the corresponding shard data on the transfer nodes according to the intermediate state data and the snapshot data of the first distributed node in the failed state.

[0089] As an alternative implementation, the data stream sharding module 52 is further specifically configured to match the target data processing type corresponding to the data stream to be processed in a preset data type library, and generate different target data processing retry policies according to the target data processing type; when the independent thread processes the sharded data and triggers a preset exception capture mechanism, determine whether the retry execution count of the target data processing retry policy is greater than a preset retry count threshold; if it is not greater than the retry count threshold, execute the target data processing retry policy and accumulate the retry execution count; if it is greater than the retry count threshold, send a data stream processing warning to a preset warning recipient.

[0090] As an alternative implementation, the data stream sharding module 52 is further specifically configured to monitor the data volume and processing time of the sharded data; if the processing time of the sharded data is greater than a preset first processing time threshold, reduce the preset value of the single sharded data volume in the sharding rule according to the data volume; if the processing time of the sharded data is less than the preset first processing time threshold, increase the preset value of the single sharded data volume in the sharding rule according to the data volume.

[0091] It should be noted that the above real-time data stream processing device can execute the real-time data stream processing method provided in the embodiments of the present application, and has the corresponding functional modules and beneficial effects for executing the method. For the technical details not described in detail in the embodiments of the real-time data stream processing device, reference can be made to the real-time data stream processing method provided in the embodiments of the present application.

[0092] Figure 6 is a schematic hardware structure diagram of an electronic device 600 for executing the real-time data stream processing method provided in the embodiments of the present application. As Figure 6 shown, the electronic device 600 includes:

[0093] One or more processors 610 and a memory 620. Figure 6 Here, one processor 610 is taken as an example.

[0094] The processor 610 and the memory 620 can be connected through a bus or other means. Figure 6 Here, the connection through the bus is taken as an example.

[0095] The memory 620, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the real-time data stream processing method in the embodiments of the present application. The processor 610 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 620, that is, implements the real-time data stream processing method in the above method embodiments.

[0096] The memory 620 may include a program storage area and a data storage area. The program storage area may store an operating system and application programs required for at least one function. The data storage area may store data created according to the use of the real-time data stream processing device, etc. In addition, the memory 620 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 620 may optionally include a memory remotely disposed relative to the processor 610, and these remote memories may be connected to the real-time data stream processing device through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0097] The one or more modules are stored in the memory 620 and, when executed by the one or more processors 610, execute the real-time data stream processing method in any of the above method embodiments. For example, execute the Figure 2 method steps S1 to S5 described above, Figure 3 method steps S301 to S304 in Figure 4 and method steps S311 to S316 in Figure 4 to implement the functions of the modules 51-55 in

[0098] The above product can execute the method provided by the embodiments of the present application, and has corresponding functional modules and beneficial effects for executing the method. For technical details not described in detail in this embodiment, reference may be made to the method provided by the embodiments of the present application.

[0099] The embodiments of the present application provide a non-volatile computer-readable storage medium storing computer-executable instructions, which when executed by one or more processors, such as Figure 6 one of the processors 610, enable the one or more processors to execute the real-time data stream processing method in any of the above method embodiments. For example, execute the Figure 2 method steps S1 to S5 described above, Figure 3 method steps S301 to S304 in Figure 4 and method steps S311 to S316 in Figure 4 to implement the functions of the modules 51-55 in

[0100] An embodiment of the present application provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by the electronic device, the electronic device can execute the real-time data stream processing method in any of the above method embodiments. For example, execute the Figure 2 method steps S1 to S5 in Figure 3 method steps S301 to S304 in Figure 4 method steps S311 to S316 in Figure 4 to implement the functions of modules 51-55 in

[0101] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0102] Through the description of the above embodiments, those of ordinary skill in the art can clearly understand that each embodiment can be implemented by means of software plus a general hardware platform, and of course also by hardware. Those of ordinary skill in the art can understand that all or part of the processes in the above method embodiments can be completed by instructing relevant hardware through a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above method embodiments. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM), etc.

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; under the idea of the present application, the technical features in the above embodiments or different embodiments can also be combined, and the steps can be implemented in any order, and there are many other changes in different aspects of the present application as described above. For the sake of brevity, they are not provided in detail; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A real-time data stream processing method, characterized in that: include: Distributing the received data stream to be processed to the deployed first distributed node through a preset load balancer; The first distributed node segments the received distribution data stream according to a preset segmentation rule and writes the segments into a message queue, and processes the segmented data in the message queue in parallel through independent threads in the multithreading; Dynamically store the intermediate state data in the shard data processing process through the memory database, generate snapshot data according to the shard data, and synchronize the snapshot data to the preset distributed storage system using the incremental snapshot mechanism; Collecting load data of the first distributed node in real time, and dynamically updating the data flow distribution strategy of the load balancer according to the load data through a trained resource usage prediction model; The processing result corresponding to the data stream to be processed is written into persistent storage through an asynchronous batch mechanism, and the processing result is returned through a preset real-time communication protocol.

2. The real-time data stream processing method according to claim 1, characterized in that: After the step of dynamically storing the intermediate state data in the shard data processing process through the memory database, the method further includes: Recording processing status data and intermediate result data of the slice data when being processed in the independent thread; Generating intermediate state data according to the processing state data and the intermediate result data, and storing the intermediate state data in the preset memory database; When the processing task corresponding to the shard data is restarted, or the first distributed node corresponding to the shard data resumes operation, the latest intermediate state data is obtained from the memory database; The execution progress of the shard data is restored according to the latest intermediate state data.

3. The real-time data stream processing method according to claim 1, characterized in that: The step of generating snapshot data according to the shard data and synchronizing the snapshot data to a preset distributed storage system using an incremental snapshot mechanism includes: When the preset snapshot generation event mechanism is triggered, or the current time reaches the preset time period switching point, it is determined whether the hash value of the current shard data processing state is the same as the hash value of the previous snapshot data; If the hash values ​​are different, the change log or incremental record of the memory database is obtained as incremental data, and snapshot data is generated according to the incremental data and the metadata of the current shard data; Perform consistent hash calculation according to the unique code of the current shard data, and map the corresponding calculation result to the target node of the distributed storage system; According to the generation timestamp of the snapshot data, hierarchically storing the snapshot data to the target node of the distributed storage system; If a faulty first distributed node is detected, obtaining the corresponding snapshot data from the target node of the distributed storage system according to the unique code of the shard data and the generation timestamp; The snapshot data is restored to a new first distributed node without any faults through the load balancer, and the data stream processing task is continued according to the data stream processing progress in the snapshot data.

4. The real-time data stream processing method according to claim 1, characterized in that: The step of collecting the load data of the first distributed node in real time and dynamically adjusting the data flow distribution strategy of the load balancer through the trained resource usage prediction model includes: Collecting in real time the processor usage rate, memory usage rate, network bandwidth occupancy rate, disk read / write rate and task processing delay time of the first distributed node as the load data; Normalizing the load data, and converting the normalized load data into time series data receivable by the resource usage prediction model, wherein the resource usage prediction model is constructed based on a long short-term memory network; Processing the time series data sequentially through the forget gate, input gate, memory unit, and output gate of the resource usage prediction model to obtain a hidden state of resource usage at each time step; Inputting the resource usage hidden state data of the last time step into the fully connected layer for processing to obtain the resource usage trend prediction result of the first distributed node; The data flow distribution strategy of the load balancer is updated according to the resource usage trend prediction result.

5. The real-time data stream processing method according to claim 1, characterized in that: After the step of processing the fragmented data in the message queue in parallel by independent threads in the multithreading, the method further includes: Using a monitoring tool to continuously monitor the response status and current performance data of each of the first distributed nodes; When it is detected that the first distributed node loses response or has abnormal performance, marking the first distributed node as a fault state; Redistributing the new data stream to be processed to the first distributed node in a non-faulty state according to the load data of the first distributed node in a non-faulty state; Acquire the first distributed node in a non-failure state as a transfer node; According to the intermediate state data and the snapshot data of the first distributed node in the fault state, the processing task of the corresponding shard data is resumed on the transfer node.

6. The real-time data stream processing method according to claim 1, characterized in that: After the step of processing the fragmented data in the message queue in parallel by independent threads in the multithreading, the method further includes: Matching the target data processing type corresponding to the data stream to be processed in a preset data type library, and generating different target data processing retry strategies according to the target data processing type; When the independent thread processes the shard data and triggers a preset exception capture mechanism, it is determined whether the number of retry executions of the target data processing retry strategy is greater than a preset retry number threshold; If it is not greater than the retry count threshold, the target data processing retry strategy is executed and the retry execution count is accumulated; If it is greater than the retry times threshold, a data stream processing alarm is sent to a preset alarm recipient.

7. The real-time data stream processing method according to claim 1, characterized in that: After the step of the first distributed node slicing the received distribution data stream according to the preset slicing rule and then writing the slicing data stream into the message queue, the method further includes: Monitoring the data volume and processing time of the sharded data; If the processing time of the shard data is greater than a preset first processing time threshold, reducing the preset value of the single shard data volume in the sharding rule according to the data volume; If the processing time of the shard data is less than a preset first processing time threshold, the preset value of the single shard data volume in the shard rule is increased according to the data volume.

8. A real-time data stream processing device, characterized in that: include: A data stream distribution module, used for distributing the received data stream to be processed to the deployed first distributed node through a preset load balancer; A data stream sharding module, used for sharding the received distribution data stream according to a preset sharding rule at the first distributed node and writing the sharded data into a message queue, and processing the sharded data in the message queue in parallel through independent threads in the multi-threading; A data processing snapshot module is used to dynamically store the intermediate state data in the shard data processing process through the memory database, generate snapshot data according to the shard data, and synchronize the snapshot data to a preset distributed storage system using an incremental snapshot mechanism; A distribution strategy optimization module, used for collecting load data of the first distributed node in real time, and dynamically updating the data flow distribution strategy of the load balancer according to the load data through a trained resource usage prediction model; The data processing result module is used to write the processing result corresponding to the data stream to be processed into persistent storage through an asynchronous batch mechanism, and return the processing result through a preset real-time communication protocol.

9. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the real-time data stream processing method according to any one of claims 1 to 7.

10. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are executed by an electronic device, the electronic device executes the real-time data stream processing method according to any one of claims 1 to 7.

Images