Real-time data processing method and system

Abstract

The invention discloses a real-time data processing method and system, and relates to the technical field of computers, and the method comprises the steps: S1, monitoring a network state between nodes based on a heartbeat detection mechanism, S2, collecting and storing real-time data generated by each node, S3, carrying out data synchronization verification when the network state is detected to be abnormal, and S4, carrying out data synchronization verification when the network state is abnormal. S4, updating the verified data to each node of the distributed system; according to the real-time data processing method and system, the network state of the nodes in the distributed system can be identified in real time, the abnormal nodes can be quickly positioned, the problem of data deviation caused by network delay or node disconnection is avoided, the problem of data inconsistency caused by receiving time deviation of different nodes is remarkably reduced, the accuracy of a data processing result is guaranteed, and the data processing efficiency is improved. The problem of data consistency caused by network instability among nodes in a distributed system in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a real-time data processing method and system. Background Art

[0002] In modern distributed systems, real-time data processing technology is widely used in finance, logistics, telecommunications and other fields. These systems work together through multiple nodes to achieve large-scale data processing and analysis. The core feature of distributed systems is to use multiple nodes to share data processing tasks to improve the system's processing efficiency and fault tolerance.

[0003] Existing distributed real-time data processing methods usually rely on event stream processing frameworks, which realize real-time data collection, transmission and calculation through efficient message queues and processing pipelines. However, in a distributed environment, unstable network communication between nodes is a common problem, leading to the following technical difficulties: due to network delays or node disconnection, there is a time deviation in the data received by different nodes, resulting in inconsistent data processing results. This situation is particularly serious in scenarios such as high-frequency trading and real-time monitoring that require high data consistency; when a node fails due to network problems, how to quickly restore data and re-participate in calculations is a difficulty in existing technologies. Most systems need to rely on complex playback mechanisms and data logging to try to restore consistency, which increases the complexity of the system; when dealing with data loss or duplication caused by network instability, existing methods usually require additional data verification or retransmission mechanisms, which affects the real-time performance of the system. Therefore, existing technologies have obvious deficiencies in solving data consistency problems caused by unstable node networks in distributed systems. These problems not only affect the accuracy of data processing, but also limit the further promotion and optimization of distributed real-time data processing technology in key application areas. Summary of the invention

[0004] The purpose of the present invention is to provide a real-time data processing method and system to solve the problem of data consistency caused by network instability between nodes in the prior art distributed system.

[0005] To achieve the above object, the present invention provides the following technical solution: a real-time data processing method, the method comprising: S1, monitor the network status between nodes based on the heartbeat detection mechanism; S2, collect and store real-time data generated by each node; S3. When an abnormal network status is detected, data synchronization verification is performed, including calculating the data verification priority value of each node that needs to be verified. The specific formula is: , , ; Among them, C represents the current data verification priority value of the node, S 0 represents the actual data state value of the current node, X represents the target data state value, t represents time, r represents the network environment suppression factor, σ represents the volatility of the node data state, and d 1 Indicates the degree to which the current node data state deviates from the target value, d 2 It represents the further correction value of the possibility of nodes achieving target consistency after considering the fluctuation of data status, N (d 1 ), N (d 2 ) is the cumulative normal distribution function, which indicates the probability of achieving data consistency in the current state; S4. Update the verified data to each node of the distributed system.

[0006] Preferably, the S1 comprises: The heartbeat detection data of each node is divided into two categories: normal state and abnormal state, and the optimal classification boundary is calculated. The specific formula is: ; Where w represents the classification boundary used to distinguish network status in node heartbeat data analysis, and Y w Indicates the internal distribution discreteness of the heartbeat data of normal and abnormal nodes, m 1 Represents the average value of the heartbeat signal data of all nodes in normal state, m 2 Indicates the average value of heartbeat signal data of all nodes in abnormal state; Based on the data distribution after calculating the optimal classification boundary, it is determined in real time whether the network status of the node is abnormal.

[0007] Preferably, S2 includes: The real-time data is divided into two types: first priority and second priority. The first priority data is stored first, and then the second priority data is stored. The dynamic changes of the first priority and second priority data are simulated to ensure that resources are allocated to the second priority data while the storage and growth of the first priority data are maintained. The specific formula is: , ; Among them, x represents the storage amount of first priority data, y represents the storage amount of second priority data, t represents time, α represents the growth coefficient of first priority data, β represents the competition coefficient of first priority data affected by second priority data, δ represents the growth coefficient of second priority data affected by first priority data, and γ represents the attenuation coefficient of second priority data.

[0008] Preferably, S4 includes: Model the updated process of the verified data, optimize the update path, dynamically adjust the propagation rate parameters, ensure that all nodes complete the update, and optimize the update path. The specific formula is: ; Among them, I represents the proportion of nodes that have successfully completed data update in the total network nodes, L represents the proportion of nodes that have not yet completed data update in the total network nodes, u represents the efficiency of nodes that have completed data update in propagating updates to nodes to be updated, v represents the rate at which nodes that have completed data update exit from the propagation task, and t represents time.

[0009] Preferably, S1 also includes sending a heartbeat request to each node based on a set heartbeat interval, receiving a heartbeat response returned by each node, and marking the corresponding node as a network abnormality when no response is received. If no response is received from a node within three consecutive heartbeat cycles, the node is considered to be failed and a backup node is started to take over.

[0010] Preferably, the actual data state value S of the current node in S3 0 The calculation formula is: ; Among them, S 0 Indicates the actual data status value of the current node, z 1 and z 2 represents the weighting coefficient, p loss represents the packet loss rate of the node within the specified time window, and b represents the network delay of the node.

[0011] Preferably, the calculation formula of the target data state value X in S3 is: ; Among them, X represents the target data state value, X base Represents the basic value of the global consistency requirement in the distributed system, k adjust represents the adjustment coefficient, p priority Indicates the priority of the node.

[0012] Preferably, the calculation formula of the competition coefficient β of the first priority data affected by the second priority data in S2 is: ; Where β represents the competition coefficient of the first priority data affected by the second priority data, x represents the storage amount of the first priority data, y represents the storage amount of the second priority data, and q 1 Represents the competition adjustment factor.

[0013] Preferably, the calculation formula of the growth coefficient δ of the second priority data in S2 affected by the first priority data is: ; Among them, δ represents the growth coefficient of the second priority data affected by the first priority data, x represents the storage capacity of the first priority data, y represents the storage capacity of the second priority data, and q 2 Indicates the gain adjustment factor.

[0014] A real-time data processing system, used to implement the steps of the real-time data processing method, the system comprising: The network status monitoring module is used to monitor the network status between nodes in the distributed system based on the heartbeat detection mechanism; Data collection and storage module, used to collect and store real-time data generated by each node; A data synchronization verification module connected to the network status monitoring module and the data collection and storage module, used to perform data synchronization verification when an abnormal network status is detected; The data update module connected to the data synchronization verification module is used to update the verified data to each node of the distributed system.

[0015] It can be seen from the above technical solution that the present invention has the following beneficial effects: The real-time data processing method and system monitor the network status between nodes based on the heartbeat detection mechanism, collect and store the real-time data generated by each node, perform data synchronization verification when an abnormal network status is detected, and update the verified data to each node of the distributed system. The method and system can identify the network status of nodes in the distributed system in real time, quickly locate abnormal nodes, avoid data deviation problems caused by network delays or node loss of connection, significantly reduce data inconsistency problems caused by reception time deviations of different nodes, ensure the accuracy of data processing results, realize automatic recovery and resynchronization of faulty node data, avoid reliance on complex playback mechanisms and data log records, greatly reduce complexity, improve response speed and repair capabilities for node failures, ensure the efficiency and stability of the distributed system, effectively improve the real-time performance of the system, adapt to scenarios with high real-time requirements, reduce the complexity of system implementation and operation and maintenance costs, and solve the problem of data consistency caused by network instability between nodes in the prior art in the distributed system. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flow chart of the method of the present invention; Figure 2 It is a schematic diagram of the connection of the system modules of the present invention. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] like Figure 1 As shown, the present invention provides a technical solution: a real-time data processing method, the method comprising: S1, monitor the network status between nodes based on the heartbeat detection mechanism; S2, collect and store real-time data generated by each node; S3. When an abnormal network status is detected, data synchronization verification is performed, including calculating the data verification priority value of each node that needs to be verified. The specific formula is: , , ; Among them, C represents the current data verification priority value of the node, S 0 represents the actual data state value of the current node, X represents the target data state value, t represents time, r represents the network environment suppression factor, σ represents the volatility of the node data state, and d 1 Indicates the degree to which the current node data state deviates from the target value, d 2 It represents the further correction value of the possibility of nodes achieving target consistency after considering the fluctuation of data status, N (d 1 ), N (d 2 ) is the cumulative normal distribution function, which indicates the probability of achieving data consistency in the current state; S4. Update the verified data to each node of the distributed system.

[0019] In the above implementation, the core of the real-time data processing method is to efficiently monitor and correct the network status and data consistency of the nodes of the distributed system. Step S1 obtains the network status information between each node in real time through the heartbeat detection mechanism to identify possible abnormal situations. Step S2 continuously collects and stores the real-time data of each node when the network is normal to provide basic data for subsequent verification. Step S3 performs data synchronization verification when a network anomaly is detected. The priority value of each node that needs to be verified is calculated by a formula, and the consistency probability of data recovery in the current environment is comprehensively considered based on the degree of deviation and volatility of the data status. The cumulative sub-distribution function N(d 1 ), N(d 2) provides a probability correction mechanism to ensure that high-priority data can still be processed first in a volatile environment, thereby optimizing the allocation of network resources and the efficiency of data consistency correction. Finally, step S4 distributes the corrected data to other nodes of the distributed system to ensure that the data of the entire system remains consistent and complete. The efficiency of data consistency verification of the distributed system in the case of network anomalies is improved. By calculating the verification priority value, important node data is processed first, and a mathematical model is used to analyze the data state deviation and volatility. It can dynamically adapt to complex network environments, improve the accuracy of data verification, perform probability correction, and significantly enhance the robustness of data consistency recovery. It reduces the waste of network resources and provides an efficient real-time data update mechanism to ensure rapid synchronization of data in a distributed system and improve the overall stability and reliability of the system.

[0020] S1 includes classifying the heartbeat detection data of each node into two categories: normal state and abnormal state, and calculating the optimal classification boundary. The specific formula is: ; Where w represents the classification boundary used to distinguish network status in node heartbeat data analysis, and Y w Indicates the internal distribution discreteness of the heartbeat data of normal and abnormal nodes, m 1 Represents the average value of the heartbeat signal data of all nodes in normal state, m 2 Indicates the average value of heartbeat signal data of all nodes in abnormal state; Based on the data distribution after calculating the optimal classification boundary, it is determined in real time whether the network status of the node is abnormal.

[0021] In the above implementation, the method realizes real-time monitoring of the network status by analyzing the heartbeat detection data of the nodes in the distributed system. First, the heartbeat detection data of the nodes are divided into two categories according to the normal state and the abnormal state. In order to effectively distinguish the two states, the method calculates the classification boundary w based on statistical principles. The calculation formula of the boundary fully considers the discrete degree Y of the distribution of the two types of data w and their respective mean differences m 1 −m 2 Specifically, the node heartbeat signal data in the normal state usually presents a stable distribution, and its average value m1 is different from the average value m in the abnormal state. 2There are significant differences; by calculating the optimal classification boundary w of the two, the normal and abnormal state nodes can be effectively divided. Finally, according to the distribution of heartbeat data and the results of the classification boundary w, it is judged in real time whether the network state of each node is abnormal. This process can quickly respond to changes in network status and provide a reliable basis for subsequent data synchronization and consistency verification. The accuracy of node network status anomaly detection is improved. The optimal classification boundary w can effectively distinguish between normal and abnormal state nodes, considering the discrete degree Y of normal and abnormal state data w and mean difference m 1 −m 2 , optimizes the calculation of classification boundaries, makes the detection process more robust, realizes real-time judgment of network status, provides a reliable basis for data verification and synchronization in distributed systems, improves the stability and response speed of system operation, and provides a network status detection method with low computational complexity, which is suitable for node status monitoring of large-scale distributed systems and reduces the demand for hardware and algorithm resources.

[0022] S2 includes dividing the real-time data into two types: first priority and second priority. The first priority data is stored first, and then the second priority data is stored. The dynamic changes of the first priority and second priority data are simulated to ensure that resources are allocated to the second priority data while the storage and growth of the first priority data are maintained. The specific formula is: , ; Among them, x represents the storage amount of first priority data, y represents the storage amount of second priority data, t represents time, α represents the growth coefficient of first priority data, β represents the competition coefficient of first priority data affected by second priority data, δ represents the growth coefficient of second priority data affected by first priority data, and γ represents the attenuation coefficient of second priority data.

[0023] In the above implementation, the storage of real-time data is efficiently managed through priority classification and dynamic change simulation. The first priority data represents the critical data in the system, and its storage priority is higher than the second priority data, so storage resources are allocated first. and , can accurately simulate the storage changes of two priority data. Equation Describes the dynamics of the first priority data: its growth is determined by the growth coefficient α, and is affected by the competition of the second priority data (described by the competition coefficient β). It describes the dynamic changes of the second priority data: its growth depends on the interaction with the first priority data (described by the growth coefficient δ), and is controlled by its own attenuation coefficient γ. This mechanism ensures that the storage growth of the first priority data will not be significantly affected by the competition of the second priority data through dynamic allocation of resources, while avoiding excessive attenuation of the second priority data, and achieves balance and optimization of data storage. Through priority classification and resource allocation, it ensures that the storage growth of the first priority data will not be significantly affected by resource competition. The dynamic change equation is used to balance the storage requirements of the first and second priority data, avoiding excessive attenuation of low priority data. The interaction and dynamic changes of priority data are accurately described through mathematical modeling, optimizing the allocation and use efficiency of storage resources. This method is suitable for the storage management of multi-type data in distributed systems, especially in environments with limited storage resources or complex data types, and can significantly improve the storage performance and stability of the system.

[0024] S4 includes modeling the updated process of verified data, optimizing the update path, dynamically adjusting the propagation rate parameters, and ensuring that all nodes complete the update. The specific formula for optimizing the update path is: ; Among them, I represents the proportion of nodes that have successfully completed data update in the total network nodes, L represents the proportion of nodes that have not yet completed data update in the total network nodes, u represents the efficiency of nodes that have completed data update in propagating updates to nodes to be updated, v represents the rate at which nodes that have completed data update exit from the propagation task, and t represents time.

[0025] In the above implementation, the present invention realizes efficient update of verified data in a distributed system by dynamically modeling and optimizing the propagation path. Describes the key dynamic relationships in the update process: The proportion of nodes I that have completed data updates increases with time t, and its growth rate is proportional to the proportion of unupdated nodes L and the propagation efficiency u. The propagation efficiency u reflects the collaborative ability of the updated nodes and the network bandwidth conditions during the propagation update process.

[0026] The exit rate vI of updated nodes describes the process of releasing resources from update propagation tasks after the nodes complete the update tasks. Node exit can release system resources, avoid resource waste, and improve the overall update efficiency.

[0027] Through the above dynamic modeling and path optimization, the system is able to adjust the propagation rate parameters u and v according to the real-time network status to balance the update speed and resource usage, ensuring that all nodes eventually complete the data update.

[0028] S1 also includes sending heartbeat requests to each node based on the set heartbeat interval, receiving heartbeat responses returned by each node, and marking the corresponding node as a network abnormality when no response is received. If no response is received from a node within three consecutive heartbeat cycles, the node is considered to be failed and the backup node is started to take over.

[0029] In the above implementation, the method realizes real-time monitoring of the network status and operation status of nodes in a distributed system by setting a heartbeat detection mechanism. Specifically: Heartbeat detection: Send a heartbeat request to each node at a preset time interval and wait for a response. This step is intended to confirm the operation status of the node through regular interaction. Abnormal marking: If no response is received from the node in a heartbeat cycle, the node is immediately marked as a network abnormality and the abnormal state is recorded for further processing. Failure determination: When a node fails to return a response for three consecutive heartbeat cycles, the node is judged to be a failed node, which may be caused by network interruption, hardware failure or system crash. Standby node takeover: After determining that the node is failed, immediately start the pre-configured standby node to take over the task of the failed node to ensure the continuity and normal operation of the system. The standby node can take over the function of the failed node through a preset mapping relationship or a dynamic allocation mechanism to avoid affecting the overall performance of the system due to a single point failure. Through the above process, the distributed system can quickly respond to abnormal changes in node status and achieve efficient node monitoring and fault recovery.

[0030] The actual data status value S of the current node in S3 0 The calculation formula is: ; Among them, S 0 Indicates the actual data status value of the current node, z 1 and z 2 represents the weighting coefficient, p loss represents the packet loss rate of the node within the specified time window, and b represents the network delay of the node.

[0031] In the above implementation, the actual data state value S of the current node 0 The calculation is based on the packet loss rate p loss and network delay b, combined with the weighting coefficient z 1 and z 2 Conduct a comprehensive evaluation. 1 (1−p loss ) indicates that the node data status is affected by the packet loss rate. The packet loss rate p loss The lower the actual data status value S of the node 0 The higher the value, the more reliable the data transmission quality. 2The (1 / b) part indicates that the node data status is affected by network delay. The lower the network delay b, the faster the node responds and the higher the data status value S 0 The higher the value, the more real-time and efficient the node is. 1 and z 2 The adjustment can flexibly adjust the loss rate and network delay to the data status value S according to the specific application scenario. 0 The relative influence weight of nodes can be adjusted to adapt to different network environments and performance requirements. This formula realizes the quantitative evaluation of node data status, comprehensively considers the two key indicators of data loss and delay, and can accurately reflect the status of nodes in the distributed system, providing a basis for subsequent data synchronization and verification.

[0032] The calculation formula of the target data status value X in S3 is: ; Among them, X represents the target data state value, X base Represents the basic value of the global consistency requirement in the distributed system, k adjust represents the adjustment coefficient, p priority Indicates the priority of the node.

[0033] In the above implementation, the method realizes personalized adjustment and optimization of the target state of nodes in the distributed system through dynamic calculation of the target data state value X. Formula The specific functions are as follows: Base Value X base It represents the global consistency requirement of the system and is the basic reference value for calculating the target state of all nodes, ensuring the overall target consistency of the system. adjust and node priority p priority The target state value X is dynamically adjusted by the combined effect of the priority. The higher the priority of the node, the higher the target state value will be based on the basic value X. base Improve more to prioritize the consistency requirements of key nodes. Adjustment coefficient k adjust It can be dynamically adjusted according to the network environment, node characteristics or task importance, so as to achieve flexible adaptation to different application scenarios. Through the above formula, the system can effectively balance the global consistency and node individualization requirements, give priority to meeting the target state requirements of key nodes, and take into account the optimization of overall system performance. By introducing a priority adjustment mechanism, the target state requirements of high-priority nodes are given priority, the data consistency of key nodes is guaranteed, and the adjustment coefficient k is adjusted. adjust The dynamic adjustability enables the system to flexibly adapt to different network conditions and task requirements, improving the adaptability of distributed systems. base, taking into account the personalized needs of nodes, balancing the contradiction between global performance and local optimization, node priority p priority It can be defined according to specific application scenarios, such as dynamic allocation based on the node's computing power, storage capacity or task importance, to achieve diversified priority strategy support. The calculation formula for the target state value is simple and efficient, suitable for the real-time computing needs of large-scale distributed systems, and reduces the consumption of computing resources.

[0034] The calculation formula of the competition coefficient β of the first priority data in S2 affected by the second priority data is: ; Where β represents the competition coefficient of the first priority data affected by the second priority data, x represents the storage amount of the first priority data, y represents the storage amount of the second priority data, and q 1 Represents the competition adjustment factor.

[0035] In the above implementation, the competition coefficient β is used to quantify the competition relationship between the first priority data and the second priority data in the storage resource allocation, and its calculation formula is: ; In the formula, x / (x+y) describes the proportion of the first-priority data in the total storage capacity. When the storage capacity x of the first-priority data increases, its proportion increases accordingly, and the competition coefficient β increases, indicating that the first-priority data has an advantage in resource competition. Adjustment factor q 1 The weight used to dynamically adjust the competition coefficient can be set according to the priority of the system to the first priority data. For example, when the first priority data has a greater impact on system performance, a higher q can be set. 1 value to further increase the weight of β. By calculating β in real time, the system can dynamically adjust the storage resource allocation strategy for the first-priority and second-priority data, giving priority to the growth of the first-priority data while ensuring the reasonable allocation of the second-priority data. This calculation formula realizes the accurate modeling of the competition relationship of priority data, and can dynamically adjust the competition coefficient according to different scenarios to optimize resource allocation efficiency. By calculating the competition coefficient β in real time, the resource allocation of the first and second priority data can be dynamically adjusted to improve the overall storage efficiency. In the competition coefficient formula, x / (x+y) ensures the advantage of the first-priority data in the storage resource allocation, and is combined with the adjustment factor q 1 Optimize resource usage of key data by adjusting q 1 The value of can flexibly adapt to different application scenarios (such as tasks with high real-time requirements), improve the applicability of the method, and the dynamic adjustment of the competition coefficient can balance the storage requirements of the first priority and second priority data, avoiding extreme resource allocation. The formula structure is simple and the calculation efficiency is high, which is suitable for real-time competition relationship modeling and evaluation in large-scale distributed systems.

[0036] The calculation formula for the growth coefficient δ of the second priority data in S2 affected by the first priority data is: ; Among them, δ represents the growth coefficient of the second priority data affected by the first priority data, x represents the storage capacity of the first priority data, y represents the storage capacity of the second priority data, and q 2 Indicates the gain adjustment factor.

[0037] In the above implementation, the growth coefficient δ describes the degree to which the second priority data is affected by the first priority data during the storage growth process, and its calculation formula is: ; The y / (x+y) in the formula reflects the proportion of the second priority data in the total storage capacity. When the storage capacity y of the second priority data is relatively large, its growth coefficient δ increases, indicating that the storage growth of the second priority data is more supported in resource allocation. Gain adjustment factor q 2 Used to adjust the growth weight of the second priority data. By adjusting q 2 The value of can control the growth rate of the second priority data according to actual needs. For example, when the storage capacity of the second priority data needs to be increased first, q can be increased. 2 The formula combines the storage ratio of first-priority data and second-priority data to dynamically balance the growth relationship between the two and ensure the reasonable allocation of resources. This formula provides a dynamic and flexible way to evaluate and regulate the growth behavior of second-priority data while taking into account the competitive relationship with first-priority data. Through the dynamic calculation of the growth coefficient δ, the reasonable allocation and balanced growth of storage resources are achieved between the first and second priority data. The gain adjustment factor q 2 The introduction of enables the system to flexibly adjust the growth rate of the second priority data according to application requirements. By calculating δ in real time, the growth weight of the second priority data can be dynamically adjusted to effectively utilize system resources and avoid resource waste. The proportional term y / (x+y) in the formula can adapt to different distributed storage environments. Dynamic adjustment of priority data can be achieved without complex calculations. In the case of limited storage resources, the excessive growth of low-priority data on high-priority data can be avoided by balancing the growth coefficient, thereby ensuring the overall stability and consistency of the system.

[0038] A real-time data processing system is also provided for implementing the steps of the real-time data processing method, the system comprising: The network status monitoring module is used to monitor the network status between nodes in the distributed system based on the heartbeat detection mechanism; Data collection and storage module, used to collect and store real-time data generated by each node; A data synchronization verification module connected to the network status monitoring module and the data collection and storage module, used to perform data synchronization verification when an abnormal network status is detected; The data update module connected to the data synchronization verification module is used to update the verified data to each node of the distributed system.

[0039] In the above implementation, the real-time data processing system consists of multiple modules working together to ensure the consistency and reliability of real-time data in the distributed system. Figure 2 , the specific functions are as follows: Network status monitoring module: Based on the heartbeat detection mechanism, it regularly sends heartbeat requests to each node of the distributed system and receives responses to monitor the network status between nodes. When a heartbeat response from a node is not received, it is marked as a network anomaly and an abnormal status signal is sent to the data synchronization verification module.

[0040] Data collection and storage module: continuously collects real-time data generated by each node in the distributed system, and stores the data by category (such as first priority and second priority), providing a reliable data foundation for data verification and update.

[0041] Data synchronization verification module: When a network anomaly is detected, data synchronization verification is performed. According to the data verification priority, target state value, and competition relationship, the nodes that need to be synchronized and their priorities are calculated, and the data is checked and corrected for consistency. The verification process uses the formula calculation method described in the claims (such as data priority competition coefficient, target state value, etc.) to ensure the accuracy and real-time performance of data verification.

[0042] Data update module: Receives verified data and distributes it to other nodes in the distributed system according to the optimized update path. By dynamically adjusting the propagation rate parameters (such as propagation efficiency and exit rate), the speed of data update and resource allocation are optimized to ensure the efficiency and consistency of the update process.

[0043] Through the network status monitoring module and the data collection and storage module, efficient monitoring and storage management of the network status and real-time node data are achieved. The data synchronization verification module, combined with the mathematical model and priority calculation method in the claims, can quickly discover and repair data inconsistency problems. The data update module improves the efficiency and accuracy of data updates through dynamic modeling and path optimization technology. It is suitable for large-scale distributed systems. The modules work together to effectively deal with network anomalies and node failures, and ensure the continuous operation and data consistency of the distributed system. The modular design of the system makes it have good scalability, and it can add new functional modules or adjust existing modules according to needs to adapt to different application scenarios.

[0044] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A real-time data processing method, characterized in that: The method comprises: S1, monitor the network status between nodes based on the heartbeat detection mechanism; S2, collect and store real-time data generated by each node; S3. When an abnormal network status is detected, data synchronization verification is performed, including calculating the data verification priority value of each node that needs to be verified. The specific formula is: ， ， ； Where C represents the current data verification priority value of the node, S0 represents the actual data status value of the current node, X represents the target data status value, t represents time, r represents the network environment suppression factor, σ represents the volatility of the node data status, d1 represents the degree of deviation of the current node data status from the target value, d2 represents the further correction value of the possibility of the node achieving target consistency after considering the data status fluctuation, N(d1) and N(d2) are cumulative normal distribution functions, which represent the probability of achieving data consistency in the current state; S4. Update the verified data to each node of the distributed system.

2. A real-time data processing method according to claim 1, characterized in that: The S1 includes: The heartbeat detection data of each node is divided into two categories: normal state and abnormal state, and the optimal classification boundary is calculated. The specific formula is: ; Where w represents the classification boundary used to distinguish network status in node heartbeat data analysis, and Y w Indicates the internal distribution discreteness of the heartbeat data of nodes in normal and abnormal states, m1 represents the average value of the heartbeat signal data of all nodes in normal state, and m2 represents the average value of the heartbeat signal data of all nodes in abnormal state; Based on the data distribution after calculating the optimal classification boundary, it is determined in real time whether the network status of the node is abnormal.

3. A real-time data processing method according to claim 1, characterized in that: The S2 includes: The real-time data is divided into two types: first priority and second priority. The first priority data is stored first, and then the second priority data is stored. The dynamic changes of the first priority and second priority data are simulated to ensure that resources are allocated to the second priority data while the storage and growth of the first priority data are maintained. The specific formula is: ， ； Among them, x represents the storage amount of first priority data, y represents the storage amount of second priority data, t represents time, α represents the growth coefficient of first priority data, β represents the competition coefficient of first priority data affected by second priority data, δ represents the growth coefficient of second priority data affected by first priority data, and γ represents the attenuation coefficient of second priority data.

4. A real-time data processing method according to claim 1, characterized in that: The S4 includes: Model the updated process of the verified data, optimize the update path, dynamically adjust the propagation rate parameters, ensure that all nodes complete the update, and optimize the update path. The specific formula is: ; Among them, I represents the proportion of nodes that have successfully completed data update in the total network nodes, L represents the proportion of nodes that have not yet completed data update in the total network nodes, u represents the efficiency of nodes that have completed data update in propagating updates to nodes to be updated, v represents the rate at which nodes that have completed data update exit from the propagation task, and t represents time.

5. A real-time data processing method according to claim 1, characterized in that: The S1 also includes sending a heartbeat request to each node based on the set heartbeat interval, receiving a heartbeat response returned by each node, and marking the corresponding node as a network abnormality when no response is received. If no response is received from a node within three consecutive heartbeat cycles, the node is considered to be failed and a backup node is started to take over.

6. A real-time data processing method according to claim 1, characterized in that: The calculation formula of the actual data state value S0 of the current node in S3 is: ； Among them, S0 represents the actual data state value of the current node, z1 and z2 represent weighting coefficients, and p loss represents the packet loss rate of the node within the specified time window, and b represents the network delay of the node.

7. A real-time data processing method according to claim 1, characterized in that: The calculation formula of the target data state value X in S3 is: ； Among them, X represents the target data state value, X base Represents the basic value of the global consistency requirement in the distributed system, k adjust represents the adjustment coefficient, p priority Indicates the priority of the node.

8. A real-time data processing method according to claim 3, characterized in that: The calculation formula of the competition coefficient β of the first priority data affected by the second priority data in S2 is: ； Wherein, β represents the competition coefficient of the first priority data affected by the second priority data, x represents the storage amount of the first priority data, y represents the storage amount of the second priority data, and q1 represents the competition adjustment factor.

9. A real-time data processing method according to claim 3, characterized in that: The calculation formula of the growth coefficient δ of the second priority data in S2 affected by the first priority data is: ； Among them, δ represents the growth coefficient of the second priority data affected by the first priority data, x represents the storage amount of the first priority data, y represents the storage amount of the second priority data, and q2 represents the gain adjustment factor.

10. A real-time data processing system, used to implement the steps of the real-time data processing method according to any one of claims 1 to 9, characterized in that: The system comprises: The network status monitoring module is used to monitor the network status between nodes in the distributed system based on the heartbeat detection mechanism; Data collection and storage module, used to collect and store real-time data generated by each node; A data synchronization verification module connected to the network status monitoring module and the data collection and storage module, used to perform data synchronization verification when an abnormal network status is detected; The data update module connected to the data synchronization verification module is used to update the verified data to each node of the distributed system.

Images