Distributed acquisition data consistency processing method
By combining the Raft algorithm, vector clock, and PBFT algorithm, and processing data according to priority, the data consistency problem of distributed data acquisition in complex communication networks is solved, and data consistency guarantee and resource optimization are achieved in different scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-28
- Publication Date
- 2026-04-21
AI Technical Summary
Distributed data acquisition in complex communication networks faces data consistency issues, especially under conditions of network latency, node failure, clock asynchrony, and malicious interference, where data timing disorder and inconsistency of replicas are difficult to resolve.
A three-layer protection system is adopted, combining the Raft algorithm, vector clock, and PBFT algorithm. Data is processed in a hierarchical manner according to priority. High-priority data uses the PBFT algorithm, medium-priority data uses asynchronous replication and periodic compensation strategies, and low-priority data uses master-slave replication consistency and causal consistency processing to ensure data consistency.
In a strong consistency scenario, the latency against Byzantine attacks is ≤100ms, the device status data fault recovery time is shortened to 2 minutes, the monitoring response speed is improved by 30%, and the non-critical data resource consumption is reduced by 40%, thus achieving distributed data consistency assurance.
Smart Images
Figure CN121907718A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data acquisition and network management technology, and specifically to a method for data consistency processing in distributed acquisition. Background Technology
[0002] The operation and maintenance of complex communication networks heavily rely on the data collection and monitoring analysis of network management systems. Currently, communication networks are becoming increasingly complex and their management scale is expanding, necessitating the distribution of data collection efforts.
[0003] This complex, distributed node acquisition method presents serious data consistency problems for data acquisition technology. Faced with the challenge of acquiring massive amounts of equipment data, the conventional approach is a distributed technology solution. However, data acquisition suffers from network latency, node failures, clock asynchrony, malicious interference, and other factors that lead to data timing errors and inconsistent replicas. Summary of the Invention
[0004] This invention provides a data consistency processing method for distributed data acquisition to fully ensure distributed data consistency.
[0005] In a first aspect of the present invention, a distributed data consistency processing method is provided, comprising:
[0006] Obtain the target data;
[0007] Calculate the priority of the target data to obtain high-priority data, medium-priority data, and low-priority data;
[0008] For high-priority data, a practical Byzantine fault-tolerant algorithm is used for data processing to obtain the first consistent data.
[0009] For medium-priority data, an asynchronous replication and periodic compensation strategy is used for data processing to obtain second-consistent data.
[0010] For low-priority data, master-slave replication consistency and causal consistency are used to obtain third-party consistent data.
[0011] The first consistent data, the second consistent data, and the third consistent data are aggregated to obtain consistent data.
[0012] In some optional embodiments of the present invention, the priority of the target data is calculated to obtain high-priority data, medium-priority data, and low-priority data, including:
[0013] Data priority is calculated based on a three-dimensional priority quantization model;
[0014] Based on the priority results, the data is divided into three categories: high-priority data, medium-priority data, and low-priority data.
[0015] In some optional embodiments of the present invention, the calculation formula for the three-dimensional priority quantization model is as follows:
[0016] P(D)=αR+βS+γI
[0017] α+β+γ=1
[0018] Wherein, P(D): represents the priority value of data item D; R: represents the real-time score; real-time is a scoring indicator that measures the sensitivity of data update response time; S: represents the security score; security measures the risk level of data leakage or tampering to the business; I: represents the business impact score; the degree of business impact refers to the degree of impact of data inconsistency on business processes; α, β, γ: represent the weight coefficients of different dimensions; the dimension weight coefficients measure the contribution of the three dimension indicators to the entire function, and the sum of the three coefficients must be 1.
[0019] In some optional embodiments of the present invention, a practical Byzantine fault-tolerant algorithm is used to process high-priority data to obtain first consistent data, including:
[0020] For high-priority data, a multi-stage consensus mechanism is used, which involves broadcasting from the master node and verifying from backup nodes, to ensure that the system remains consistent even when malicious nodes are present, thus obtaining the first consistent data.
[0021] In some optional embodiments of the present invention, an asynchronous replication and periodic compensation strategy is used to process medium-priority data to obtain second consistent data, including:
[0022] For medium-priority data, it is aggregated to the service node server in the form of a message queue, while peak data is allowed to arrive at the target node with a delay through the queue.
[0023] A periodic compensation strategy corrects data inconsistencies that extend far beyond the window period, resulting in second-consistent data.
[0024] In some optional embodiments of the present invention, low-priority data is processed using master-slave replication consistency and causal consistency to obtain third-party consistent data, including:
[0025] For low-priority data, the Raft algorithm is used to replicate logs between master and slave nodes; and the causal relationship of related events is recorded through a logical clock array to ensure eventual consistency and obtain second-consistent data.
[0026] In a second aspect of the present invention, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the distributed acquisition data consistency processing method according to any one of the embodiments of the first aspect.
[0027] In a third aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the distributed acquisition data consistency processing method described in any one of the embodiments of the first aspect.
[0028] The beneficial effects provided by this invention are:
[0029] I. Fully guarantee distributed data consistency
[0030] Strong consistency scenario: Byzantine attack resistance is achieved through the PBFT algorithm. In a cluster with 5 nodes (tolerating 1 malicious node), the consensus latency is ≤100ms, ensuring the accuracy of control command transmission. Eventual consistency scenario: Device status data is synchronized between master and slave using the Raft algorithm. Node failure recovery time is ≤2 minutes, which is ≥20% faster than master-slave replication. Vector clock technology ensures the correct order of configuration changes and status feedback, reducing the occurrence rate of causal disorder to below 1.2%.
[0031] II. Optimize the allocation of computing resources
[0032] Critical business data: Dedicated communication channels and computing resources are allocated, and the latency is ≤50ms based on strong consistency processing; Network management and monitoring data: A causal consistency eventual consistency model is adopted, which improves the monitoring response speed by about 30% and reduces the fault detection time from 10 minutes to less than 3 minutes; Non-critical data (such as device logs): Through asynchronous replication and periodic compensation, the system resource consumption is reduced by about 40%.
[0033] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description
[0034] Figure 1 A flowchart illustrating the consistency processing provided in an embodiment of the present invention. Detailed Implementation
[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.
[0036] This invention proposes a distributed data consistency processing method. For complex scenarios involving diverse and heterogeneous networks, this method combines Raft, vector clocks, and PBFT (Practical Byzantine Fault Tolerance) algorithms as needed to construct a differentiated "three-layer protection system." Among core nodes, Raft is used to ensure high data consistency in the face of network fluctuations and node failures. For non-urgent tasks such as changes to basic distributed system configurations and routine network status data monitoring, which are essential for maintaining system functionality, vector clock technology is applied to reduce performance overhead within a certain time window. For critical data such as configuration commands that are vulnerable to malicious attacks, PBFT is enabled. Through master node broadcasting and backup node verification mechanisms, it tolerates ≤(N-1) / 3 Byzantine nodes, resisting data tampering and interference, and ensuring the consistency and security of core services under extreme scenarios. These three types of algorithms precisely match scenario requirements, forming a comprehensive data protection system that is fault-tolerant, efficient, and attack-resistant.
[0037] I. Distributed Data Consistency Strategy
[0038] Distributed consistency is crucial for ensuring reliable data transmission and stable system operation. It is generally classified into three types according to consistency strength: strong consistency, weak consistency, and eventual consistency.
[0039] Strong consistency: requires that after data is updated, all nodes immediately obtain the latest data, and all read requests from all nodes return consistent data.
[0040] In this embodiment of the invention, strong data consistency is achieved based on PBFT (Practical Byzantine Fault Tolerance). Through multi-stage consensus involving master node broadcasting and backup node verification, the system can maintain consistency even when malicious nodes (Byzantine nodes) are present.
[0041] Weak consistency: The system allows for a window of inconsistency after data updates, and does not guarantee that all nodes will immediately obtain the latest data for read requests. The system allows some nodes to access old data and tolerates long-term data inconsistency.
[0042] In this embodiment of the invention, data from the acquisition terminal is aggregated to the service node server via a message queue, allowing peak data to arrive at the target node with a delay due to the queue. To compensate for potential long-term data inconsistencies between nodes, a compensation mechanism is designed based on weak consistency to correct data inconsistencies that far exceed the window period, thereby improving overall processing performance.
[0043] Eventual consistency: After data is updated, there is still a window of inconsistency in the system, which eventually reaches global consistency after a period of time. Eventual consistency requires treating the inconsistency "window" as a temporary state and actively eliminating inconsistencies through algorithm design.
[0044] In this embodiment of the invention, two algorithms are selected to achieve eventual consistency. One is the master-slave replication consistency implemented by the Raft algorithm, which achieves eventual consistency through the replication of logs between master and slave nodes; the other is the vector clock consistency algorithm, which records the causal relationship of related events through a logical clock array to ensure causal consistency in eventual consistency.
[0045] II. Priority-based strategy
[0046] Prioritizing data acquisition tasks based on different acquisition types and business requirements is a crucial measure to ensure data acquisition quality and system performance. In communication systems, different types of data have varying degrees of importance for system operation and decision-making; therefore, different priorities need to be set according to their importance and real-time requirements.
[0047] Based on consistency classification and priority grading, corresponding consistency processing methods are further adopted to achieve data consistency processing for distributed network management data collection.
[0048] III. Implementation of Priority Quantification Model
[0049] The calculation formula for the three-dimensional priority quantization model is as follows:
[0050] P(D)=αR+βS+γI
[0051] a+β+γ=1
[0052] The meaning of the formula is as follows:
[0053] P(D): Represents the priority value of data item D (value range: 0-100, the larger the value, the higher the priority);
[0054] R: Represents the real-time score (0-100, such as R=90 for communication data); real-time is a scoring indicator that measures the sensitivity of data update response time.
[0055] S: Represents the security score (0-100, e.g., S=100); security measures the risk level of data breach or tampering (including data corruption) to the business.
[0056] I: Represents the business impact score (0-100, such as I=80 for monitoring fault alarms); the degree of business impact refers to the degree of impact of data inconsistency on business processes.
[0057] α, β, γ: represent different dimension weight coefficients (based on the preset values of the scenario, such as α = 0.5, β = 0.3, γ = 0.2 for the task scenario); the dimension weight coefficients measure the contribution of the three dimension indicators to the entire function, and the sum of the three coefficients must be 1.
[0058] IV. Implementation of Consistency Verification Module
[0059] like Figure 1 As shown, based on the "three-dimensional priority quantization model" to calculate data priority, the logic driving the consistency strategy selection is as follows:
[0060] P(D)≥80: Strong consistency (PBFT algorithm, n=3f+1);
[0061] 50≤P(D)<80: Eventual consistency (Raft algorithm, master-slave replication);
[0062] P(D) < 50: Weak consistency (asynchronous replication, periodic compensation).
[0063] Based on priority P(D), the data is divided into three categories, and different consistency models and technical solutions are mapped to match them:
[0064] High priority: For critical business data (such as operational command and control orders), select a strong consistency model and adopt Byzantine fault-tolerant consistency technology based on the PBFT algorithm.
[0065] Medium priority: For monitoring data (such as the status of communication equipment), select the eventual consistency model and adopt master-slave replication consistency technology based on the Raft algorithm and vector clock consistency technology.
[0066] Low priority: For non-critical data (such as system logs), choose a weak consistency model and use asynchronous replication and periodic compensation techniques.
[0067] It should be understood that in the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples" refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this description, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate different embodiments or examples described in this specification, as well as some features of different embodiments or examples.
[0068] Of course, those skilled in the art can make various corresponding changes and modifications based on the present invention without departing from its spirit and essence, but such changes and modifications should all fall within the protection scope of the claims of the present invention.
Claims
1. A method for processing data consistency in distributed data acquisition, characterized in that, include: Obtain the target data; Calculate the priority of the target data to obtain high-priority data, medium-priority data, and low-priority data; For high-priority data, a practical Byzantine fault-tolerant algorithm is used for data processing to obtain the first consistent data. For medium-priority data, an asynchronous replication and periodic compensation strategy is used for data processing to obtain second-consistent data. For low-priority data, master-slave replication consistency and causal consistency are used to obtain third-party consistent data. The first consistent data, the second consistent data, and the third consistent data are aggregated to obtain consistent data.
2. The distributed data consistency processing method according to claim 1, characterized in that, Calculate the priority of the target data to obtain high-priority data, medium-priority data, and low-priority data, including: Data priority is calculated based on a three-dimensional priority quantization model; Based on the priority results, the data is divided into three categories: high-priority data, medium-priority data, and low-priority data.
3. The distributed data consistency processing method according to claim 2, characterized in that, The calculation formula for the three-dimensional priority quantization model is as follows: P(D)=αR+βS+γI α+β+γ=1 Wherein, P(D): represents the priority value of data item D; R: represents the real-time score; real-time is a scoring indicator that measures the sensitivity of data update response time; S: represents the security score; security measures the risk level of data leakage or tampering to the business; I: represents the business impact score; the degree of business impact refers to the degree of impact of data inconsistency on business processes; α, β, γ: represent different dimension weight coefficients: the dimension weight coefficients measure the contribution of the three dimension indicators to the entire function, and the sum of the three coefficients must be 1.
4. The distributed data consistency processing method according to claim 1, characterized in that, For high-priority data, a practical Byzantine fault-tolerant algorithm is used for data processing to obtain the first consistent data, including: For high-priority data, a multi-stage consensus mechanism is used, which involves broadcasting from the master node and verifying from backup nodes, to ensure that the system remains consistent even when malicious nodes are present, thus obtaining the first consistent data.
5. The distributed data consistency processing method according to claim 1, characterized in that, For medium-priority data, an asynchronous replication and periodic compensation strategy is used for data processing to obtain second-consistency data, including: For medium-priority data, it is aggregated to the service node server in the form of a message queue, while peak data is allowed to arrive at the target node with a delay through the queue. A periodic compensation strategy corrects data inconsistencies that extend far beyond the window period, resulting in second-consistent data.
6. The distributed data consistency processing method according to claim 1, characterized in that, For low-priority data, master-slave replication consistency and causal consistency are used to obtain third-party consistent data, including: For low-priority data, the Raft algorithm is used to replicate logs between master and slave nodes; and the causal relationship of related events is recorded through a logical clock array to ensure eventual consistency and obtain second-consistent data.
7. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the distributed acquisition data consistency processing method according to any one of claims 1-6.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the distributed acquisition data consistency processing method according to any one of claims 1-6.