Enterprise data updating management system based on cloud computing

Abstract

The invention discloses an enterprise data updating management system based on cloud computing, relates to the technical field of data management, and solves the technical problems that a user request needs to go back and forth to a cloud across a core network, and meanwhile, a full-amount synchronization mechanism continuously occupies the bandwidth of a backbone network to form a dual-performance bottleneck. High-frequency update data are deployed at edge nodes close to a user, so that the round-trip time of the cloud is reduced, and the user experience is improved; according to the method, high-frequency data is prevented from being frequently transmitted across a data center, the bandwidth pressure of a core network is reduced, global verification is carried out on all logic fragments in combination with a Raft + PBFT hybrid consensus mechanism, and the updating consistency and safety of node data are guaranteed. The topological graph is adjusted in real time along with service changes to adapt to data model evolution; the priority is comprehensively calculated in combination with the correlation degree, the data importance and the node load, it is ensured that the node with the high update priority quickly obtains the update request, and low-delay, high-availability and strong-consistency update propagation is achieved.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of data management, and particularly relates to an enterprise data update management system based on cloud computing. BACKGROUND

[0002] In the digital transformation tide, the enterprise data scale grows exponentially, and the data update frequency continues to increase. The traditional centralized architecture has been difficult to meet the needs of high-frequency update scenarios due to high latency, bandwidth overload, and lack of consistency guarantee. The enterprise data update management system based on cloud computing systematically solves these pain points through distributed architecture, edge computing, intelligent routing, and hybrid consensus mechanism, and becomes the key infrastructure supporting real-time decision-making and business innovation of enterprises.

[0003] If the prior art relies on the central cloud to centrally process all update data, the user request needs to cross the core network and return to the cloud, and the full-synchronization mechanism will continuously occupy the backbone network bandwidth, forming a double performance bottleneck. SUMMARY

[0004] The present application aims to at least solve one of the technical problems existing in the prior art; for this purpose, the present application proposes an enterprise data update management system based on cloud computing, which is used to solve the technical problem that the user request needs to cross the core network and return to the cloud, and the full-synchronization mechanism will continuously occupy the backbone network bandwidth, forming a double performance bottleneck.

[0005] To solve the above problems, the first aspect of the present application provides an enterprise data update management system based on cloud computing, comprising: Edge management layer: According to the region code of the enterprise data, the enterprise data is divided into a plurality of logical shards, and an edge data center is set in the region corresponding to the region code of each logical shard, and the logical shard data in the edge data center is managed by a plurality of master nodes and auxiliary nodes; Divide the logical shard data into high-frequency update data and ordinary update data, and deploy the high-frequency update data and the ordinary update data near the user's node in the edge data center; A Raft+PBFT hybrid consensus mechanism is used to globally verify all logical shards, monitor the running state of the master nodes of the edge data center, perform dynamic master node switching, and trigger automatic switching of the auxiliary nodes through heartbeat detection and blockchain log rollback; Cloud update request management layer: A change data capture module is deployed in the cloud computing environment, real-time monitoring of the database logs of each logical shard is performed, a data change event stream of each logical shard is generated, the update correlation degree between each node is analyzed, and a node update correlation degree topology graph is constructed; According to the data change event stream and the inter-node update correlation degree topology, the update priority of each node in the edge data center is calculated, the update request is routed to the nearest high update priority node using Anycast IP, and after the high update priority nodes synchronize the update request with each other, the update request is synchronized to other nodes.

[0006] Optionally, in one example of the above aspect, according to the region code of enterprise data, the enterprise data is divided into several logical shards, and in each logical shard region code corresponds to a regional edge data center, and the logical shard data in the edge data center is managed by several master nodes and secondary nodes, including the following steps: According to the region code of enterprise data, set the shard strategy selection, distribute enterprise data to different logical shards according to the geographical area level of the code prefix of enterprise data, perform hash calculation on the region code, set a maximum threshold for the data volume of each logical shard, and split the logical shard that exceeds the threshold; Deploy an edge data center in each region corresponding to a logical shard, and set several master nodes and secondary nodes in the region, and the edge data center manages the master nodes and secondary nodes to process the logical shard data; The master node is used to process all write requests for the corresponding logical shard data, and distributes the write requests to the corresponding nodes, and the master node uses a rewritable non-volatile memory and supports high-frequency read and write; The secondary node stores persistent data and uses LSM-Tree log-structured merge tree to manage data, and periodically changes data according to write requests.

[0007] Optionally, in one example of the above aspect, each logical shard data is divided into high-frequency update data and ordinary update data, and the high-frequency update data and the ordinary update data are deployed in the nodes close to the user in the edge data center, including the following steps: According to the data update log, for the data of each logical shard, the update frequency of the logical shard data within a unit time is calculated to obtain the update frequency; dynamic threshold setting is performed, and the average value of the global top 30% highest update frequency is set as the high-frequency update threshold; The data with an update frequency greater than or equal to the high-frequency update threshold is divided into high-frequency update data, and the remaining data is divided into ordinary update data; Deploy a lightweight data copy of the high-frequency update data in the master node close to the user in the edge data center, and deploy a lightweight data copy of the ordinary update data in the secondary node close to the user; Randomly select a distribution master node as a unified write entry, and route all write requests to the distribution master node first; The distribution master node synchronizes the update request of the high-frequency update data to the local and corresponding high-frequency storage area of other master nodes; The distribution master node forwards the normal update request to the corresponding secondary node asynchronously, and the secondary node updates the storage area data according to the normal update request; The master node pressure control is performed, and the number of concurrent single master node update requests Chot is set as α*number of master node CPU cores, wherein α is an empirical coefficient, and the flow control is triggered when the number of concurrent requests exceeds the number; The secondary node receives the update request, lists the update request in the message queue of the secondary node, and consumes the update request according to the priority.

[0008] Optionally, in one example of the above aspect, a Raft+PBFT hybrid consensus mechanism is used for global verification of all logical shards, including the following steps: The master node initiates a global verification request periodically for each logical shard, the update request of the data in the master node adopts Raft consensus, and the update request of the data in the secondary node adopts PBFT consensus; The Raft consensus is used for detection, and a consistency coefficient of the master node data is calculated: PRaft=(N0-1) / 2-Nf, wherein N0 is the total amount of updated data in the master node, and Nf is the amount of data in the master node whose update delay exceeds a threshold value; The PBFT consensus is used for detection, and a security coefficient of the secondary node data is calculated: Ppbft=(2f+1) / N1; Wherein f is the amount of data in the secondary node that has an error in the update process, and N1 is the total amount of updated data in the secondary node; According to the verification results returned by the nodes of each logical shard, global verification is performed on all logical shards to verify whether the consistency coefficient of the master node data and the security coefficient of the secondary node data in each logical shard exceed a threshold value, if yes, the global verification is completed, otherwise, the nodes corresponding to the threshold values are marked; The marked nodes are detected again, if the detection result exceeds the threshold value again, the global verification is completed, otherwise, for the marked nodes, if the master node, the secondary node is downgraded, if the secondary node, the node is revoked, and after the corresponding node is added and the data is transferred, the global verification is performed again until the global verification is passed.

[0009] Optionally, in one example of the above aspect, the running state of the master node of the edge data center is monitored, dynamic master node switching is performed, and the secondary node triggers automatic switching through heartbeat detection and blockchain log rollback, including the following steps: The secondary node periodically and randomly sends a heartbeat request to the master node, and marks it as abnormal if it does not respond within a timeout period; The secondary node periodically and randomly requests the usage rate, memory occupation and network bandwidth utilization rate data of the master node, and dynamically calculates the node health degree through a formula: HealthScore = β1 * (1 - CPU usage) + β2 * (1 - memory occupancy) + β3 * network bandwidth utilization; wherein, HealthScore is the node health degree, β1, β2 and β3 are weight coefficients, The secondary node periodically checks the blockchain log submitted by the primary node, and if a fork or invalid block is found, initiates a dispute resolution process. Set the switching trigger condition: k consecutive failures of heartbeat detection; the health score is continuously lower than the threshold value for more than T time; the primary node is detected to be malicious by the blockchain log rollback; If the primary node meets the switching trigger condition, switch the corresponding primary node, and the secondary node elects a new primary node through the Raft algorithm, and enters the candidate state after meeting the quorum; The new primary node synchronizes data from the switched primary node, pulls the latest blockchain log, executes the view change protocol of the PBFT consensus, updates the switched primary node cluster configuration and broadcasts it to all switched primary nodes, and the new primary node starts processing write requests, and the original primary node is downgraded to a secondary node.

[0010] Optionally, in one example of the above aspect, the change data capture module is deployed in a cloud computing environment, real-time monitoring of database logs of each logical shard, generating data change event streams of each logical shard, analyzing the update correlation degree between each node, and constructing an inter-node update correlation degree topology graph, including the following steps: Deploy the change data capture module in the cloud computing environment, capture the data changes of the database logs of each logical shard by analyzing the database transaction logs, deploy an independent open source CDC change data capture tool for each logical shard, and capture the data changes of the database logs of each logical shard; Set the data change event stream structure, each change event corresponding to a logical shard contains operation type, timestamp, primary key, and pre / post change data, and generate a corresponding logical shard data change event stream by grouping change events; According to the corresponding logical shard data change event stream data, and the consistency coefficient of the primary node data and the security coefficient of the secondary node data in the logical shard, analyze the update correlation degree between each node, and construct an inter-node update correlation degree topology graph.

[0011] Optionally, in one example of the above aspect, according to the corresponding logical shard data change event stream data, and the consistency coefficient of the primary node data and the security coefficient of the secondary node data in the logical shard, analyze the update correlation degree between each node, and construct an inter-node update correlation degree topology graph, including the following steps: Analyze the update correlation degree between each master node, the update correlation degree between each secondary node, and the update correlation degree between each node respectively; sum up the update correlation degrees calculated between each node corresponding to a logical shard, rank them from high to low, establish a correlation edge for the top 30% nodes, and construct an update correlation degree topology graph between nodes.

[0012] Optionally, in one example of the above aspect, according to the data change event stream and the update correlation degree topology graph between nodes, the update priority of each node in the edge data center is calculated, including the following steps: According to the constructed update correlation degree topology graph between nodes and the sum of the update correlation degrees calculated between each node corresponding to a logical shard, the weight wij of the correlation edge between nodes is calculated, wij = rij / rmax, wherein, rij is the sum of the update correlation degrees calculated between nodes, and rmax is the maximum value of the sum of the update correlation degrees calculated between each node in the update correlation degree topology graph; According to the update correlation degree topology graph between nodes, the weighted degree centrality of node i is calculated ; wherein Aij is an element of the adjacency matrix A, if node i and node j establish a correlation edge, then Aij = 1, otherwise 0, j ∈ (1, 2, …, N), N is the total number of correlation edges established by node i; Scan the Docker container of node i using Trivy, count the number and severity of security vulnerabilities, and calculate the data security score ; wherein Vi is the vulnerability set discovered by Trivy scanning of node i, wseverity(v) is the severity weight of vulnerability v, and wtype(v) is the vulnerability type weight; According to the obtained weighted degree centrality of nodes and node data security score, the update priority of each node in the edge data center is calculated.

[0013] Optionally, in one example of the above aspect, according to the obtained weighted degree centrality of nodes and node data security score, the update priority of each node in the edge data center is calculated, including the following steps: The obtained weighted degree centrality Si and node data security score Si of nodes are normalized; According to whether the node sends a priority update request for the data change event sk, the time sensitivity function T(sk) of the data change event sk is set: ; Combining the normalized weighted degree centrality Di' and node data security score Si' of nodes, and the time sensitivity function T(sk), the update priority coefficient Pi of each node in the edge data center is calculated: Pi = γ1 * Si' + γ2 * Di' + γ3 * T(sk) ; Wherein, γ1, γ2 and γ3 are weight coefficients.

[0014] Optionally, in one example of the above aspect, the update request is routed to the nearest high update priority node using Anycast IP, and after the high update priority nodes synchronize the update request with each other, the update request is synchronized to other nodes, including the following steps: According to the update priority coefficients of each node in the edge data center, the update priority coefficients of the nodes are sorted from high to low, and the nodes that are high update priority nodes are selected as high update priority nodes; All high update priority nodes announce the same Anycast IP through BGP, and the router selects the nearest high update priority node according to the AS_PATH length; After the high update priority nodes synchronize the update request with each other through gRPC Stream, the high update priority nodes publish the update event through MQTT / Kafka, and other nodes subscribe and pull, and the update request is synchronized to other nodes.

[0015] Compared with the prior art, the beneficial effects of the present application are: The high-frequency update data of the present application is deployed in the edge node close to the user, which reduces the round-trip time of the cloud and improves the user experience; avoids frequent transmission of high-frequency data across data centers, reduces the bandwidth pressure of the core network, and combines the Raft+PBFT hybrid consensus mechanism to perform global verification on all logical shards, thereby ensuring the update consistency and security of the node data.

[0016] The present application adjusts in real time through the topology graph with the change of business, adapts to the evolution of data model; combines the correlation degree, data importance and node load to calculate the priority, and at the same time uses Anycast IP to route the update request to the nearest high update priority node, and after the high update priority nodes synchronize the update request with each other, the update request is synchronized to other nodes, which ensures that the high update priority nodes quickly obtain the update request, and at the same time, realizes low-latency, high-availability and strong-consistency update propagation, while considering security and business continuity. When actually deployed, the parameters need to be adjusted according to the network size and business requirements. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0018] Fig. 1 A schematic diagram of a system framework of the present application is shown in Figure 1. Fig. 2 A schematic diagram of a logical slice data management method of an edge management layer of the present application is shown in Figure 2. DETAILED DESCRIPTION

[0019] The technical solutions of the present application will be described in detail below with reference to the embodiments. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0020] Referring to Figure 1, Figs. 1-2 The first aspect of the present application provides a cloud computing-based enterprise data update management system, which comprises: An edge management layer: According to the region code of the enterprise data, the enterprise data is divided into a plurality of logical slices, and an edge data center is set in the region corresponding to the region code of each logical slice, and the logical slice data in the edge data center is managed by a plurality of master nodes and auxiliary nodes; The logical slice data is divided into high-frequency update data and ordinary update data, and the high-frequency update data and the ordinary update data are deployed in the nodes close to the users in the edge data center; Raft is used for intra-slice fast election (to solve the brain split problem), and PBFT (practical Byzantine fault tolerance) is used for cross-slice global verification (to resist Byzantine attack).

[0021] A Raft+PBFT hybrid consensus mechanism is used to perform global verification on all logical slices, monitor the running state of the master nodes of the edge data center, perform dynamic master node switching, and the auxiliary nodes automatically switch through heartbeat detection and blockchain log rollback triggering; A cloud update request management layer: A change data capture module is deployed in the cloud computing environment to monitor the database logs of each logical slice in real time, generate data change event streams of each logical slice, analyze the update correlation degree between each node, and construct an update correlation degree topology graph between nodes; According to the data change event stream and the update correlation degree topology graph between nodes, the update priority of each node in the edge data center is calculated, the update request is routed to the nearest high update priority node using Anycast IP, and after the high update priority nodes synchronize the update request with each other, the update request is synchronized to other nodes.

[0022] Specifically, in this embodiment, high-frequency update data is deployed in edge nodes close to users, reducing cloud round-trip time and improving user experience; avoiding frequent cross-data center transmission of high-frequency data, reducing core network bandwidth pressure. The single master election and log replication mechanism of Raft is suitable for high-frequency data, ensuring low-latency writing (usually <100 ms). Compared with PBFT, Raft's communication is lightweight and suitable for resource-constrained edge node scenarios; PBFT is used for ordinary update data (such as financial records and configuration information) to tolerate some node malfunctions or failures; ensure that all edge nodes agree on the modification of key data to avoid data divergence.

[0023] The secondary node detects the survival state of the primary node through heartbeat, and triggers an election if it times out. The blockchain stores all operation logs, and the new primary node can roll back to the most recent consistent state to avoid data loss. Through the election restriction of Raft, only nodes that receive a certain number of votes to reach a preset value are allowed to become primary nodes, realizing the preliminary setting of primary nodes. At the same time, through the Raft+PBFT hybrid consensus mechanism, all logical shards are globally checked to ensure the consistency and security of node data updates; Raft ensures strong consistency among edge nodes, but only in a non-Byzantine environment. PBFT prevents malicious nodes from tampering through digital signatures and three-phase submission (Pre-Prepare / Prepare / Commit).

[0024] All update operations can also be recorded on the blockchain to provide an unalterable audit trail. Regulatory agencies can verify data integrity to meet compliance requirements.

[0025] By analyzing database transaction logs, data changes can be captured without modifying application code, avoiding performance interference with business systems; converting changed data into standardized event streams (such as JSON / Protobuf format) for easy subsequent processing and transmission. Through Anycast technology, update requests are automatically routed to the nearest edge node based on user geographic location or network topology, reducing transmission delay. Traditional DNS polling or load balancing may cause delay fluctuations due to network congestion, while Anycast provides deterministic low latency. If the nearest high-update priority node is overloaded, Anycast can automatically distribute requests to other high-update priority nodes to avoid single-point overload.

[0026] By analyzing historical change events, the correlation strength between data updates among nodes can be calculated (such as Pearson correlation coefficient, mutual information). If the frequent updates of node A (user information) trigger cascading updates of node B (order information), the correlation between A→B is high.

[0027] The topology graph is adjusted in real time with business changes, adapts to data model evolution, and calculates priority in combination with correlation degree, data importance, and node load (CPU / memory / bandwidth). Raft / PBFT hybrid consensus is used to ensure data consistency between nodes and avoid split-brain. At the same time, low-priority nodes asynchronously receive updates through a message queue (such as Kafka) to tolerate short delays, but eventually achieve consistency.

[0028] The topology graph is adjusted in real time with business changes, adapts to data model evolution, and calculates priority in combination with correlation degree, data importance, and node load. At the same time, Anycast IP is used to route update requests to the nearest high-update-priority node. After high-update-priority nodes synchronize update requests with each other, the update requests are synchronized to other nodes to ensure that high-update-priority nodes quickly obtain update requests. At the same time, low-latency, high-availability, and strong-consistency update propagation can be achieved while considering security and business continuity. Actual deployment requires adjusting parameters according to network size and business needs.

[0029] In one embodiment of the present application, according to the region code of enterprise data, the enterprise data is divided into several logical shards, and in the region code corresponding area of each logical shard, an edge data center is set up. The logical shard data in the edge data center is managed by several master nodes and auxiliary nodes, including the following steps: According to the region code of enterprise data, set the shard strategy selection, and according to the geographical area level of the code prefix of enterprise data, such as the postal code and administrative division code, distribute the enterprise data to different logical shards. Hash calculate the region code, set the maximum threshold of the data volume of each logical shard, and split the logical shard that exceeds the threshold; According to the code level division: Provincial shard: divided by the first two digits of the code (such as 11-Beijing); City-level shard: divided by the first four digits (such as 1101-Beijing municipal area); Deploy an edge data center in the area corresponding to each logical shard, and set up several master nodes and auxiliary nodes in the area. The edge data center manages the master nodes and auxiliary nodes to process logical shard data; The master node is used to process all write requests for the corresponding logical shard data, and distributes the write requests to the corresponding nodes. The master node uses rewritable non-volatile memory, such as 3D XPoint, to support high-frequency read and write; The auxiliary node stores persistent data and uses LSM-Tree log-structured merge tree to manage data, and periodically changes data according to write requests.

[0030] In this embodiment, when the capacity of a shard is expanded, a trigger condition is set: when the data volume of a shard exceeds a threshold (e.g., 100 GB) or the QPS is continuously too high, the split is automatically triggered.

[0031] Split process: suspend the write operation of the shard. Redistribute the data according to the new shard strategy (e.g., hash modulo).

[0032] Synchronize the routing table to direct part of the requests to the new shard. Resume the service and monitor the migration progress.

[0033] In one embodiment of the present application, the data of each logical shard is divided into high-frequency update data and ordinary update data, and the high-frequency update data and the ordinary update data are deployed in the nodes close to the users in the edge data center, including the following steps: According to the data update log, for the data of each logical shard, the update frequency of the data of the logical shard within a unit time is counted to obtain the update frequency; dynamic threshold setting is performed, and the average value of the top 30% highest update frequencies globally is set as the high-frequency update threshold; Data with an update frequency greater than or equal to the high-frequency update threshold is divided into high-frequency update data, and the remaining data is divided into ordinary update data; The lightweight data replicas of the high-frequency update data are deployed in the master nodes close to the users in the edge data center, and the lightweight data replicas of the ordinary update data are deployed in the auxiliary nodes close to the users; Randomly select a distribution master node as a unified write entry, and route all write requests to the distribution master node first; The distribution master node synchronizes the update request of the high-frequency update data to the high-frequency storage area of the local and corresponding other master nodes; The distribution master node asynchronously forwards the ordinary update request to the corresponding auxiliary node, and the auxiliary node updates the storage area data according to the ordinary update request; Perform master node pressure control, set the concurrency number Chot of the update request of a single master node to α*CPU core number of the master node, wherein α is an empirical coefficient, and in this embodiment, the value is 0.8, and the flow control is triggered when the concurrency number is exceeded; The auxiliary node receives the update request and adds the update request to the message queue of the auxiliary node for priority consumption.

[0034] In one embodiment of the present application, a Raft+PBFT hybrid consensus mechanism is used to perform global verification on all logical shards, including the following steps: The master node in each logical shard initiates a global verification request regularly, and the update request of the data in the master node adopts Raft consensus, and the update request of the data in the auxiliary node adopts PBFT consensus; The Raft consensus is used for detection, and a consistency coefficient of the master node data is calculated: PRaft=(N0-1) / 2-Nf, wherein N0 is the total amount of updated data of the master node, and Nf is the amount of data in the master node whose update delay exceeds a threshold value; The PBFT consensus is used for detection, and a security coefficient of the secondary node data is calculated: Ppbft=(2f+1) / N1; wherein f is the amount of data in the secondary node that appears an error in the update process, and N1 is the total amount of updated data of the secondary node; According to the check results of the logical shards, global checking is performed on all logical shards to verify whether the consistency coefficient of the master node data and the security coefficient of the secondary node data in each logical shard exceed the threshold value, if yes, the global checking is completed, otherwise, the nodes corresponding to the threshold values are marked; The marked nodes are re-detected, if the re-detection result exceeds the threshold value, the global checking is completed, otherwise, for the marked nodes, if they are master nodes, they are downgraded to secondary nodes, if they are secondary nodes, they are revoked, and then the corresponding nodes are re-added and the data is transferred, and the global checking is performed again until the global checking is passed.

[0035] In this embodiment, the consistency of the master node data and the security of the secondary node data in the logical shard are calculated by collecting a large amount of data of the fault nodes, and the corresponding consistency coefficient and security coefficient are calculated, and the average value is taken as the threshold value of the consistency coefficient and the security coefficient.

[0036] In one embodiment of the present application, the running state of the master node of the edge data center is monitored, dynamic master node switching is performed, and the secondary node automatically switches through heartbeat detection and blockchain log rollback triggering, including the following steps: The secondary node periodically and randomly sends a heartbeat request to the master node, and marks it as abnormal if it does not respond within a timeout period; The secondary node periodically and randomly requests the usage rate, memory occupation and network bandwidth utilization rate data from the master node, and dynamically calculates the node health degree through a formula: HealthScore=β1*(1-CPU usage rate)+β2*(1-memory occupation)+β3*network bandwidth utilization rate; wherein HealthScore is the node health degree, β1, β2 and β3 are weight coefficients (in this embodiment, β1=0.5, β2=0.3, β3=0.2), The secondary node periodically checks the blockchain log submitted by the master node, and starts a dispute resolution process if it finds a fork or invalid block; The switching trigger condition is set as follows: The heartbeat detection fails continuously for k times (in this embodiment, k=3).

[0037] The health score continuously falls below a threshold (in this embodiment, the health score threshold is set to 0.3) for more than T time (in this embodiment, T = 5 minutes); The blockchain log rollback detects that the primary node is malicious; If the primary node meets the switching trigger condition, the corresponding primary node is switched, the secondary node elects a new primary node through the Raft algorithm, and meets the quorum to enter the candidate state; The new primary node synchronizes data from the switched primary node, pulls the latest blockchain log, executes the view change protocol of the PBFT consensus, updates the switched primary node cluster configuration and broadcasts it to all switched primary nodes, and the new primary node starts processing write requests. The original primary node is downgraded to a secondary node.

[0038] In one embodiment of the present application, a change data capture module is deployed in a cloud computing environment to monitor database logs of each logical shard in real time, generate data change event streams for each logical shard, analyze the update correlation degree between each node, and construct a node-to-node update correlation degree topology graph, including the following steps: In a cloud computing environment, a change data capture module is deployed by analyzing database transaction logs (such as MySQL binlog, MongoDB oplog, and PostgreSQL WAL), and an independent open source CDC change data capture tool is deployed for each logical shard to capture data changes in the database logs of each logical shard; Tool selection: Debezium: an open source CDC tool that supports Kafka as a message middleware and is compatible with MySQL, PostgreSQL, and other mainstream databases. Amazon DocumentDB Change Stream: a cloud-native service that directly captures change events from MongoDB-compatible databases and supports DDL events (such as creating indexes and shards). Google CloudDatastream: a serverless CDC service that supports real-time replication from Oracle and MySQL to BigQuery, simplifying the configuration process.

[0039] Set the data change event stream structure, each change event corresponding to a logical shard contains the operation type, timestamp, primary key, and pre / post change data (such as the updateDescription field in MongoDB), and the change events are grouped to generate the corresponding logical shard data change event stream; According to the corresponding logical shard data change event stream data, as well as the consistency coefficient of the primary node data and the security coefficient of the secondary node data in the logical shard, the update correlation degree between each node is analyzed, and a node-to-node update correlation degree topology graph is constructed.

[0040] In one embodiment of the present application, according to the data change event stream of the corresponding logical shard and the consistency coefficient of the master node data and the security coefficient of the secondary node data in the logical shard, the update correlation degree between each node is analyzed, and a node update correlation degree topology graph is constructed, including the following steps: For each master node in the logical shard, according to the data change event stream of the corresponding logical shard and the consistency coefficient of the master node data in the logical shard, the update correlation degree between each master node is analyzed: Wherein, Raij is the update correlation degree between the master nodes, Mh0 is the common change event number between the master nodes, Ms0 is the product of the total change event number between the master nodes, Zaij is the difference between the consistency coefficients, Saij is the average value of the consistency coefficients of the master nodes in the logical shard, is the time decay coefficient; For each secondary node in the logical shard, according to the data change event stream of the corresponding logical shard and the security coefficient of the secondary node data, the update correlation degree between each secondary node is analyzed: Wherein, Rbij is the update correlation degree between the secondary nodes, Ah0 is the common change event number between the secondary nodes, As0 is the product of the total change event number between the secondary nodes, Zxij is the difference between the security coefficients, Sxij is the average value of the security coefficients of the secondary nodes in the logical shard; According to the data change event stream of the corresponding logical shard, the update correlation degree between each node is analyzed: Wherein, Rij is the update correlation degree between nodes i and j, eik is the count of the kth type of change event of node i in the time window T, and N is the total number of change event types; The time decay coefficient is calculated by the following formula: Wherein, Δtij is the time difference (in seconds) of the last common change of nodes i and j, and λ is the time decay rate; (such as λ=0.01, which means that the correlation degree is halved every 100 seconds).

[0041] The sum of the update correlation degrees calculated between each node of the corresponding logical shard is counted, and ranked from high to low, and the nodes ranked in the top 30% are connected to build a node update correlation degree topology graph.

[0042] In one embodiment of the present application, according to the data change event stream and the node update correlation degree topology graph, the update priority of each node in the edge data center is calculated, including the following steps: According to the constructed inter-node update correlation degree topology graph and the sum of the calculated update correlation degrees between the nodes of each logical shard, the weight wij of the correlation edge between the nodes is calculated, wherein wij=rij / rmax, rij is the sum of the calculated update correlation degrees between the nodes, and rmax is the maximum value of the sum of the calculated update correlation degrees between the nodes in the update correlation degree topology graph. According to the inter-node update correlation degree topology graph, the weighted degree centrality of node i is calculated ; wherein Aij is an element of the adjacency matrix A, if node i and node j establish a correlation edge, then Aij=1, otherwise 0, j∈(1,2, …, N), and N is the total number of correlation edges established by node i. The Trivy is used to scan the Docker container of node i, the number and severity of security vulnerabilities are counted, and the data security score is calculated ; wherein Vi is the vulnerability set discovered by the Trivy scanning of node i, wseverity(v) is the severity weight of vulnerability v (such as Critical=3, High=2, Medium=1, Low=0.5), and wtype(v) is the vulnerability type weight (such as RCE vulnerability weight is higher). According to the obtained weighted degree centrality of the node and the data security score of the node, the update priority of each node in the edge data center is calculated.

[0043] In one embodiment of the present application, according to the obtained weighted degree centrality of the node and the data security score of the node, the update priority of each node in the edge data center is calculated, including the following steps: The obtained weighted degree centrality Si of the node and the data security score Si of the node are normalized. According to whether the node sends a priority update request for the data change event sk, the time sensitivity function T(sk) of the data change event sk is set: ; The update priority coefficient Pi of each node in the edge data center is calculated in combination with the normalized weighted degree centrality Di' of the node and the data security score Si' of the node, and the time sensitivity function T(sk): Pi=γ1*Si'+γ2*Di'+γ3*T(sk); Wherein γ1, γ2 and γ3 are weight coefficients, which control the relative importance of security and business correlation degree, in the present embodiment, γ1, γ2 and γ3 are set to 0.4, 0.3 and 0.3 respectively.

[0044] In one embodiment of the present application, Anycast IP is used to route the update request to the nearest high update priority node, and after the high update priority nodes synchronize the update request with each other, the update request is synchronized to other nodes, including the following steps: According to the update priority coefficients of each node in the edge data center, the update priority coefficients of the nodes are sorted from high to low, and the nodes with high update priority are selected as high update priority nodes; All high update priority nodes announce the same Anycast IP (such as 198.51.100.1) through BGP, and the router selects the nearest high update priority node according to the AS_PATH length; After the high update priority nodes synchronize the update request with each other through gRPC Stream, the high update priority nodes publish the update event through MQTT / Kafka, and other nodes subscribe and pull, so as to synchronize the update request to other nodes.

[0045] The above embodiments are only used to illustrate the technical method of the present application and are not limited. Although the present application has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical method of the present application can be modified or replaced equivalently without departing from the spirit and scope of the technical method of the present application.

Claims

1. A cloud computing-based enterprise data update management system, characterized by, The application relates to an edge management layer, a cloud update request management layer and a data update request processing method. According to the region code of enterprise data, the enterprise data is divided into a plurality of logical fragments, an edge data center is arranged in a region corresponding to the region code of each logical fragment, and the logical fragment data in the edge data center is managed by a plurality of master nodes and auxiliary nodes. High-frequency update data and ordinary update data are arranged in the nodes close to users in the edge data center. A Raft+PBFT hybrid consensus mechanism is adopted to globally check all logical fragments, monitor the running state of the master nodes of the edge data center, dynamically switch the master nodes, and automatically switch the auxiliary nodes through heartbeat detection and blockchain log rollback triggering. A change data capture module is arranged in a cloud computing environment to monitor the database logs of the logical fragments in real time, generate a data change event stream of the logical fragments, analyze the update correlation degree between the nodes, and construct a node update correlation degree topology graph. According to the data change event stream and the node update correlation degree topology graph, the update priority of each node in the edge data center is calculated, the update request is routed to the nearest high update priority node using Anycast IP, and the update request is synchronized to other nodes after being synchronized between the high update priority nodes. According to the region code of enterprise data, the enterprise data is divided into a plurality of logical fragments, an edge data center is arranged in a region corresponding to the region code of each logical fragment, and the logical fragment data in the edge data center is managed by a plurality of master nodes and auxiliary nodes. According to the region code of enterprise data, a fragmentation strategy is selected, the enterprise data is allocated to different logical fragments according to the geographical region level of the code prefix of the enterprise data, the region code is hashed, a maximum threshold of the data volume of each logical fragment is set, and the logical fragments exceeding the threshold are split.

2. The cloud computing based enterprise data update management system as claimed in claim 1, wherein, An edge data center is arranged in a region corresponding to each logical fragment, and a plurality of master nodes and auxiliary nodes are arranged in the region, and the edge data center manages the master nodes and auxiliary nodes to process the logical fragment data. The master node is used for processing all write requests of the corresponding logical fragment data, and the write request is distributed to the corresponding node. The auxiliary node stores persistent data and adopts an LSM-Tree log-structured merge tree to manage the data and periodically changes the data according to the write request. According to the data update log, the update frequency of the data of each logical fragment in a unit time is obtained by counting the update times of the logical fragment data in a unit time, a dynamic threshold is set, and the average value of the top 30% highest update frequencies is set as a high-frequency update threshold. The data with the update frequency greater than or equal to the high-frequency update threshold is divided into high-frequency update data, and the remaining data is divided into ordinary update data.

3. The cloud computing based enterprise data update management system as claimed in claim 1, wherein, ​ ​ ​ Deploying a light data copy of high-frequency update data in the edge data center close to the user's master node, and deploying a light data copy of ordinary update data in the user's secondary node close to the user; Randomly select a distribution master node as a unified write entry, and route all write requests to the distribution master node first; The distribution master node synchronizes the update request of the high-frequency update data to the high-frequency storage area of the local and corresponding other master nodes; The distribution master node forwards the ordinary update request to the corresponding secondary node asynchronously, and the secondary node updates the storage area data according to the ordinary update request; Perform master node pressure control, set the concurrency of single master node update request Chot=α*master CPU core number, where α is an empirical coefficient, and trigger flow limiting when the concurrency is exceeded; The secondary node receives the update request, lists the update request in the message queue of the secondary node, and consumes it according to the priority.

4. The cloud computing based enterprise data update management system as claimed in claim 1, wherein, A Raft+PBFT hybrid consensus mechanism is used to perform global verification on all logical shards, including the following steps: Periodically initiate a global verification request for the master node in each logical shard, and use Raft consensus for the update request in the master node and PBFT consensus for the update request in the secondary node; Use Raft consensus to detect and calculate the consistency coefficient of the master node data: PRaft=(N0-1) / 2-Nf, where N0 is the total amount of updated data in the master node, and Nf is the amount of data in the master node whose update delay exceeds the threshold value; Use PBFT consensus to detect and calculate the security coefficient of the secondary node data: Ppbft=(2f+1) / N1, where f is the amount of data that has an error in the update process in the secondary node, and N1 is the total amount of updated data in the secondary node; According to the verification results returned by the nodes of each logical shard, perform global verification on all logical shards to verify whether the consistency coefficient of the master node data and the security coefficient of the secondary node data in each logical shard exceed the threshold value, if yes, complete the global verification, otherwise, mark the corresponding nodes that exceed the threshold value; Re-detect the marked nodes, if the re-detection result exceeds the threshold value, complete the global verification, otherwise, for the marked nodes, if they are master nodes, downgrade them to secondary nodes, if they are secondary nodes, revoke them, and then re-add the corresponding nodes and transfer data, and then perform global verification again until the global verification is passed.

5. The cloud computing based enterprise data update management system as claimed in claim 1, wherein, Monitor the running state of the master node of the edge data center, and perform dynamic master node switching, including the following steps: The secondary node periodically sends a heartbeat request to the master node at random, and marks it as abnormal if it does not respond within a timeout period; The secondary node periodically requests the usage rate, memory occupation and network bandwidth utilization rate data from the master node at random, and dynamically calculates the node health score through the formula: HealthScore=β1*(1-CPU usage rate)+β2*(1-memory occupation)+β3*network bandwidth utilization rate; where HealthScore is the node health score, β1, β2 and β3 are weight coefficients, The secondary node periodically verifies the blockchain log submitted by the master node, and starts the dispute resolution process if it finds a fork or invalid block. Set the switching trigger condition: heartbeat detection fails continuously k times; health score is continuously lower than the threshold value for more than T time; blockchain log rollback detects that the primary node is malicious; If the primary node meets the switching trigger condition, switch the corresponding primary node, and the secondary node elects a new primary node through the Raft algorithm. After meeting the quorum, it enters the candidate state; The new primary node synchronizes data from the switched primary node, pulls the latest blockchain log, executes the view change protocol of the PBFT consensus, updates the switched primary node cluster configuration and broadcasts it to all switched primary nodes. The new primary node starts processing write requests, and the original primary node is downgraded to a secondary node.

6. The cloud computing based enterprise data update management system as claimed in claim 1, wherein, Deploy a change data capture module in a cloud computing environment to monitor database logs of each logical shard in real time, generate data change event streams for each logical shard, analyze the update correlation between each node, and build a node update correlation topology graph, including the following steps: Deploy a change data capture module in a cloud computing environment to capture data changes in database logs of each logical shard by analyzing database transaction logs. Each logical shard deploys an independent open-source CDC change data capture tool to capture data changes in database logs of each logical shard. Set the data change event stream structure. Each change event for a logical shard includes operation type, timestamp, primary key, and pre / post change data. Group the change events to generate a data change event stream for the corresponding logical shard. According to the data change event stream data of the corresponding logical shard, and the consistency coefficient of the primary node data and the security coefficient of the secondary node data in the logical shard, analyze the update correlation between each node, and build a node update correlation topology graph.

7. The cloud computing based enterprise data update management system as claimed in claim 6, wherein, According to the data change event stream data of the corresponding logical shard, and the consistency coefficient of the primary node data and the security coefficient of the secondary node data in the logical shard, analyze the update correlation between each node, and build a node update correlation topology graph, including the following steps: For each primary node in each logical shard, analyze the update correlation between each primary node based on the data change event stream data of the corresponding logical shard and the consistency coefficient of the primary node data in the logical shard. Wherein, Raij is the update correlation degree between the master nodes, Mh0 is the common change event number between the master nodes, Ms0 is the product of the total change event number between the master nodes, Zaij is the difference of the consistency coefficients between the master nodes, Saij is the average of the consistency coefficients of the master nodes in the logical shard, is a time decay coefficient; For each secondary node in each logical shard, analyze the update correlation between each secondary node based on the data change event stream data of the corresponding logical shard and the security coefficient of the secondary node data. Where Rbij is the update correlation between secondary nodes, Ah0 is the product of the number of common change events between secondary nodes and As0 is the total number of change events between secondary nodes, Zxij is the difference between the security coefficients of secondary nodes, and Sxij is the average security coefficient of secondary nodes in the logical shard. According to the data change event stream data of the corresponding logical shard, analyze the update correlation between each node: Where Rij is the update correlation between nodes i and j, eik is the count of the kth type of change event for node i within the time window T, and N is the total number of change event types. The time decay coefficient is calculated by the following formula: Where Δtij is the time difference of the last common change between nodes i and j, and λ is the time decay rate. The sum of the update correlation degrees calculated between each node of the corresponding logical shard is counted, ranked from high to low, and the nodes ranked in the top 30% are connected by an association edge to construct an update correlation degree topology graph between nodes.

8. The cloud computing based enterprise data update management system as claimed in claim 1, wherein, According to the data change event stream and the update correlation degree topology graph between nodes, the update priority of each node in the edge data center is calculated, including the following steps: According to the constructed update correlation degree topology graph between nodes and the sum of the update correlation degrees calculated between each node of the corresponding logical shard, the weight wij of the association edge between nodes is calculated wij=rij / rmax, wherein rij is the sum of the update correlation degrees calculated between nodes, and rmax is the maximum value of the sum of the update correlation degrees calculated between each node in the update correlation degree topology graph. According to the inter-node updating correlation degree topology map, the weighted degree centrality of the node i is calculated ; wherein, Aij is an element of the adjacency matrix A, if the node i and the node j establish a correlation edge, then Aij=1, otherwise 0, j∈(1,2,…,N), N is the total number of correlation edges established by the node i; Scan the Docker containers of node i using Trivy, count the number and severity of security vulnerabilities, and calculate the data security score ; wherein, Vi is the set of vulnerabilities discovered by Trivy scanning of node i, wseverity(v) is the severity weight of vulnerability v, and wtype(v) is the vulnerability type weight. According to the obtained weighted degree centrality of the node and the node data security score, the update priority of each node in the edge data center is calculated.

9. The cloud computing based enterprise data update management system as claimed in claim 8, wherein, According to the obtained weighted degree centrality of the node and the node data security score, the update priority of each node in the edge data center is calculated, including the following steps: The obtained weighted degree centrality Si of the node and the node data security score Si are normalized. According to whether the node sends a priority update request for the data change event sk, the time sensitivity function T(sk) of the data change event sk is set: ； In combination with the normalized weighted degree centrality Di' of the node and the node data security score Si', and the time sensitivity function T(sk), the update priority coefficient Pi of each node in the edge data center is calculated: Pi=γ1*Si′+γ2*Di′+γ3*T(sk); Wherein, γ1, γ2 and γ3 are weight coefficients.

10. The cloud computing based enterprise data update management system as claimed in claim 1, wherein, Anycast IP is used to route the update request to the nearest high update priority node, and after the high update priority nodes synchronize the update request with each other, the update request is synchronized to other nodes, including the following steps: According to the update priority coefficient of each node in the edge data center, the update priority coefficients of the nodes are sorted from high to low, and the nodes with high update priority are selected as high update priority nodes; All high update priority nodes announce the same Anycast IP through BGP, and the router selects the nearest high update priority node according to the AS_PATH length; After the high update priority nodes synchronize the update request with each other through gRPC Stream, the high update priority nodes publish the update event through MQTT / Kafka, and other nodes subscribe and pull to synchronize the update request to other nodes.

Images