Cloud computing-based enterprise data update management system

Abstract

The application discloses an enterprise data update management system based on cloud computing and relates to the technical field of data management, solves the technical problem that user requests need to go back and forth between the cloud and the core network, and meanwhile, the full-amount synchronization mechanism continuously occupies the backbone network bandwidth, forming a double performance bottleneck, through high-frequency update data deployment in the edge node close to the user, reducing the cloud round-trip time and improving the user experience, avoiding frequent cross-data center transmission of high-frequency data, reducing the core network bandwidth pressure, combining the Raft+PBFT hybrid consensus mechanism to perform global checking on all logical fragments, and guaranteeing the update consistency and safety of node data. Through real-time adjustment of the topological graph with business changes, the data model evolution is adapted; in combination with the correlation degree, data importance and node load, the priority is comprehensively calculated, the high-update priority node is ensured to quickly acquire the update request, and low-delay, high-availability and strong-consistency update propagation are realized.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of data management, and in particular to an enterprise data update management system based on cloud computing. BACKGROUND

[0002] In the wave of digital transformation, the scale of enterprise data grows exponentially, and the frequency of data updates continues to increase. Traditional centralized architecture has been difficult to meet the needs of high-frequency update scenarios due to high latency, bandwidth overload, and lack of consistency protection. The enterprise data update management system based on cloud computing systematically solves these pain points through distributed architecture, edge computing, intelligent routing, and hybrid consensus mechanism, becoming the key infrastructure supporting real-time decision-making and business innovation for enterprises.

[0003] If the prior art relies on the central cloud to centrally process all update data, user requests need to cross the core network and return to the cloud, and the full-synchronization mechanism will continuously occupy the backbone network bandwidth, forming a double performance bottleneck. SUMMARY

[0004] The present application aims to at least solve one of the technical problems existing in the prior art; for this purpose, the present application proposes an enterprise data update management system based on cloud computing to solve the technical problem that user requests need to cross the core network and return to the cloud, and the full-synchronization mechanism will continuously occupy the backbone network bandwidth, forming a double performance bottleneck.

[0005] To solve the above problems, the first aspect of the present application provides an enterprise data update management system based on cloud computing, comprising:

[0006] Edge management layer:

[0007] According to the region code of the enterprise data, the enterprise data is divided into several logical shards, and an edge data center is set up in the region corresponding to the region code of each logical shard, and the logical shard data in the edge data center is managed by several master nodes and auxiliary nodes;

[0008] Divide each logical shard data into high-frequency update data and ordinary update data, and deploy the high-frequency update data and the ordinary update data near the user's node in the edge data center;

[0009] A Raft+PBFT hybrid consensus mechanism is used to globally verify all logical shards, monitor the running state of the master nodes of the edge data center, and perform dynamic master node switching, and the auxiliary nodes automatically switch through heartbeat detection and blockchain log rollback triggering;

[0010] Cloud update request management layer:

[0011] The change data capture module is deployed in a cloud computing environment, real-time monitoring of each logical shard database log, generating each logical shard data change event stream, analyzing the update correlation between each node, and constructing the update correlation topology between nodes;

[0012] According to the data change event stream and the update correlation topology between nodes, the update priority of each node in the edge data center is calculated, the update request is routed to the nearest high update priority node using Anycast IP, and after the high update priority nodes are synchronized with each other, the update request is synchronized to other nodes.

[0013] Optionally, in one example of the above aspect, according to the region code of the enterprise data, the enterprise data is divided into several logical shards, and the edge data center is set in the region corresponding to each logical shard region code. The logical shard data in the edge data center is managed by several master nodes and secondary nodes, comprising the following steps:

[0014] According to the region code of the enterprise data, set the shard strategy selection, according to the geographical area level of the code prefix of the enterprise data, allocate the enterprise data to different logical shards, hash the region code, set the maximum threshold of the data volume of each logical shard, and split the logical shard that exceeds the threshold;

[0015] Deploy the edge data center in the region corresponding to each logical shard, and set several master nodes and secondary nodes in the region. The edge data center manages the master nodes and secondary nodes to process the logical shard data;

[0016] The master node is used to process all write requests of the corresponding logical shard data, and distributes the write requests to the corresponding nodes. The master node uses rewritable non-volatile memory and supports high-frequency read and write.

[0017] The secondary node stores persistent data and uses LSM-Tree log-structured merge tree to manage data, and periodically changes data according to write requests.

[0018] Optionally, in one example of the above aspect, each logical shard data is divided into high-frequency update data and ordinary update data, and the high-frequency update data and ordinary update data are deployed in the node close to the user in the edge data center, comprising the following steps:

[0019] According to the data update log, for the data of each logical shard, the update frequency of the logical shard data in a unit of time is obtained by counting the number of updates; dynamic threshold setting is performed, and the average value of the global top 30% highest update frequency is set as the high-frequency update threshold;

[0020] The data with an update frequency greater than or equal to the high-frequency update threshold is divided into high-frequency update data, and the remaining data is divided into ordinary update data.

[0021] In edge data centers, deploy lightweight data replicas of frequently updated data on the primary node closer to the user, and deploy lightweight data replicas of normally updated data on the secondary node closer to the user;

[0022] Randomly select a distribution master node as the unified write entry point, and route all write requests to the distribution master node first;

[0023] The distribution master node synchronizes update requests for frequently updated data to the local and corresponding high-frequency storage areas of other master nodes;

[0024] The primary node asynchronously forwards ordinary update requests to the corresponding secondary nodes, which then update the data in the storage area based on the ordinary update requests.

[0025] To implement master node pressure control, the concurrency limit for a single master node update request is set to Chot = α * number of master node CPU cores, where α is an empirical coefficient. Rate limiting is triggered when the concurrency limit is exceeded.

[0026] The secondary node receives the update request, adds the update request to the secondary node's message queue, and consumes it according to priority.

[0027] Optionally, in one example of the above aspects, a Raft+PBFT hybrid consensus mechanism is used to perform global verification of all logical shards, including the following steps:

[0028] The master nodes in each logical shard periodically initiate global verification requests. Data update requests in the master nodes use Raft consensus, while data update requests in the slave nodes use PBFT consensus.

[0029] Raft consensus is used for detection, and the consistency coefficient of the master node data is calculated as follows: PRaft=(N0-1) / 2-Nf, where N0 is the total amount of data updated by the master node, and Nf is the amount of data in the master node whose update latency exceeds the threshold.

[0030] The security coefficient of the auxiliary node data is calculated using the PBFT consensus mechanism: Ppbft=(2f+1) / N1;

[0031] Where f is the amount of data that encountered errors during the update process in the secondary node, and N1 is the total amount of data updated in the secondary node;

[0032] Based on the verification results returned by each logical shard, a global verification is performed on all logical shards to verify whether the consistency coefficient of the primary node data and the security coefficient of the secondary node data in each logical shard both exceed the threshold. If so, the global verification is completed; otherwise, the corresponding nodes that exceed the threshold are marked.

[0033] Re-detect the marked node, if the re-detection result exceeds the threshold value, the global check is completed, otherwise, for the marked node, if it is the master node, it is downgraded to the secondary node, if it is the secondary node, it is revoked, and the corresponding node is re-added and the data is transferred, and the global check is performed again until the global check is passed.

[0034] Optionally, in one example of the above aspect, the running state of the master node of the edge data center is monitored, dynamic master node switching is performed, the secondary node triggers automatic switching through heartbeat detection and blockchain log rollback, including the following steps:

[0035] The secondary node periodically and randomly sends a heartbeat request to the master node, and marks it as abnormal if it does not respond within a timeout period;

[0036] The secondary node periodically and randomly requests the usage rate, memory occupation and network bandwidth utilization rate data from the master node, and dynamically calculates the node health score through a formula:

[0037] HealthScore = β1*(1-CPU usage rate) + β2*(1-memory occupation) + β3*network bandwidth utilization rate; wherein HealthScore is the node health score, β1, β2 and β3 are weight coefficients,

[0038] The secondary node periodically checks the blockchain log submitted by the master node, and if it finds a fork or invalid block, it starts a dispute resolution process;

[0039] Set the switching trigger conditions: k consecutive failures in heartbeat detection; the health score is continuously below the threshold value for more than T time; the master node is detected to be malicious through blockchain log rollback;

[0040] If the master node meets the switching trigger condition, the corresponding master node is switched, the secondary node elects a new master node through the Raft algorithm, and enters the candidate state after meeting the quorum;

[0041] The new master node synchronizes data from the switched master node, pulls the latest blockchain log, executes the view change protocol of the PBFT consensus, updates the switched master node cluster configuration and broadcasts it to all switched master nodes, and the new master node starts processing write requests, and the original master node is downgraded to a secondary node.

[0042] Optionally, in one example of the above aspect, a change data capture module is deployed in a cloud computing environment, real-time monitoring of database logs of each logical shard is performed, data change event streams of each logical shard are generated, update correlation degrees between nodes are analyzed, and a node-to-node update correlation degree topology graph is constructed, including the following steps:

[0043] The change data capture module is deployed in a cloud computing environment, each logical shard deploys an independent open source CDC change data capture tool by parsing the database transaction log, and data changes of the database log of each logical shard are captured;

[0044] A data change event stream structure is set, each change event of the corresponding logical shard includes an operation type, a timestamp, a primary key, pre-change / post-change data, and a corresponding logical shard data change event stream is generated by grouping change events;

[0045] According to the corresponding logical shard data change event stream data, the consistency coefficient of the primary node data in the logical shard, and the security coefficient of the secondary node data, the update correlation degree between each node is analyzed, and a node update correlation degree topology graph is constructed.

[0046] Optionally, in one example of the above aspect, according to the corresponding logical shard data change event stream data, the consistency coefficient of the primary node data in the logical shard, and the security coefficient of the secondary node data, the update correlation degree between each node is analyzed, and a node update correlation degree topology graph is constructed, including the following steps:

[0047] The update correlation degree between each primary node, the update correlation degree between each secondary node, and the update correlation degree between each node are analyzed respectively; the sum of the update correlation degrees calculated between each node of the corresponding logical shard is counted, ranked from high to low, and the top 30% of the nodes are associated with edges to construct a node update correlation degree topology graph.

[0048] Optionally, in one example of the above aspect, according to the data change event stream and the node update correlation degree topology graph, the update priority of each node in the edge data center is calculated, including the following steps:

[0049] According to the constructed node update correlation degree topology graph and the sum of the update correlation degrees calculated between each node of the corresponding logical shard, the weight wij of the associated edge between the nodes is calculated, wherein,

[0050] rij is the sum of the update correlation degrees calculated between the nodes, and rmax is the maximum value of the sum of the update correlation degrees calculated between each node in the update correlation degree topology graph;

[0051] According to the node update correlation degree topology graph, the weighted degree centrality of node i is calculated ; wherein Aij is an element of the adjacency matrix A, if node i and node j establish an associated edge, then Aij=1, otherwise 0, j∈(1,2, …, N), N is the total number of associated edges established by node i;

[0052] Scan the Docker container of node i using Trivy, count the number and severity of security vulnerabilities, and calculate the data security score ; wherein Vi is the set of vulnerabilities discovered by Trivy scanning of node i, wseverity(v) is the severity weight of vulnerability v, and wtype(v) is the type weight of vulnerability;

[0053] According to the obtained weighted degree centrality of the node and the node data security score, the update priority of each node in the edge data center is calculated.

[0054] Optionally, in one example of the above aspect, according to the obtained weighted degree centrality of the node and the node data security score, the update priority of each node in the edge data center is calculated, including the following steps:

[0055] The obtained weighted degree centrality Si and node data security score Si of the node are normalized;

[0056] According to whether the node sends a priority update request for the data change event sk, the time sensitivity function T(sk) of the data change event sk is set:

[0057] ;

[0058] In combination with the normalized weighted degree centrality Di' and node data security score Si' of the node, and the time sensitivity function T(sk), the update priority coefficient Pi of each node in the edge data center is calculated:

[0059] Pi=γ1*Si′+γ2*Di′+γ3*T(sk);

[0060] wherein γ1, γ2 and γ3 are weight coefficients.

[0061] Optionally, in one example of the above aspect, the update request is routed to the nearest high update priority node using Anycast IP, and after the high update priority nodes synchronize the update request with each other, the update request is synchronized to other nodes, including the following steps:

[0062] According to the update priority coefficient of each node in the edge data center, the update priority coefficients of the nodes are sorted from high to low, and the nodes with high update priority are selected as high update priority nodes;

[0063] All high update priority nodes announce the same Anycast IP through BGP, and the router selects the nearest high update priority node according to the AS_PATH length;

[0064] After the high update priority nodes are synchronized with each other through gRPC Stream, the high update priority nodes publish update events through MQTT / Kafka, and other nodes subscribe and pull to synchronize the update request to other nodes.

[0065] Compared with the prior art, the present application has the following advantages:

[0066] The high-frequency update data of the present application is deployed in the edge node close to the user, reducing the round-trip time of the cloud and improving the user experience; avoiding frequent transmission of high-frequency data across data centers, reducing the bandwidth pressure of the core network, and combining the Raft+PBFT hybrid consensus mechanism to perform global verification on all logical shards, ensuring the update consistency and security of node data.

[0067] The present application adjusts in real time through the topology graph with business changes, adapts to data model evolution; combines the correlation degree, data importance and node load to calculate the priority, and uses Anycast IP to route the update request to the nearest high update priority node, synchronizes the update request to other nodes after the high update priority nodes are synchronized with each other, ensures that the high update priority node quickly obtains the update request, and at the same time, realizes low-latency, high-availability and strong-consistency update propagation, while taking into account security and business continuity. When actually deployed, the parameters need to be adjusted according to the network size and business requirements. BRIEF DESCRIPTION OF DRAWINGS

[0068] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0069] Fig. 1 The figure is a schematic diagram of the system framework of the present application.

[0070] Fig. 2 The figure is a schematic diagram of the logical shard data management method flow of the edge management layer of the present application. DETAILED DESCRIPTION

[0071] The technical solutions of the present application will be described in detail below in conjunction with the embodiments. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0072] Please refer to Figs. 1-2The first aspect of the present application provides a cloud computing-based enterprise data update management system, comprising:

[0073] An edge management layer:

[0074] According to the region code of the enterprise data, the enterprise data is divided into a plurality of logical shards, and an edge data center is set in a region corresponding to the region code of each logical shard, and the logical shard data in the edge data center is jointly managed by a plurality of master nodes and auxiliary nodes;

[0075] The logical shard data is divided into high-frequency update data and ordinary update data, and the high-frequency update data and the ordinary update data are deployed on the nodes close to the user in the edge data center;

[0076] Raft is used for intra-shard fast election (to solve the brain split problem); PBFT (practical Byzantine fault tolerance) is used for cross-shard global verification (to resist Byzantine attacks).

[0077] A Raft+PBFT hybrid consensus mechanism is adopted to perform global verification on all logical shards, monitor the running state of the master nodes of the edge data center, perform dynamic master node switching, and trigger automatic switching of the auxiliary nodes through heartbeat detection and blockchain log rollback;

[0078] A cloud update request management layer:

[0079] A change data capture module is deployed in the cloud computing environment, real-time monitoring of the database logs of each logical shard is performed, a data change event stream of each logical shard is generated, the update correlation degree between nodes is analyzed, and a node-to-node update correlation degree topology graph is constructed;

[0080] According to the data change event stream and the node-to-node update correlation degree topology graph, the update priority of each node in the edge data center is calculated, the update request is routed to the nearest high update priority node using Anycast IP, and after the update requests between the high update priority nodes are synchronized, the update requests are synchronized to other nodes.

[0081] Specifically, in the present embodiment, the high-frequency update data is deployed on the edge nodes close to the user, reducing the round-trip time of the cloud and improving the user experience; avoiding frequent cross-data center transmission of high-frequency data, reducing the bandwidth pressure of the core network. The single master election and log replication mechanism of Raft is suitable for high-frequency data, ensuring low-latency writing (usually <100ms). Compared with PBFT, Raft has lightweight communication and is suitable for edge node resource limited scenarios; PBFT is used for ordinary update data (such as financial records and configuration information), tolerating partial node malpractice or failure; ensuring that all edge nodes reach an agreement on the modification of key data, avoiding data bifurcation.

[0082] The secondary node detects the survival state of the primary node through heartbeat, and triggers election if it is timed out. The blockchain stores all operation logs, and the new primary node can roll back to the latest consistent state to avoid data loss. The election limit of Raft can be used to allow only the nodes that get a certain number of votes to become the primary node, realizing the preliminary setting of the primary node. At the same time, the Raft+PBFT hybrid consensus mechanism is used to globally check all logical shards, ensuring the consistency and security of node data updates; Raft ensures strong consistency among edge nodes, but only in a non-Berlin environment. PBFT prevents malicious nodes from tampering through digital signature and three-phase commit (Pre-Prepare / Prepare / Commit).

[0083] All update operations can also be recorded in the blockchain to provide an unalterable audit trail. Regulatory agencies can verify data integrity to meet compliance requirements.

[0084] By analyzing database transaction logs, data changes can be captured without modifying application code, avoiding performance interference with business systems; and the changed data is converted into a standardized event stream (such as JSON / Protobuf format) for easy subsequent processing and transmission. Through Anycast technology, update requests are automatically routed to the nearest edge node based on user geographic location or network topology, reducing transmission delay. Traditional DNS polling or load balancing may cause delay fluctuations due to network congestion, while Anycast provides deterministic low latency. If the nearest high-update-priority node is overloaded, Anycast can automatically distribute requests to other high-update-priority nodes to avoid single-point overload.

[0085] By analyzing historical change events, the correlation strength between data updates among nodes (such as Pearson correlation coefficient, mutual information) is calculated. If the frequent updates of node A (user information) trigger cascading updates of node B (order information), the correlation between A→B is high.

[0086] The topology graph is adjusted in real time as the business changes to adapt to data model evolution; the priority is calculated based on correlation, data importance, and node load (CPU / memory / bandwidth). Raft / PBFT hybrid consensus is used to ensure data consistency among nodes and avoid split-brain, and low-priority nodes receive updates asynchronously through a message queue (such as Kafka) to tolerate short delays but achieve eventual consistency.

[0087] Through real-time adjustment of the topology graph with business changes, data model evolution is adapted; priority is calculated comprehensively in combination with correlation degree, data importance, and node load, and meanwhile, an update request is routed to a nearest high update priority node using Anycast IP, after which the high update priority nodes are synchronized with the update request, and the update request is synchronized to other nodes, so as to ensure that the high update priority nodes quickly acquire the update request, and meanwhile, low-delay, high-availability, and strong-consistency update propagation is implemented, while security and business continuity are taken into account. When actually deployed, parameters need to be adjusted according to network scale and business requirements.

[0088] In one of the embodiments of the present application, according to the region code of enterprise data, the enterprise data is divided into several logical shards, and in the edge data center corresponding to the region code of each logical shard, the logical shard data is managed by several master nodes and auxiliary nodes, including the following steps:

[0089] According to the region code of enterprise data, the shard strategy is selected, the enterprise data is allocated to different logical shards according to the geographical area level of the code prefix of the enterprise data, such as the postal code and the administrative division code, the region code is calculated by hash, the maximum threshold of the data volume of each logical shard is set, and the logical shard exceeding the threshold is split;

[0090] According to the code level, the enterprise data is divided into several logical shards, and the region code is calculated by hash.

[0091] Provincial shard: divided according to the first two digits of the code (such as 11-Beijing);

[0092] City-level shard: divided according to the first four digits (such as 1101-Beijing municipal area);

[0093] In the region corresponding to each logical shard, an edge data center is deployed, and several master nodes and auxiliary nodes are set in the region, and the edge data center manages the master nodes and auxiliary nodes to process the logical shard data.

[0094] The master node is used to process all write requests of the corresponding logical shard data, and the write request is allocated to the corresponding node, and the master node uses a rewritable non-volatile memory such as 3D XPoint, which supports high-frequency read and write.

[0095] The auxiliary node stores persistent data and uses LSM-Tree log-structured merge tree to manage data, and periodically changes data according to write requests.

[0096] In this embodiment, when the shard capacity is expanded, the trigger condition is set: when the data volume of a certain shard exceeds the threshold (such as 100 GB) or the QPS is continuously too high, the split is automatically triggered.

[0097] Split process: suspend the write operation of the shard. Reallocate data according to the new shard strategy (such as hash modulo).

[0098] Synchronously update the routing table to direct part of the requests to the new shard.

[0099] In one embodiment of the present application, the data of each logical shard is divided into high-frequency update data and ordinary update data, and the high-frequency update data and the ordinary update data are deployed in the nodes close to the users in the edge data center, including the following steps:

[0100] According to the data update log, for the data of each logical shard, the update frequency of the data of the logical shard within a unit time is counted to obtain the update frequency; dynamic threshold setting is performed, and the average value of the top 30% highest update frequencies globally is set as the high-frequency update threshold;

[0101] Data with an update frequency greater than or equal to the high-frequency update threshold is divided into high-frequency update data, and the remaining data is divided into ordinary update data;

[0102] The high-frequency update data is deployed in the lightweight data replica of the master node close to the user in the edge data center, and the ordinary update data is deployed in the lightweight data replica of the auxiliary node close to the user;

[0103] Randomly select a distribution master node as a unified write entry, and route all write requests to the distribution master node first;

[0104] The distribution master node synchronizes the update request of the high-frequency update data to the high-frequency storage area of the local and corresponding other master nodes;

[0105] The distribution master node asynchronously forwards the ordinary update request to the corresponding auxiliary node, and the auxiliary node updates the storage area data according to the ordinary update request;

[0106] The master node pressure control is performed, and the number of concurrent update requests of a single master node Chot is set as α*CPU core number of the master node, wherein α is an empirical coefficient, and in the present embodiment, the value is 0.8, and the flow control is triggered when the number of concurrent requests exceeds;

[0107] The auxiliary node receives the update request, and lists the update request in the message queue of the auxiliary node, and consumes according to the priority.

[0108] In one embodiment of the present application, a Raft+PBFT hybrid consensus mechanism is used to perform global verification on all logical shards, including the following steps:

[0109] The master node in each logical shard initiates a global verification request periodically, the update request of the data in the master node adopts Raft consensus, and the update request of the data in the auxiliary node adopts PBFT consensus;

[0110] The Raft consensus is used for detection, and a consistency coefficient of the master node data is calculated: PRaft=(N0-1) / 2-Nf, wherein N0 is the total amount of updated data of the master node, and Nf is the amount of data in the master node whose update delay exceeds a threshold value;

[0111] The PBFT consensus is used for detection, and a security coefficient of the secondary node data is calculated: Ppbft=(2f+1) / N1; wherein f is the amount of data in the secondary node that appears an error in the update process, and N1 is the total amount of updated data of the secondary node;

[0112] According to the check results of the logical shards, global checking is performed on all logical shards, and whether the consistency coefficient of the master node data and the security coefficient of the secondary node data in each logical shard exceed a threshold value is verified, if yes, the global checking is completed, otherwise, the nodes corresponding to the threshold values are marked;

[0113] The marked nodes are detected again, if the detection result again exceeds the threshold value, the global checking is completed, otherwise, for the marked nodes, if the master node, the secondary node is downgraded, if the secondary node, the corresponding node is added again and the data is transferred, and the global checking is performed again until the global checking is passed.

[0114] In the embodiment, the consistency of the master node data and the security of the secondary node data in the logical shard are collected from a large number of fault nodes, the corresponding consistency coefficient and security coefficient are calculated, and the average value is taken as the threshold value of the consistency coefficient and the security coefficient.

[0115] In one embodiment of the present application, the running state of the master node of the edge data center is monitored, dynamic master node switching is performed, and the secondary node automatically switches through heartbeat detection and blockchain log rollback triggering, including the following steps:

[0116] The secondary node periodically and randomly sends a heartbeat request to the master node, and marks it as abnormal if it does not respond within a timeout period;

[0117] The secondary node periodically and randomly requests the usage rate, memory occupation and network bandwidth utilization rate data from the master node, and dynamically calculates the node health degree through a formula:

[0118] HealthScore=β1*(1-CPU usage rate)+β2*(1-memory occupation)+β3*network bandwidth utilization rate; wherein HealthScore is the node health degree, β1, β2 and β3 are weight coefficients (in the embodiment, β1=0.5, β2=0.3, β3=0.2),

[0119] The secondary node periodically checks the blockchain log submitted by the master node, and starts a dispute resolution process if a fork or invalid block is found;

[0120] Switching trigger condition is set:

[0121] Heartbeat detection fails continuously for k times (in this embodiment, k = 3).

[0122] Health score is continuously lower than a threshold (in this embodiment, the health score threshold is set to 0.3) for T time (in this embodiment, T = 5 minutes);

[0123] The blockchain log rollback detects that the primary node is malicious;

[0124] If the primary node meets the switching trigger condition, the corresponding primary node is switched, the secondary node elects a new primary node through the Raft algorithm, and enters the candidate state after meeting the quorum;

[0125] The new primary node synchronizes data from the switched primary node, pulls the latest blockchain log, executes the view change protocol of the PBFT consensus, updates the switched primary node cluster configuration and broadcasts it to all switched primary nodes, the new primary node starts processing write requests, and the original primary node is downgraded to a secondary node.

[0126] In one embodiment of the present application, a change data capture module is deployed in a cloud computing environment to listen to the database logs of each logical shard in real time, generate data change event streams for each logical shard, analyze the update correlation degree between each node, and construct a node update correlation degree topology graph, including the following steps:

[0127] In a cloud computing environment, a change data capture module is deployed by analyzing database transaction logs (such as MySQL binlog, MongoDB oplog, and PostgreSQL WAL), an independent open source CDC change data capture tool is deployed for each logical shard to capture data changes in the database logs of each logical shard;

[0128] Tool selection: Debezium: an open source CDC tool that supports Kafka as a message middleware and is compatible with mainstream databases such as MySQL and PostgreSQL. Amazon DocumentDB Change Stream: a cloud-native service that directly captures change events from MongoDB-compatible databases and supports DDL events (such as creating indexes and sharding). Google CloudDatastream: a serverless CDC service that supports real-time replication from Oracle and MySQL to BigQuery, simplifying the configuration process.

[0129] The data change event stream structure is set, each change event of the corresponding logical shard contains an operation type, a timestamp, a primary key, pre-change / post-change data (such as the updateDescription field of MongoDB), the change events are grouped to form a set, and a data change event stream corresponding to the logical shard is generated;

[0130] According to the data change event stream data corresponding to the logical shard and the consistency coefficient of the primary node data and the security coefficient of the secondary node data in the logical shard, the update correlation degree between each node is analyzed, and a node-to-node update correlation degree topology graph is constructed.

[0131] In one embodiment of the present application, according to the data change event stream data corresponding to the logical shard and the consistency coefficient of the primary node data and the security coefficient of the secondary node data in the logical shard, the update correlation degree between each node is analyzed, and a node-to-node update correlation degree topology graph is constructed, including the following steps:

[0132] For each primary node in the logical shard, according to the data change event stream data corresponding to the logical shard and the consistency coefficient of the primary node data in the logical shard, the update correlation degree between each primary node is analyzed:

[0133]

[0134] Wherein, Raij is the update correlation degree between the primary nodes, Mh0 is the number of common change events between the primary nodes, Ms0 is the product of the total number of change events between the primary nodes, Zaij is the difference between the consistency coefficients of the primary nodes, Sxij is the average value of the consistency coefficients of the primary nodes in the logical shard, is a time decay coefficient;

[0135] For each secondary node in the logical shard, according to the data change event stream data corresponding to the logical shard and the security coefficient of the secondary node data, the update correlation degree between each secondary node is analyzed:

[0136]

[0137] Wherein, Rbij is the update correlation degree between the secondary nodes, Ah0 is the number of common change events between the secondary nodes, As0 is the product of the total number of change events between the secondary nodes, Zxij is the difference between the security coefficients of the secondary nodes, Sxij is the average value of the security coefficients of the secondary nodes in the logical shard;

[0138] According to the data change event stream data corresponding to the logical shard, the update correlation degree between each node is analyzed:

[0139]

[0140] Wherein, Rij is the updated correlation degree between nodes i and j, eik is the count of the kth type of change event of node i in the time window T, and N is the total number of change event types;

[0141] The time decay coefficient is calculated by the following formula:

[0142]

[0143] Wherein, Atij is the time difference (in seconds) of the last common change of nodes i and j, and λ is the time decay rate; (such as λ = 0.01, which means that the correlation degree is halved every 100 seconds).

[0144] The sum of the updated correlation degrees calculated between the nodes of the corresponding logical shard is counted, and ranked from high to low, and the nodes ranked in the top 30% are established to build the updated correlation degree topology graph between nodes.

[0145] In one embodiment of the present application, according to the data change event stream and the updated correlation degree topology graph between nodes, the update priority of each node in the edge data center is calculated, including the following steps:

[0146] According to the constructed updated correlation degree topology graph between nodes and the sum of the updated correlation degrees calculated between the nodes of the corresponding logical shard, the weight wij of the associated edge between nodes is calculated, wherein rij is the sum of the updated correlation degrees calculated between the nodes, and rmax is the maximum value of the sum of the updated correlation degrees calculated between the nodes in the updated correlation degree topology graph;

[0147] According to the updated correlation degree topology graph between nodes, the weighted degree centrality of node i is calculated ; Wherein, Aij is an element of the adjacency matrix A, if nodes i and j establish an associated edge, then Aij = 1, otherwise 0, j ∈ (1, 2, …, N), and N is the total number of associated edges established by node i;

[0148] The Trivy scans the Docker container of node i, counts the number and severity of security vulnerabilities, and calculates the data security score ; Wherein, Vi is the vulnerability set found by Trivy scanning of node i, wseverity(v) is the severity weight of vulnerability v (such as Critical = 3, High = 2, Medium = 1, Low = 0.5), and wtype(v) is the vulnerability type weight (such as RCE vulnerability weight is higher);

[0149] According to the obtained weighted degree centrality of nodes and node data security score, the update priority of each node in the edge data center is calculated.

[0150] In one embodiment of the present application, the update priority of each node in the edge data center is calculated according to the weighted degree centrality and the node data security score of the obtained node, including the following steps:

[0151] The weighted degree centrality Si and the node data security score Si of the obtained node are normalized.

[0152] According to whether the node sends a priority update request for the data change event sk, the time sensitivity function T(sk) of the data change event sk is set:

[0153]

[0154] The update priority coefficient Pi of each node in the edge data center is calculated by combining the normalized weighted degree centrality Di' and the node data security score Si' of the node, and the time sensitivity function T(sk):

[0155] Pi = γ1*Si' + γ2*Di' + γ3*T(sk);

[0156] Wherein, γ1, γ2 and γ3 are weight coefficients, controlling the relative importance of security and business relevance, in the present embodiment, γ1, γ2 and γ3 are set to 0.4, 0.3 and 0.3 respectively.

[0157] In one embodiment of the present application, Anycast IP is used to route the update request to the nearest high update priority node, and after the high update priority nodes synchronize the update request with each other, the update request is synchronized to other nodes, including the following steps:

[0158] According to the update priority coefficient of each node in the edge data center, the update priority coefficients of the nodes are sorted from high to low, and the nodes with high update priority are selected as high update priority nodes.

[0159] All high update priority nodes announce the same Anycast IP (such as 198.51.100.1) through BGP, and the router selects the nearest high update priority node according to the AS_PATH length.

[0160] After the high update priority nodes synchronize the update request with each other through gRPC Stream, the high update priority nodes publish the update event through MQTT / Kafka, and other nodes subscribe and pull, so as to synchronize the update request to other nodes.

[0161] ​The above examples are only used to illustrate the technical method of the present application but not limit the present application. Although the present application is described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present application can be modified or equivalently replaced without departing from the spirit and scope of the technical method of the present application.

Claims

1. A cloud computing-based enterprise data update management system, characterized by, The application relates to an edge management layer, a cloud update request management layer and a data center management layer. According to the region code of enterprise data, the enterprise data is divided into several logical fragments, and an edge data center is arranged in a region corresponding to the region code of each logical fragment; the logical fragment data in the edge data center is managed by a plurality of master nodes and auxiliary nodes; the logical fragment data is divided into high-frequency update data and ordinary update data, and the high-frequency update data and the ordinary update data are arranged in the nodes close to the users in the edge data center; a Raft+PBFT hybrid consensus mechanism is adopted to perform global checking on all the logical fragments, to monitor the running state of the master nodes in the edge data center, to perform dynamic master node switching, and to trigger automatic switching of the auxiliary nodes through heartbeat detection and blockchain log rollback; The cloud update request management layer comprises the following steps. In the cloud computing environment, a change data capture module is arranged, the database logs of the logical fragments are monitored in real time, the data change event streams of the logical fragments are generated, the update correlation degrees among the nodes are analyzed, and a node-to-node update correlation degree topology graph is constructed. According to the data change event streams and the node-to-node update correlation degree topology graph, the update priorities of the nodes in the edge data center are calculated, the update request is routed to the nearest high update priority node by using an Anycast IP, the update requests are synchronized among the high update priority nodes, and then the update requests are synchronized to other nodes. In the cloud computing environment, a change data capture module is arranged, the database logs of the logical fragments are monitored in real time, the data change event streams of the logical fragments are generated, the update correlation degrees among the nodes are analyzed, and a node-to-node update correlation degree topology graph is constructed. In the cloud computing environment, a change data capture module is arranged, the database logs of the logical fragments are monitored in real time, the data change event streams of the logical fragments are generated, the update correlation degrees among the nodes are analyzed, and a node-to-node update correlation degree topology graph is constructed. For the master nodes in each logical fragment, the update correlation degrees among the master nodes are analyzed according to the data change event stream data of the corresponding logical fragment and the consistency coefficient of the master node data in the logical fragment. For the auxiliary nodes in each logical fragment, the update correlation degrees among the auxiliary nodes are analyzed according to the data change event stream data of the corresponding logical fragment and the security coefficient of the auxiliary node data. ​ ​ ​ ​ ​ Wherein, Raij is the update correlation degree between the master nodes, Mh0 is the common change event number between the master nodes, Ms0 is the product of the total change event number between the master nodes, Zaij is the difference of the consistency coefficients between the master nodes, Saij is the average of the consistency coefficients of the master nodes in the logical shard, is a time decay coefficient; ​ Wherein, Rbij is the update correlation between the secondary nodes, Ah0 is the common change event number between the secondary nodes, As0 is the product of the total change event number between the secondary nodes, Zxij is the difference between the security coefficients between the secondary nodes, Sxij is the average value of the security coefficients of the secondary nodes in the logical fragment; According to the corresponding logical fragment data change event stream data, the update correlation between each node is analyzed: Wherein, Rij is the update correlation between nodes i and j, eik is the count of the kth type of change event of node i in the time window T, and N is the total number of change event types; The time decay coefficient is calculated by the following formula: Wherein, Δtij is the time difference of the last common change of nodes i and j, and λ is the time decay rate; The sum of the update correlations calculated between each node in the corresponding logical fragment is counted, and ranked from high to low. The top 30% of the nodes are associated with edges to construct the update correlation topology between nodes.

2. The cloud computing based enterprise data update management system as claimed in claim 1, wherein, According to the region code of enterprise data, the enterprise data is divided into several logical fragments, and the edge data center is set in the region corresponding to the region code of each logical fragment. The logical fragment data in the edge data center is managed by several master nodes and secondary nodes, including the following steps: According to the region code of enterprise data, set the sharding strategy selection, distribute the enterprise data to different logical fragments according to the geographical area level of the code prefix of the enterprise data, hash calculate the region code, set the maximum threshold of the data volume of each logical fragment, and split the logical fragment that exceeds the threshold; Deploy edge data centers in the region corresponding to each logical fragment, and set several master nodes and secondary nodes in the region. The edge data center manages the master nodes and secondary nodes to process the logical fragment data. The master node is used to process all write requests of the corresponding logical fragment data, and distributes the write requests to the corresponding nodes. The master node uses rewritable non-volatile memory and supports high-frequency read and write. The secondary node stores persistent data and uses LSM-Tree log-structured merge tree to manage data, and periodically changes data according to write requests.

3. The cloud computing based enterprise data update management system as claimed in claim 1, wherein, Divide the data of each logical fragment into high-frequency update data and ordinary update data, and deploy the high-frequency update data and ordinary update data near the user's node in the edge data center, including the following steps: According to the data update log, for the data of each logical fragment, the update frequency of the logical fragment data in unit time is counted to obtain the update frequency; dynamic threshold setting is performed, and the average value of the top 30% highest update frequency in the whole is set as the high-frequency update threshold; The data with an update frequency greater than or equal to the high-frequency update threshold is divided into high-frequency update data, and the remaining data is divided into ordinary update data; Deploy a lightweight data copy of the high-frequency update data near the user's master node in the edge data center, and deploy a lightweight data copy of the ordinary update data near the user's secondary node; Randomly select a distribution master node as a unified write entrance, and route all write requests to the distribution master node first; The distribution master node synchronizes the update request of the high-frequency update data to the local and the high-frequency storage area of the corresponding other master nodes. The distribution master node forwards the normal update request to the corresponding secondary node asynchronously, and the secondary node updates the storage area data according to the normal update request; Perform master node pressure control, set the number of concurrent single master node update requests Node CPU core number, where α is an empirical coefficient, and the number of concurrent requests triggers flow control; The secondary node receives the update request, adds the update request to the message queue of the secondary node, and consumes it according to the priority.

4. The cloud computing based enterprise data update management system as claimed in claim 1, wherein, A mixed consensus mechanism of Raft+PBFT is adopted to perform global verification on all logical shards, including the following steps: The master node in each logical shard initiates a global verification request periodically, and the update request of the data in the master node adopts Raft consensus, and the update request of the data in the secondary node adopts PBFT consensus; Raft consensus is used for detection, and the consistency coefficient of the master node data is calculated: PRaft=(N0-1) / 2-Nf, wherein N0 is the total amount of updated data of the master node, and Nf is the amount of data whose update delay exceeds the threshold value in the master node; PBFT consensus is used for detection, and the security coefficient of the secondary node data is calculated: Ppbft=(2f+1) / N1; wherein f is the amount of data that has an error in the update process in the secondary node, and N1 is the total amount of updated data of the secondary node; According to the verification results returned by the nodes of each logical shard, global verification is performed on all logical shards to verify whether the consistency coefficient of the master node data and the security coefficient of the secondary node data in each logical shard exceed the threshold value, if yes, the global verification is completed, otherwise, the nodes corresponding to the threshold values are marked; The marked nodes are detected again, if the detection result exceeds the threshold value again, the global verification is completed, otherwise, for the marked nodes, if they are master nodes, they are downgraded to secondary nodes, if they are secondary nodes, they are revoked, and then the corresponding nodes are added again, the data is transferred, and the global verification is performed again until the global verification is passed.

5. The cloud computing based enterprise data update management system as claimed in claim 1, wherein, The running state of the master node of the edge data center is monitored, dynamic master node switching is performed, and the secondary node triggers automatic switching through heartbeat detection and blockchain log rollback, including the following steps: The secondary node periodically sends a heartbeat request to the master node at random, and marks it as abnormal if it does not respond within a timeout period; The secondary node periodically requests the usage rate, memory occupation and network bandwidth utilization rate data of the master node at random, and dynamically calculates the node health degree through a formula: network bandwidth utilization rate; wherein, HealthScore is the node health degree, β1, β2 and β3 are weight coefficients, The secondary node periodically verifies the blockchain log submitted by the master node, and starts the dispute resolution process if a fork or invalid block is found; Set the switching trigger conditions: k times of continuous failure of heartbeat detection; the health score is continuously lower than the threshold value for more than T time; the master node is detected to be malicious through blockchain log rollback; If the master node meets the switching trigger condition, the corresponding master node is switched, the secondary node elects a new master node through the Raft algorithm, and enters the candidate state after meeting the quorum; The new master node synchronizes data from the switched master node, pulls the latest blockchain log, executes the view change protocol of PBFT consensus, updates the switched master node cluster configuration and broadcasts it to all switched master nodes, and the new master node starts processing the write request, and the original master node is downgraded to a secondary node.

6. The cloud computing based enterprise data update management system as claimed in claim 1, wherein, According to the data change event stream and the update correlation degree topology between nodes, the update priority of each node in the edge data center is calculated, including the following steps: According to the constructed inter-node update correlation degree topology graph and the sum of the calculated update correlation degrees between the nodes of each logical shard, the weight wij of the correlation edge between the nodes is calculated as wij=rij / rmax, wherein rij is the sum of the calculated update correlation degrees between the nodes, and rmax is the maximum value of the sum of the calculated update correlation degrees between the nodes in the update correlation degree topology graph; According to the inter-node updating correlation degree topology map, the weighted degree centrality Di of the node i is calculated ; wherein, Aij is an element of the adjacency matrix A, if the node i and the node j establish a correlation edge, then Aij=1, otherwise 0, j∈(1,2,…,N), N is the total number of correlation edges established by the node i; Scan the Docker container of the node i using Trivy, count the number and severity of security vulnerabilities, and calculate the data security score Si ; wherein Vi is the set of vulnerabilities discovered by the Trivy scan of node i, wseverity(v) is the severity weight of vulnerability v, and wtype(v) is the vulnerability type weight. According to the obtained weighted degree centrality of the nodes and the node data security score, the update priority of each node in the edge data center is calculated.

7. The cloud computing based enterprise data update management system as claimed in claim 6, wherein, According to the obtained weighted degree centrality of the nodes and the node data security score, the update priority of each node in the edge data center is calculated, including the following steps: The obtained weighted degree centrality Si of the nodes and the node data security score Si are normalized; According to whether the node sends a priority update request for the data change event sk, a time sensitivity function T(sk) of the data change event sk is set: T(sk) = T (sk) - T (s0) ; In combination with the normalized weighted degree centrality Di' of the nodes and the node data security score Si' and the time sensitivity function T(sk), the update priority coefficient Pi of each node in the edge data center is calculated: ； Wherein, γ1, γ2 and γ3 are weight coefficients.

8. The cloud computing based enterprise data update management system as claimed in claim 1, wherein, The update request is routed to the nearest high update priority node using Anycast IP, and after the high update priority nodes synchronize the update request with each other, the update request is synchronized to other nodes, including the following steps: According to the update priority coefficient of each node in the edge data center, the update priority coefficients of the nodes are sorted from high to low, and the nodes are used as high update priority nodes; All high update priority nodes announce the same Anycast IP through BGP, and the router selects the nearest high update priority node according to the AS_PATH length; After the high update priority nodes synchronize the update request with each other through gRPC Stream, the high update priority nodes publish the update event through MQTT / Kafka, and other nodes subscribe and pull, so as to synchronize the update request to other nodes.

Images