Distributed database synchronization method, computer device, medium and product

Abstract

The embodiment of the invention discloses a distributed database synchronization method, a computer device, a medium and a product. The distributed database synchronization method comprises the following steps: determining to-be-synchronized information in a source database; the to-be-synchronized information comprises a plurality of nodes and data table information corresponding to each node; capturing data change information of the to-be-synchronized information in a preset period; the data change information comprises a data change frequency and a query frequency in a data table corresponding to each node; determining business influence based on the data change frequency and the query frequency; dynamically adjusting the data synchronization priority of the corresponding node based on the service influence and the data change information; and executing a data synchronization task based on the dynamically adjusted data synchronization priority. According to the method, data synchronization in different environments can be met, the flexibility is high, the universality is good, the synchronization efficiency is high, the accuracy is high, and the timeliness and the consistency of the data in the distributed environment can be realized.

Description

Technical Field

[0001] The present disclosure relates to the technical field of data synchronization, and in particular to a distributed database synchronization method, computer device, medium, and product. Background Art

[0002] In current Internet and IoT applications, data synchronization across distributed databases has become critical to ensuring system performance and data consistency. However, traditional database synchronization methods often lack the flexibility to adapt to the frequency and varying importance of data changes. Different types of data have varying priorities within a business, and existing technologies often handle synchronization tasks statically, failing to adjust priorities in real time based on actual business needs.

[0003] When data changes frequently or the data volume is huge, traditional synchronization solutions are prone to excessive synchronization load, which not only consumes system resources but also causes response delays, thus affecting overall system performance. Many existing synchronization solutions rely on data change logs to obtain change information. Due to the differences in logging mechanisms and implementation methods of different databases, the effectiveness of these solutions in different environments is limited and their universality is poor. In scenarios with large data volumes and insufficient resource conditions, traditional synchronization methods often cannot ensure timely and efficient synchronization, further affecting the real-time nature of the business and data consistency. Summary of the Invention

[0004] In view of this, the embodiments of the present disclosure provide a distributed database synchronization method, computer device, medium, and product, which can solve the problems of low data synchronization efficiency when the data volume is large and inapplicable to data synchronization of different databases in the methods disclosed in the prior art.

[0005] In a first aspect, an embodiment of the present disclosure provides a distributed database synchronization method, which specifically includes the following contents:

[0006] Determine the information to be synchronized in the source database; the information to be synchronized includes several nodes and data table information corresponding to each node;

[0007] Capturing data change information of the information to be synchronized within a preset period; the data change information includes the data change frequency and query frequency in the data table corresponding to each node;

[0008] Determining business influence based on the data change frequency and the query frequency;

[0009] Dynamically adjust the data synchronization priority of the corresponding node based on the business influence and the data change information;

[0010] Execute data synchronization tasks based on the dynamically adjusted data synchronization priority.

[0011] Optionally, determining the business influence based on the data change frequency and the query frequency includes:

[0012] Determining the data change frequency, the change weight of the query frequency, and the query weight respectively;

[0013] Determining the data change influence based on the data change frequency, the change weight, and a preset change threshold;

[0014] Determining the query influence based on the query frequency, query weight, and a preset query threshold;

[0015] Based on the data change influence and the query influence, the business influence is obtained.

[0016] Optionally, the business influence is 1 b :I b =I1+I2;

[0017] I1=(F c / A0)*V1;

[0018] YY+242266P

[0019] I2=(F q / B0)*V2;

[0020] Among them, I1 is the data change influence, I2 is the query influence, F c is the data change frequency, A0 is the preset change threshold, V1 is the change weight, F q is the query frequency, B0 is the preset query threshold, and V2 is the query weight.

[0021] Optionally, dynamically adjusting the data synchronization priority of the corresponding node based on the business influence and the data change information includes:

[0022] Analyze the data change information and the business influence based on a preset scoring algorithm to obtain scoring information of each node in the source database;

[0023] S=w1×F c +w2×F q +w3×I b ; Where S is the final score in the scoring information, w1, w2, and w3 are the weights of each factor;

[0024] Dynamically adjust the data synchronization priority of the corresponding node based on the scoring information;

[0025] The data synchronization priority of the corresponding node is dynamically adjusted based on the scoring information.

[0026] Optionally, dynamically adjusting the data synchronization priority of the corresponding node based on the scoring information includes:

[0027] Based on the scoring information, obtaining the node with the highest score as the target node;

[0028] If the priority of the target node is not the highest, dynamically adjust the task synchronization priority of the target node to the highest level.

[0029] Optionally, dynamically adjusting the data synchronization priority of the corresponding node based on the scoring information includes:

[0030] Based on the rating information, obtaining a rating ranking from largest to smallest;

[0031] It is determined whether the score ranking is consistent with the preset task synchronization order. If not, the preset task synchronization order is dynamically adjusted according to the score ranking, and the adjusted preset task synchronization order is used as the current data synchronization priority.

[0032] Optionally, the method further includes:

[0033] Obtain resource usage of each node in real time; the resource usage includes CPU load, memory usage or network bandwidth;

[0034] Obtaining a resource usage rate of each node based on the resource usage;

[0035] If the resource usage exceeds a preset threshold, mark the node whose resource usage exceeds the preset threshold as a bottleneck node;

[0036] Determine whether the priority of the bottleneck node is first based on the adjusted data synchronization priority; if so, migrate unexecuted tasks in the bottleneck node to the node with the lowest resource utilization rate according to the task scheduler, and prioritize processing on the node with the lowest resource utilization rate;

[0037] If not, the data synchronization task is executed according to the dynamically adjusted data synchronization priority.

[0038] Optionally, the method further includes:

[0039] Real-time monitoring of the number of nodes in the information to be synchronized;

[0040] When the number of nodes changes, the data synchronization priorities of the corresponding nodes are dynamically adjusted.

[0041] Optionally, dynamically adjusting the data synchronization priority of corresponding nodes when the number of nodes changes includes:

[0042] Get the updated node list;

[0043] Obtain the data change frequency and query frequency in the data table corresponding to each node in the update node list;

[0044] Determining business influence based on the data change frequency and the query frequency;

[0045] Based on the business influence and the data change information, the data synchronization priority of the corresponding node is dynamically adjusted.

[0046] Optionally, the method further includes:

[0047] When an abnormality occurs in a node in the information to be synchronized, a fault tolerance mechanism is triggered;

[0048] Automatically adjust tasks based on the fault-tolerant mechanism and reallocate tasks of abnormal nodes to other normal nodes.

[0049] Optionally, triggering the execution of the fault tolerance mechanism includes:

[0050] Determine the node where the abnormality occurs and record it as an abnormal node;

[0051] Obtain the tasks being executed and unfinished on the abnormal node, and mark the tasks to be reallocated;

[0052] Identify healthy nodes with the lowest load and highest resource availability;

[0053] Allocating the task to be reallocated to the normal node;

[0054] The automatic adjustment of tasks based on the fault-tolerant mechanism to reallocate the tasks of the abnormal node to other normal nodes includes: obtaining the priorities of the abnormal node and the normal node based on the dynamically adjusted data synchronization priority; if the priority of the abnormal node is higher than the priority of the normal node, the task to be reallocated is executed preferentially on the normal node.

[0055] In a second aspect, an embodiment of the present disclosure further provides a distributed database synchronization system, including:

[0056] A determination module is used to determine the information to be synchronized in the source database; the information to be synchronized includes several nodes and data table information corresponding to each node;

[0057] The capture module is used to capture the data change information of the information to be synchronized within a preset period; the data change information includes the data change frequency in the data table corresponding to each node, the query YY+242266P

[0058] Inquiry frequency;

[0059] An acquisition module, configured to determine business influence based on the data change frequency and the query frequency;

[0060] A dynamic adjustment module, configured to dynamically adjust the data synchronization priority of a corresponding node based on the business influence and the data change information;

[0061] Execute data synchronization tasks based on the dynamically adjusted data synchronization priority.

[0062] In a third aspect, the embodiments of the present disclosure further provide a computer device that adopts the following technical solution:

[0063] The computer device comprises:

[0064] at least one processor; and,

[0065] a memory communicatively connected to the at least one processor; wherein,

[0066] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform any of the above-mentioned distributed database synchronization methods.

[0067] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute any of the above-mentioned distributed database synchronization methods.

[0068] In a fifth aspect, an embodiment of the present disclosure further provides a computer program product, comprising a computer program / instruction, which implements the steps of any of the above methods when executed by a processor.

[0069] The distributed database synchronization method disclosed in the present application adapts to different business needs by dynamically adjusting data synchronization priorities, avoids resource waste and performance bottlenecks by prioritizing key data, adopts diversified acquisition methods to adapt to different database characteristics, improves overall synchronization efficiency by prioritizing synchronization of key data, is not limited to a fixed method for obtaining change information, and improves the flexibility of the solution; through these improvements, the solution of the present application can effectively improve the efficiency, accuracy and adaptability of distributed database synchronization, and ensure the timeliness and consistency of data in a distributed environment.

[0070] The above description is only an overview of the technical solution of the present disclosure. In order to more clearly understand the technical means of the present disclosure, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present disclosure more obvious and easy to understand, the following specifically cites preferred embodiments and describes them in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0072] Figure 1 A flowchart of a distributed database synchronization method provided in an embodiment of the present disclosure.

[0073] Figure 2 A flowchart of a method for obtaining business influence provided in an embodiment of the present disclosure.

[0074] Figure 3 A flowchart of a method for dynamically adjusting the data synchronization priority of corresponding nodes based on business influence and data change information provided by an embodiment of the present disclosure.

[0075] Figure 4 for Figure 3 FIG. 1 is a flow chart of a first embodiment of a method for dynamically adjusting the data synchronization priority of a corresponding node based on scoring information.

[0076] Figure 5 for Figure 3 FIG. 4 is a flow chart of a second embodiment of a method for dynamically adjusting the data synchronization priority of a corresponding node based on scoring information.

[0077] Figure 6 A flowchart of a method for processing a bottleneck node provided in an embodiment of the present disclosure.

[0078] Figure 7 A flowchart of a method for dynamically adjusting the number of nodes when the number of nodes changes according to an embodiment of the present disclosure is provided.

[0079] Figure 8 for Figure 7 A flowchart of a method for dynamically adjusting the data synchronization priority of corresponding nodes when the number of nodes changes.

[0080] Figure 9 A flowchart of a dynamic adjustment method when an abnormality occurs in a node in the information to be synchronized provided by an embodiment of the present disclosure.

[0081] Figure 10 A flowchart of a method for triggering the execution of a fault tolerance mechanism provided in an embodiment of the present disclosure.

[0082] Figure 11 A flowchart of a method for reallocating tasks of abnormal nodes provided in an embodiment of the present disclosure.

[0083] Figure 12 A schematic diagram of the structure of a computer device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0084] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0085] It should be clear that the following embodiments of the present disclosure are described through specific concrete examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other in the absence of conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0086] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on this disclosure, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.

[0087] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The illustrations only show components related to the present disclosure and are not drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component can be changed at will, and the component layout type may also be more complicated.

[0088] Additionally, in the following description, specific details are provided to provide a thorough understanding of the examples. However, one skilled in the art will appreciate that the aspects described can be practiced without these specific details.

[0089] Reference Figure 1 , this application discloses a distributed database synchronization method, specifically including:

[0090] S100, determining information to be synchronized in a source database; the information to be synchronized includes several nodes and data table information corresponding to each node.

[0091] In this embodiment, the source database may include multiple, each source database may include multiple nodes, each node has its own corresponding data table information; wherein the data table information is not limited to the table structure, index, data storage capacity, etc.

[0092] Furthermore, a list of information to be synchronized may be generated, recording basic information of all nodes and corresponding data table information.

[0093] S200, capturing data change information of information to be synchronized within a preset period; the data change information includes data change frequency and query frequency in the data table corresponding to each node.

[0094] The data change frequency refers to the number of addition, deletion, and modification operations recorded within a preset period (for example, every minute or every hour), which can be achieved through database triggers, logs, or data change event capture technology.

[0095] Query frequency refers to the number of query requests recorded within a preset period, which can be achieved through database query logs or performance monitoring tools.

[0096] Furthermore, the data changes and query information captured by the monitoring agent can be stored in a centralized data warehouse for subsequent analysis.

[0097] As for the data format, each record may include a timestamp, a node ID, a data table name, an operation type (add, delete, modify, query), and the number of operations.

[0098] S300 determines business impact based on data change frequency and query frequency.

[0099] specifically refer to Figure 2 , the specific methods for obtaining business influence include:

[0100] S310, respectively determining the data change frequency, the change weight of the query frequency, and the query weight.

[0101] Specifically, weights can be flexibly set based on specific business needs and data characteristics to ensure that the calculation of business influence is more in line with actual business scenarios.

[0102] S320: Determine the influence of the data change based on the data change frequency, change weight, and preset change threshold.

[0103] Specifically, the data change influence is I1: I1=(F c / A0)*V1; where Fc is the data change frequency, A0 is the preset change threshold, and V1 is the change weight.

[0104] By presetting change thresholds, the data change frequency of different nodes can be standardized to facilitate comparison and calculation; by changing weights, data changes that have a greater impact on the business can be highlighted, ensuring that the influence of key data is reflected in the business impact calculation.

[0105] S330: Determine the query influence based on the query frequency, query weight, and preset query threshold.

[0106] Specifically, the query influence is I2: I2 = (F q / B0)*V2; where F q is the query frequency, B0 is the preset query threshold, and V2 is the query weight.

[0107] S340, based on data change influence and query influence, obtains the business YY+242266P of each node

[0108] influence.

[0109] Specifically, the business impact is I b :I b =I1+I2.

[0110] In this step, by comprehensively considering the impact of data changes and queries, the importance of each node to the business can be fully evaluated.

[0111] In this embodiment, the method for obtaining business influence disclosed in S310-S340 can flexibly set weights according to specific business needs and data characteristics to ensure that the calculation of business influence is more in line with actual business scenarios; by presetting change thresholds and query thresholds, the data change frequency and query frequency of different nodes are standardized to facilitate comparison and calculation; by comprehensively considering the influence of data changes and queries, the importance of each node to the business can be comprehensively evaluated to ensure that the synchronization priority of key data is higher; business influence is used as the basis for dynamically adjusting data synchronization priority to ensure that the synchronization priority of key data is higher, thereby improving overall synchronization efficiency and business responsiveness.

[0112] For example, a node has 200 data changes and 1500 query requests within an hour. The change frequency weight is 0.6 and the query frequency weight is 0.4. Then:

[0113] Change frequency score = (200 / 100)*0.6 = 1.2*0.6 = 0.72;

[0114] Query frequency score = (1500 / 1000)*0.4 = 1.5*0.4 = 0.6;

[0115] Total score = 0.72 + 0.6 = 1.32. The higher the score, the greater the influence.

[0116] S400 dynamically adjusts the data synchronization priority of corresponding nodes based on business influence and data change information;

[0117] Execute data synchronization tasks based on the dynamically adjusted data synchronization priority.

[0118] In this embodiment, by dynamically adjusting the priority, it is ensured that limited resources (such as network bandwidth, CPU, storage space) are used first for the most important data synchronization tasks, and the synchronization frequency of data that has a great impact on the business is prioritized, while the synchronization frequency of data that is not important to the business is reduced, thereby improving YY+242266P

[0119] Overall synchronization efficiency.

[0120] The distributed database synchronization method disclosed in the present application ensures that all nodes and data tables that need to be synchronized are taken into consideration by determining the information to be synchronized in the source database, thereby avoiding missing important data; by capturing the data change information of the information to be synchronized within a preset period, it can timely capture the latest changes in the data and ensure the timeliness of the synchronized data; by determining the business influence of each node, it can identify which data is more important to the business, providing a basis for subsequent priority adjustment; based on the business influence and data change information, the data synchronization priority of the corresponding node is dynamically adjusted, that is, the priority is dynamically adjusted according to the actual information, ensuring that limited resources (such as network bandwidth, CPU, storage space) are used first for the most important data synchronization tasks, thereby improving resource utilization efficiency.

[0121] Reference Figure 3 , a method for dynamically adjusting the data synchronization priority of corresponding nodes based on business influence and data change information, specifically including:

[0122] S410: Analyze data change information and business influence based on a preset scoring algorithm to obtain scoring information of each node in the source database.

[0123] Among them, the scoring information is the final score S of each node:

[0124] S=w1×F c +w2×F q +w3×I b Among them, F c is the data change frequency, F q is the query frequency, I b is the business influence, w1, w2, and w3 are the weights of each factor.

[0125] By comprehensively considering the frequency of data changes, query frequency, and business impact, we can comprehensively assess the importance of each node to the business; by setting different weights, we can flexibly adjust the scoring criteria according to business needs to ensure that the scoring results are more in line with actual business scenarios.

[0126] S420: Dynamically adjust the data synchronization priority of the corresponding node based on the scoring information.

[0127] By dynamically adjusting priorities, critical data is synchronized first, avoiding resource waste and improving synchronization efficiency. Synchronization strategies can be dynamically adjusted according to business needs to ensure that data synchronization always keeps pace with business development. By prioritizing data that has a greater impact on the business, resource waste is avoided and resource utilization efficiency is improved.

[0128] Furthermore, the dynamic priority adjustment mechanism disclosed in this embodiment, which assigns different importance scores to different data items based on data change frequency, query frequency and business impact, and dynamically adjusts the priority of data synchronization tasks, can provide more resources to high-priority items and synchronize key data first, thereby improving the efficiency and response speed of the overall system.

[0129] Reference Figure 4 In a first embodiment, a method for dynamically adjusting the data synchronization priority of a corresponding node based on scoring information specifically includes:

[0130] A110, based on the scoring information, obtains the node with the highest score as the target node;

[0131] A120 , determining whether the priority of the target node in the preset task synchronization sequence is the highest priority; if not, dynamically adjusting the task synchronization priority of the target node to the optimal level; if so, executing based on the preset task synchronization sequence.

[0132] In this embodiment, the method of dynamically adjusting the data synchronization priority of the corresponding node is to dynamically adjust the task synchronization priority of the target node to the optimal level when the priority of the target node in the preset task synchronization order is not the highest priority, and obtain a new task synchronization order, that is, only the one with the highest score is taken into account, and the others are temporarily ignored.

[0133] The solution disclosed in this embodiment ensures that the nodes with the highest scores (i.e., the nodes with the greatest impact on the business) are synchronized first, ensuring the timeliness and consistency of key data. It can dynamically adjust the task synchronization priority according to business needs and scoring results to ensure that the nodes with the highest scores are synchronized first. It only focuses on the nodes with the highest scores and ignores the priority adjustment of other nodes, simplifies the operation process, and improves execution efficiency. By giving priority to the nodes with the highest scores, it avoids resource waste and improves resource utilization efficiency.

[0134] Reference Figure 5 In a second embodiment, a method for dynamically adjusting the data synchronization priority of a corresponding node based on the scoring information specifically includes:

[0135] A210, based on the rating information, obtains the rating sorting from largest to smallest;

[0136] A220 determines whether the score ranking is consistent with the preset task synchronization order. If not, dynamically adjust the preset task synchronization order according to the score ranking, and use the adjusted preset task synchronization order as the current data synchronization priority.

[0137] If the score ranking is inconsistent with the preset task synchronization order, the system dynamically adjusts the preset task synchronization order according to the score ranking and uses it as the current data synchronization priority to ensure that the adjusted preset task synchronization order is consistent with the score ranking, which can fully reflect the impact of each node on the business and ensure that nodes with high scores are synchronized first. At the same time, the task synchronization priority can be dynamically adjusted according to business needs and scoring results to ensure that the synchronization strategy is always consistent with business needs.

[0138] In this embodiment, by sorting all nodes from large to small, the impact of each node on the business can be comprehensively evaluated to ensure that nodes with high scores have higher priority; the adjusted preset task synchronization order is ensured to be consistent with the score ranking, which can comprehensively reflect the impact of each node on the business and ensure that nodes with high scores are synchronized first; the task synchronization priority can be dynamically adjusted according to business needs and scoring results to ensure that the synchronization strategy is always consistent with business needs; through the adjusted preset task synchronization order, key nodes are ensured to be synchronized first, thereby improving overall synchronization efficiency and business responsiveness; by giving priority to nodes with high scores, resource waste is avoided and resource utilization efficiency is improved.

[0139] Reference Figure 6 The distributed database synchronization method disclosed in this application also includes a method for processing bottleneck nodes, which specifically includes the following contents:

[0140] Obtain resource usage of each node in real time; resource usage includes CPU load, memory usage or network bandwidth;

[0141] Get the resource usage of each node based on resource usage;

[0142] If the resource usage exceeds the preset threshold, the node with the resource usage exceeding the preset threshold is marked as a bottleneck node;

[0143] Based on the adjusted data synchronization priority, determine whether the bottleneck node has the highest priority. If so, migrate the unexecuted tasks in the bottleneck node to the node with the lowest resource utilization rate according to the task scheduler and process them first on the node with the lowest resource utilization rate.

[0144] If not (ie, if the resource usage does not exceed the preset threshold), the data synchronization task is executed based on the dynamically adjusted data synchronization priority.

[0145] The resource utilization rate refers to the resource utilization rate of any one of the three: CPU load, memory usage, and network bandwidth.

[0146] In this step, if the resource usage of any one of the three, CPU load, memory usage, and network bandwidth, exceeds a preset threshold, the corresponding node is marked as a bottleneck node.

[0147] Specifically, the system administrator sets a preset threshold for resource usage based on business requirements and node capabilities.

[0148] Among them, priority is given to the node with the lowest resource utilization. It can be executed in priority order first. When it reaches the node with the lowest resource utilization, the assigned tasks are executed first, and then the tasks of the node itself are executed. You can also ignore the priority order and execute the highest priority tasks assigned to the node with the lowest resource utilization first, and then execute others in priority order.

[0149] The task scheduler can be a Kubernetes scheduler, Apache Mesos, or Hadoop YARN.

[0150] In this embodiment, by real-time monitoring of the resource usage of each node, resource bottlenecks can be discovered in time to avoid degradation of system performance; by quantifying the resource usage rate, the resource usage of each node can be evaluated more intuitively, facilitating the identification of bottleneck nodes; by setting preset thresholds, bottleneck nodes with excessively high resource usage can be identified in time to prevent degradation of system performance; by migrating the tasks of the bottleneck node to the node with the lowest resource usage rate, load balancing can be effectively achieved to avoid overloading of a single node, and the tasks of the bottleneck node are prioritized on the node with the lowest resource usage rate to ensure the timely completion of critical tasks. The task execution strategy can be dynamically adjusted according to the resource usage of the node and the data synchronization priority to ensure system performance and business continuity.

[0151] Reference Figure 7 The distributed database synchronization method disclosed in this application also includes: a dynamic adjustment method when the number of nodes changes, specifically including the following steps:

[0152] B100, monitors the number of nodes in the information to be synchronized in real time.

[0153] By monitoring the number of nodes in real time, changes in the number of nodes can be discovered in a timely manner to ensure timely adjustment of the synchronization strategy.

[0154] Specifically, you can use service discovery tools (such as Consul, etcd, Zookeeper) to monitor changes in nodes. When a node joins or exits, the service discovery tool will notify the system.

[0155] In this application, the node will periodically send a heartbeat signal to the central management node. If a node does not send a heartbeat signal within a predetermined time, the central management node considers that the node has exited.

[0156] B200, when the number of nodes changes, dynamically adjusts the data synchronization priority of the corresponding nodes.

[0157] In this step, the data synchronization priority can be dynamically adjusted according to the changes in the number of nodes to ensure that the synchronization strategy for adding or reducing nodes can respond to business changes in a timely manner; by dynamically adjusting the priority, synchronization resources can be allocated more reasonably, ensuring priority synchronization of key nodes, avoiding resource waste, and ensuring that the data synchronization strategy can be adjusted in a timely manner when the number of nodes changes, thereby ensuring business continuity and data consistency.

[0158] Reference Figure 8 ,When the number of nodes changes, the method of dynamically adjusting the data synchronization priority of the corresponding nodes specifically includes:

[0159] B210, obtain the updated node list.

[0160] Specifically, when it is monitored that the number of nodes in the information to be synchronized changes, an updated node list is obtained, that is, the newly added or reduced node information is extracted from the information to be synchronized to form an updated node list.

[0161] In this step, the node list is updated according to the latest node status, including adding new nodes and removing failed nodes.

[0162] Timely updates can ensure that the system can obtain the latest status of nodes in a timely manner, providing an accurate data basis for subsequent priority adjustments.

[0163] B220, obtain the data change YY+242266P in the data table corresponding to each node in the updated node list

[0164] Frequency, query frequency.

[0165] Obtaining the data change frequency for each node in the updated node list reflects the data changes in the data table. Obtaining the query frequency for each node in the updated node list reflects the access frequency of the data in the data table. By obtaining the data change frequency and query frequency, you can comprehensively analyze the business activity and data changes of each node.

[0166] B230 determines business impact based on data change frequency and query frequency.

[0167] Specifically, the business impact is I b :

[0168] I b =I1+I2=(F c / A0)*V1+(F q / B0)*V2.

[0169] B240 dynamically adjusts the data synchronization priority of corresponding nodes based on business influence and data change information.

[0170] Dynamic adjustment method reference Figure 4 、 Figure 5 There are two different embodiments, so they will not be described here.

[0171] The method disclosed in this embodiment ensures that the system can obtain the latest status of the node in a timely manner, providing an accurate data basis for subsequent priority adjustment; by obtaining data change frequency and query frequency, it can comprehensively analyze the business activity and data changes of each node; by quantifying business influence, it can more intuitively evaluate the importance of each node to the business, providing a scientific basis for priority adjustment; it can dynamically adjust the data synchronization priority according to the business influence and data change information of the node, ensuring priority synchronization of key nodes; through reasonable priority adjustment, it ensures that data synchronization tasks can be executed efficiently, avoids resource waste, and improves synchronization efficiency; ensures that the data synchronization strategy can be adjusted in time when the number of nodes changes, ensuring business continuity and data consistency.

[0172] Reference Figure 9 The distributed database synchronization method disclosed in this application also includes a dynamic adjustment method when an abnormality occurs in a node in the information to be synchronized, specifically including:

[0173] C100, when an exception occurs in a node in the information to be synchronized, the fault tolerance mechanism is triggered.

[0174] Specific reference Figure 10 , the methods for triggering the execution of the fault tolerance mechanism include:

[0175] C110, determine the node where the abnormality occurs and record it as an abnormal node.

[0176] The system monitors all nodes in the information to be synchronized in real time to detect whether any nodes have abnormal conditions, such as network failure, hardware failure, software error, etc.; when a node is detected to be abnormal, the system marks it as an abnormal node and records the identification information of the abnormal node; by accurately identifying abnormal nodes, it can quickly locate the problem node and provide accurate targets for subsequent task reallocation.

[0177] C120 obtains the tasks being executed and unfinished on the abnormal node and marks the tasks to be reallocated.

[0178] Specifically, the system queries the currently executing and unfinished tasks on the abnormal node, obtaining detailed information about these tasks, including task type, status, and priority. These tasks are then marked as pending tasks for reallocation, allowing for subsequent task reallocation. By obtaining and marking tasks on the abnormal node, the system ensures that these tasks are not lost due to the node anomaly, maintaining task continuity and integrity.

[0179] C130,determines the healthy node with the lowest load and highest resource availability.

[0180] Specifically, evaluate the load of all normal nodes, including CPU load, memory load, network load, etc., and find the node with the lowest load; evaluate the resource availability of all normal nodes, including CPU availability, memory availability, network bandwidth availability, etc., and find the node with the highest resource availability; by determining the node with the lowest load, load balancing can be effectively achieved, overloading of a single node can be avoided, and the overall performance and stability of the system can be improved; by determining the node with the highest resource availability, it can ensure that the task is executed on the node with the most resources, thereby improving the efficiency of task execution.

[0181] C140 allocates the tasks to be reallocated to normal nodes.

[0182] Specifically, the task to be reallocated is allocated to the normal node determined in step C130. The task allocation method can be reallocated according to the existing priority or readjusted according to the new priority. The task to be reallocated is migrated to the normal node to ensure that the task can continue to be executed and avoid task interruption or loss.

[0183] By allocating tasks to be reallocated to normal nodes, we ensure that tasks can continue to be executed and maintain the continuity and integrity of tasks; through task reallocation, we can effectively achieve load balancing, avoid overloading of a single node, and improve the overall performance and stability of the system; by allocating tasks to nodes with the highest resource availability, we can ensure that tasks are executed on nodes with the most abundant resources, thereby improving task execution efficiency.

[0184] C200 automatically adjusts tasks based on the fault-tolerant mechanism and reallocates tasks of abnormal nodes to other normal nodes.

[0185] Specific reference Figure 11 ,The methods for reallocating tasks of abnormal nodes include:

[0186] C210 obtains the priorities of abnormal nodes and normal nodes based on the dynamically adjusted data synchronization priority.

[0187] By obtaining the priority information of nodes, a scientific basis can be provided for task reallocation, ensuring the rationality and effectiveness of task allocation.

[0188] C220: If the priority of the abnormal node is higher than that of the normal node, the task to be reallocated will be executed on the normal node first.

[0189] Compare the priorities of abnormal nodes and normal nodes. If the abnormal node has a higher priority than the normal node, the task to be reallocated is executed first. Prioritize the reallocated tasks on normal nodes to ensure that high-priority tasks are completed in a timely manner. By prioritizing high-priority tasks, critical business needs can be met in a timely manner, improving system responsiveness and business continuity.

[0190] C230, if the priority of the abnormal node is not higher than the priority of the normal node, the task to be reallocated is executed after the preset task is executed on the normal node, where the preset task is a task pre-allocated to the normal node.

[0191] Compare the priorities of abnormal nodes and normal nodes. If the abnormal node's priority is not higher than that of normal nodes, the scheduled tasks are executed on the normal nodes before the tasks to be reallocated are executed. The tasks to be reallocated are queued and executed in order of priority after the scheduled tasks are completed on the normal nodes. By rationally sorting the execution order of tasks, tasks are executed according to priority, avoiding task conflicts and resource waste. This ensures that scheduled tasks on normal nodes are completed first, maintaining business continuity and stability, and preventing business interruptions caused by task reallocation.

[0192] Furthermore, the distributed database synchronization method disclosed in the present application also includes displaying the system operation status and manually adjusting the synchronization strategy through a user operation interface, which is convenient for management personnel to operate and monitor.

[0193] Traditional database synchronization methods handle synchronization tasks in a static manner and are unable to adjust priorities in real time according to actual business needs. The distributed database synchronization method disclosed in this application can determine business impact based on data change frequency and query frequency, and dynamically adjust data synchronization priority. This allows data of different importance to obtain different synchronization priorities according to actual business needs, thereby achieving flexible data synchronization management, that is, being able to adjust synchronization strategies in real time according to business needs, ensuring that key data is synchronized first, and improving overall data synchronization efficiency and business responsiveness.

[0194] When data changes frequently or the data volume is huge, traditional synchronization solutions are prone to excessive synchronization load, consuming system resources and causing response delays. The distributed database synchronization method disclosed in this application can prioritize the synchronization of data that has a large impact on the business and reduce the synchronization frequency of data that is not important to the business, thereby effectively reducing the synchronization load, dynamically adjusting the synchronization priority, and ensuring that limited resources are used first for the most important data synchronization tasks, avoiding resource waste and performance bottlenecks. That is, by prioritizing the processing of key data, resource waste is avoided, resource utilization efficiency is improved, unnecessary synchronization tasks are reduced, system load is reduced, and overall system performance and response speed are improved.

[0195] Many existing synchronization solutions rely on data change logs to obtain change information, which results in limited effectiveness of these solutions in different environments and poor universality. The distributed database synchronization method disclosed in this application can not only obtain change information through data change logs, but also use triggers, monitoring tools and other methods. That is, the capture of data change information does not rely on the log mechanism of a specific database, which effectively improves the universality of the solution. That is, through diversified acquisition methods, it can adapt to the characteristics and log mechanisms of different databases, thereby improving the effectiveness and universality of the solution in various environments.

[0196] When the amount of data is large and resource conditions are insufficient, traditional synchronization methods often cannot ensure timely and efficient synchronization, affecting the real-time nature of the business and data consistency. The distributed database synchronization method disclosed in this application can prioritize the synchronization of data that has a greater impact on the business through priority adjustment, reduce unnecessary synchronization tasks, and improve overall synchronization efficiency. That is, by prioritizing the processing of key data, unnecessary synchronization tasks are reduced, and the timeliness and consistency of data in a distributed environment are ensured.

[0197] The distributed database synchronization method disclosed in the present application adapts to different business needs by dynamically adjusting data synchronization priorities, avoids resource waste and performance bottlenecks by prioritizing key data, adopts diversified acquisition methods to adapt to different database characteristics, improves overall synchronization efficiency by prioritizing synchronization of key data, is not limited to a fixed method for obtaining change information, and improves the flexibility of the solution; through these improvements, the solution of the present application can effectively improve the efficiency, accuracy and adaptability of distributed database synchronization, and ensure the timeliness and consistency of data in a distributed environment.

[0198] The distributed database synchronization method disclosed in this application is suitable for application scenarios that require efficient and flexible data synchronization, such as large-scale cloud services, financial transaction systems, social network data processing, etc. It can intelligently adjust the synchronization strategy according to the real-time importance of the data, which not only ensures the real-time nature of key business data, but also optimizes the overall resource utilization efficiency.

[0199] In a second aspect, an embodiment of the present disclosure further provides a distributed database synchronization system, including:

[0200] A determination module is used to determine the information to be synchronized in the source database; the information to be synchronized includes several nodes and the data table information corresponding to each node;

[0201] The capture module is used to capture the data change information of the information to be synchronized within a preset period; the data change information includes the data change frequency and query frequency in the data table corresponding to each node;

[0202] The acquisition module is used to determine business impact based on data change frequency and query frequency;

[0203] Dynamic adjustment module, used to dynamically adjust the data synchronization priority of corresponding nodes based on business influence and data change information;

[0204] Execute data synchronization tasks based on the dynamically adjusted data synchronization priority.

[0205] It should be noted that the methods in the distributed database synchronization method disclosed in the first aspect of the embodiment of the present disclosure are all applicable to the acquisition of corresponding solutions in the distributed database synchronization system disclosed in the second aspect of the present application, so they will not be repeated here.

[0206] It should be noted that the methods in the distributed database synchronization method disclosed in the first aspect of the embodiment of the present disclosure are all applicable to the acquisition of corresponding solutions in the distributed database synchronization system disclosed in the second aspect of the present application, so they will not be repeated here.

[0207] The computer device according to an embodiment of the present disclosure includes a memory and a processor. The memory is used to store non-transitory computer-readable instructions. Specifically, the memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache). The non-volatile memory may include, for example, read-only memory (ROM), a hard disk, a flash memory, etc.

[0208] The processor may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the computer device to perform desired functions. In one embodiment of the present disclosure, the processor is configured to execute the computer-readable instructions stored in the memory, causing the computer device to execute all or part of the steps of the distributed database synchronization method described in each embodiment of the present disclosure.

[0209] Those skilled in the art should understand that in order to solve the technical problem of how to obtain a good user experience, this embodiment may also include well-known structures such as a communication bus and an interface, and these well-known structures should also be included in the scope of protection of this disclosure.

[0210] like Figure 12 The present invention provides a schematic diagram of the structure of a computer device according to an embodiment of the present invention. Figure 12 The computer device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0211] like Figure 12 As shown, the computer device may include a processor (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) or a program loaded from a storage device into a random access memory (RAM). Various programs and data required for the operation of the computer device are also stored in the RAM. The processor, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0212] Typically, the following devices can be connected to the I / O interface: input devices such as sensors or visual information acquisition devices; output devices such as display screens; storage devices such as tapes and hard disks; and communication devices. The communication device can allow the computer device to communicate with other devices (such as edge computing devices) wirelessly or by wire to exchange data. Figure 12A computer device having various devices is shown, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0213] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processor, all or part of the steps of the distributed database synchronization method of the embodiment of the present disclosure are performed.

[0214] For detailed description of this embodiment, please refer to the corresponding description in the aforementioned embodiments, which will not be repeated here.

[0215] According to the computer-readable storage medium of the embodiment of the present disclosure, non-transitory computer-readable instructions are stored thereon. When the non-transitory computer-readable instructions are executed by a processor, all or part of the steps of the distributed database synchronization method of each embodiment of the present disclosure are executed.

[0216] The above-mentioned computer-readable storage media include, but are not limited to, optical storage media (e.g., CD-ROMs and DVDs), magneto-optical storage media (e.g., MOs), magnetic storage media (e.g., magnetic tapes or mobile hard disks), media with built-in rewritable non-volatile memory (e.g., memory cards), and media with built-in ROM (e.g., ROM cartridges).

[0217] For detailed description of this embodiment, please refer to the corresponding description in the aforementioned embodiments, which will not be repeated here.

[0218] The basic principles of the present disclosure have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in this disclosure are merely illustrative and not restrictive, and should not be construed as necessarily possessed by each embodiment of the present disclosure. Furthermore, the specific details disclosed above are provided for illustrative purposes and to facilitate understanding, rather than as limitations. These details do not limit the present disclosure to necessarily being implemented using these specific details.

[0219] In the present disclosure, relational terms such as first and second, etc. are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. The block diagrams of the devices, devices, equipment, and systems involved in the present disclosure are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, or configured in any manner. Words such as "including," "comprising," "having," and the like are open-ended words, meaning "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or" and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0220] Additionally, as used herein, "or" used in a list of items beginning with "at least one" indicates a separate list, so that, for example, a list of "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not mean that the example described is preferred or better than other examples.

[0221] It should also be noted that in the system and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0222] Various changes, substitutions, and modifications may be made to the technology described herein without departing from the teachings defined by the appended claims. Moreover, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, compositions of things, means, methods, and actions described above. Currently existing or later developed processes, machines, manufactures, compositions of things, means, methods, or actions that perform substantially the same function or achieve substantially the same results as the corresponding aspects described herein may be utilized. Accordingly, the appended claims include within their scope such processes, machines, manufactures, compositions of things, means, methods, or actions.

[0223] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0224] The above description has been provided for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A distributed database synchronization method, characterized in that: include: Determine the information to be synchronized in the source database; The information to be synchronized includes several nodes and data table information corresponding to each node; Capturing data change information of the information to be synchronized within a preset period; the data change information includes the data change frequency and query frequency in the data table corresponding to each node; Determine the business influence based on the data change frequency and the query frequency; Dynamically adjust the data synchronization priority of the corresponding node based on the business influence and the data change information; The data synchronization task is executed based on the dynamically adjusted data synchronization priority.

2. The distributed database synchronization method according to claim 1, characterized in that: The determining of the business influence based on the data change frequency and the query frequency includes: Determine the data change frequency, the change weight of the query frequency, and the query weight respectively; Determine the data change influence based on the data change frequency, the change weight, and a preset change threshold; Determine the query influence based on the query frequency, query weight, and preset query threshold; Based on the data change influence and the query influence, the business influence is obtained.

3. The distributed database synchronization method according to claim 2, characterized in that: The business impact is I b :I b =I1+I2; I1=(F c / A0)*V1; <h2 style=";text-align:left;direction:ltr">I2=(F<h2 style=";text-align:left;direction:ltr"> q <h2 style=";text-align:left;direction:ltr"> / B0)*V2; Where I1 is the data change influence, I2 is the query influence, and F c is the data change frequency, A0 is the preset change threshold, V1 is the change weight, F q is the query frequency, B0 is the preset query threshold, and V2 is the query weight.

4. The distributed database synchronization method according to claim 3, characterized in that: The dynamically adjusting the data synchronization priority of the corresponding node based on the business influence and the data change information includes: Analyze the data change information and the business influence based on a preset scoring algorithm to obtain scoring information of each node in the source database; S=w1×F c +w2×F q +w3×I b ; Among them, S is the final score in the scoring information, w1, w2, and w3 are the weights of each factor; The data synchronization priority of the corresponding node is dynamically adjusted based on the scoring information.

5. The distributed database synchronization method according to claim 4, characterized in that: The dynamically adjusting the data synchronization priority of the corresponding node based on the scoring information includes: Based on the scoring information, obtaining the node with the highest score as the target node; If the priority of the target node is not the highest, dynamically adjust the task synchronization priority of the target node to the highest level.

6. The distributed database synchronization method according to claim 4, characterized in that: The dynamically adjusting the data synchronization priority of the corresponding node based on the scoring information includes: Based on the rating information, obtain a rating ranking from largest to smallest; It is determined whether the score ranking is consistent with the preset task synchronization order. If not, the preset task synchronization order is dynamically adjusted according to the score ranking, and the adjusted preset task synchronization order is used as the current data synchronization priority.

7. The distributed database synchronization method according to claim 1, characterized in that: The method further includes: Obtain resource usage of each node in real time; the resource usage includes CPU load, memory usage or network bandwidth; Obtaining a resource usage rate of each node based on the resource usage; If the resource usage rate exceeds a preset threshold, mark the node whose resource usage rate exceeds the preset threshold as a bottleneck node; Based on the adjusted data synchronization priority, determine whether the priority of the bottleneck node is the first, and if so, migrate the unexecuted tasks in the bottleneck node to the node with the lowest resource utilization rate according to the task scheduler, and give priority to processing on the node with the lowest resource utilization rate; If not, the data synchronization task is executed according to the dynamically adjusted data synchronization priority.

8. The distributed database synchronization method according to claim 1, characterized in that: The method further includes: Real-time monitoring of the number of nodes in the information to be synchronized; When the number of nodes changes, the data synchronization priorities of the corresponding nodes are dynamically adjusted.

9. The distributed database synchronization method according to claim 8, characterized in that: When the number of nodes changes, dynamically adjusting the data synchronization priority of the corresponding nodes includes: Get the updated node list; Get the data change frequency YY+242266P in the data table corresponding to each node in the update node list rate, query frequency; Determine the business influence based on the data change frequency and the query frequency; Based on the business influence and the data change information, the data synchronization priority of the corresponding node is dynamically adjusted.

10. The distributed database synchronization method according to claim 1, characterized in that: The method further includes: When an abnormality occurs in a node in the information to be synchronized, a fault tolerance mechanism is triggered; The tasks are automatically adjusted based on the fault-tolerant mechanism, and the tasks of abnormal nodes are reallocated to other normal nodes.

11. The distributed database synchronization method according to claim 10, characterized in that: The triggering execution fault tolerance mechanism includes: Determine the node where the abnormality occurs and record it as an abnormal node; Obtaining the tasks being executed and unfinished on the abnormal node, and marking the tasks to be reallocated; Identify healthy nodes with the lowest load and highest resource availability; Allocating the task to be reallocated to the normal node; The automatic adjustment of tasks based on the fault-tolerant mechanism to reallocate the tasks of the abnormal nodes to other normal nodes includes: obtaining the priorities of the abnormal nodes and the normal nodes based on the dynamically adjusted data synchronization priorities; if the priority of the abnormal node is higher than that of the normal node, preferentially executing the tasks to be reallocated on the normal nodes.

12. A computer device, characterized in that: The computer device includes: YY+242266P at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the distributed database synchronization method described in any one of claims 1-11.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the distributed database synchronization method described in any one of claims 1-11.

14. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images