A method and system for synchronizing cyclone working data based on distributed database

By dynamically integrating data nodes in a distributed database, calculating adaptive factors based on load volume and read load, and integrating cached data to determine synchronization trigger locks, the problem of unbalanced data nodes in cyclone working data synchronization is solved, and the problem of unbalanced data nodes is improved Stability and efficiency of data synchronization.

CN118467640BActive Publication Date: 2024-11-05WEIHAI SHANGPIN MASCH EQUIP TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410722316.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-06-05
Publication Date
2024-11-05
Estimated Expiration
2044-06-05

AI Technical Summary

Technical Problem

In the prior art, when the working data of the cyclone is synchronized, fixed data nodes lead to unbalanced load of the data node, resulting in waste of performance, and affecting the stability and efficiency of data synchronization.

Method used

By dynamically integrating data nodes in a distributed database, the adaptive load factor is calculated based on the storage load amount of each node and the reading load entropy, the cache hit data is integrated, the synchronization trigger lock is determined, the data increment synchronization is realized, and the data nodes are dynamically managed. load.

Benefits of technology

It improves the stability and efficiency of the working data synchronization of the cyclone, optimizes the load balancing and performance performance of the distributed database system, and avoids idleness and overload of data nodes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118467640B_ABST
    Figure CN118467640B_ABST
Patent Text Reader

Abstract

The present application provides a method and system for synchronizing cyclone working data based on a distributed database, which determines the storage load of each data node based on the synchronous storage data; extracts the synchronous read data from the node operation information, and then determines the read load entropy of each data node through the synchronous read data and the network load of each data node, and determines the adaptive load factor of each data node according to the read load entropy of each data node and the corresponding storage load value; integrates all data nodes through cache hit data and each adaptive load factor to obtain a synchronization node domain; when using the data nodes in the synchronization node domain for data synchronization, the synchronization trigger lock of each data node in the synchronization node domain is used as the data lock of the corresponding trigger in the distributed database for data incremental synchronization. Based on the above scheme, dynamic adaptive integration of data nodes used for cyclone working data synchronization in a distributed database can be achieved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of data synchronization, and more specifically, to a method and system for synchronizing cyclone working data based on a distributed database. Background Art

[0002] Data synchronization refers to the process of maintaining data consistency between multiple data sources in a distributed system. Data synchronization covers a series of methods and strategies. Data synchronization aims to ensure the accuracy, integrity and real-time performance of data. The development of data synchronization technology is closely related to distributed computing, cloud computing, big data analysis and other fields. It has a wide range of applications, involving data backup, replication, sharing, migration and other aspects.

[0003] The cyclone working data synchronization method involves data collection, storage and synchronization processes. The cyclone operating data, such as flow data, pressure data, etc., are collected through sensors. The cyclone working data is stored in a distributed storage system, such as a Hadoop database. The data synchronization of the cyclone working data adopts a data increment synchronization strategy, and the latest data is regularly transmitted to the target system to ensure data accuracy and real-time cyclone working data. In the prior art, the cyclone working data is usually synchronized by a fixed data node to synchronize the database. When a large number of cyclones are working, the fixed data node method will cause the data node for data synchronization to be too heavy during the data synchronization process, while other data nodes are in an idle state. The above method will cause the waste of the idle data node performance, thereby affecting the stability and efficiency of data synchronization. The dynamic integration of data nodes that need to be synchronized can effectively reduce the load of a single data node, thereby balancing the load of each data node and improving the stability of data synchronization. Therefore, how to realize the dynamic integration of data nodes in a distributed database to improve the stability of the cyclone working data synchronization process is a difficult problem faced by the industry. Summary of the invention

[0004] The present application provides a cyclone working data synchronization method and system based on a distributed database, which can realize dynamic adaptive integration of data nodes used for cyclone working data synchronization in the distributed database, thereby improving the stability of cyclone working data synchronization.

[0005] In a first aspect, the present application provides a method for synchronizing cyclone working data based on a distributed database, comprising:

[0006] Obtain the synchronous storage data of the cyclone from the distributed database storing the working data of the cyclone;

[0007] Determining the storage load of each data node based on the synchronous storage data and disk usage information of each data node;

[0008] Acquire synchronous read data in a distributed database, and then determine the read load entropy of each data node through the synchronous read data and the network load of each data node, and determine the adaptive load factor of each data node in the data synchronization process according to the read load entropy of each data node and the corresponding storage load value;

[0009] All data nodes are integrated through cache hit data and various adaptive load factors during data synchronization to obtain the synchronization node domain of the distributed database, and then the synchronization trigger lock of each data node in the synchronization node domain is determined;

[0010] When each data node in the synchronization node domain is used to synchronize the cyclone working data, the corresponding synchronization trigger lock is used as the data lock of the trigger in the distributed database, and then the cyclone working data is synchronized incrementally based on the data lock.

[0011] In some embodiments, determining the storage load of each data node based on the synchronous storage data and the disk usage information of each data node specifically includes:

[0012] For each data node, obtaining a synchronous storage record of the data node in the synchronous storage data;

[0013] Obtaining a data synchronization value of a data node in the synchronization storage record;

[0014] Determine a read-write delay ratio of a data node according to the synchronous storage record;

[0015] Determine the disk load through the disk usage information of the data node;

[0016] The storage load of the data node is determined based on the data synchronization value, the read-write delay ratio and the disk load, thereby obtaining the storage load of each data node.

[0017] In some embodiments, determining the read load entropy of each data node by synchronously reading the data and the network load of each data node specifically includes:

[0018] For each data node, obtaining a synchronous reading record of the data node in the synchronous reading data;

[0019] Determine the data read amount of the data node through the synchronous reading record;

[0020] Determine the impact of network load on data reading process through the network load of data nodes;

[0021] The read load entropy of the data node is determined based on the data read amount and the network load influence, and then the read load entropy of each data node is obtained.

[0022] In some embodiments, all data nodes are integrated through cache hit data and various adaptive load factors during data synchronization to obtain a synchronization node domain of a distributed database, specifically including:

[0023] Determine the cache hit rate and cache response time of each data node through the cache hit data;

[0024] Determine the comprehensive load of each data node according to the cache hit rate of each data node, the corresponding cache response time and the corresponding adaptive load factor;

[0025] All data nodes are synchronously clustered according to all comprehensive loads to obtain a synchronous node domain.

[0026] In some embodiments, determining the synchronization trigger lock of each data node in the synchronization node domain specifically includes:

[0027] For each data node in the synchronization node domain, obtain the comprehensive load of the data node;

[0028] A trigger lock is initialized, a trigger threshold and a trigger strategy of the trigger lock are set according to the comprehensive load, a synchronous trigger lock of the data node is obtained, and then the synchronous trigger lock of each data node is determined.

[0029] In some embodiments, the data lock of the trigger is a row-level update lock.

[0030] In some embodiments, the distributed database is a cluster database composed of two or more data nodes.

[0031] In a second aspect, the present application provides a cyclone working data synchronization system based on a distributed database, including a data synchronization unit, wherein the data synchronization unit includes:

[0032] An acquisition module, used for acquiring synchronous storage data of the cyclone from a distributed database storing working data of the cyclone;

[0033] A processing module, configured to determine the storage load of each data node based on the synchronous storage data and the disk usage information of each data node;

[0034] The processing module is also used to obtain synchronous read data in the distributed database, and then determine the read load entropy of each data node through the synchronous read data and the network load of each data node, and determine the adaptive load factor of each data node in the data synchronization process according to the read load entropy of each data node and the corresponding storage load value;

[0035] The processing module is also used to integrate all data nodes through cache hit data and various adaptive load factors in the data synchronization process to obtain a synchronization node domain of the distributed database, and then determine the synchronization trigger lock of each data node in the synchronization node domain;

[0036] The execution module is used to use each data node in the synchronization node domain to synchronize the cyclone working data, use the corresponding synchronization trigger lock as the data lock of the trigger in the distributed database, and then synchronize the cyclone working data incrementally based on the data lock.

[0037] In a third aspect, the present application provides a computer device, comprising a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the computer device executes the above-mentioned cyclone working data synchronization method based on a distributed database.

[0038] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions or codes, and when the instructions or codes are executed on a computer, the computer implements the above-mentioned cyclone working data synchronization method based on a distributed database.

[0039] The technical solution provided by the embodiments disclosed in this application has the following beneficial effects:

[0040] The present application provides a cyclone working data synchronization method and system based on a distributed database, which obtains the synchronous storage data of the cyclone from the distributed database storing the cyclone working data; determines the storage load of each data node based on the synchronous storage data and the disk usage information of each data node; obtains the synchronous read data in the distributed database, and then determines the read load entropy of each data node through the synchronous read data and the network load of each data node, and determines the adaptive load factor of each data node in the data synchronization process according to the read load entropy of each data node and the corresponding storage load value; integrates all data nodes through cache hit data and each adaptive load factor in the data synchronization process to obtain the synchronization node domain of the distributed database, and then determines the synchronization trigger lock of each data node in the synchronization node domain; when using each data node in the synchronization node domain to synchronize the cyclone working data, the corresponding synchronization trigger lock is used as the data lock of the trigger in the distributed database, and then the data incremental synchronization of the cyclone working data is performed based on the data lock.

[0041] In the present application, when the data nodes in the synchronization node domain are used to synchronize the cyclone working data, each synchronization trigger lock can be used as a data lock of a trigger in a distributed database to synchronize data increments; first, the storage load is determined by synchronizing the storage data and disk usage information. The above steps can dynamically and systematically determine and manage the storage load of each data node to ensure the efficient operation and stability of the distributed database system during the data synchronization process; secondly, the read load entropy is determined by synchronously reading the data, and then the adaptive load factor of each data node in the synchronization process is determined by the read load entropy and the storage load value. The above steps can comprehensively consider the impact of the read data and the network load on the read load, calculate the read load entropy of each data node in real time, and effectively evaluate And optimize the read load balancing and performance of the distributed database system, so as to facilitate the load balancing of subsequent data nodes; then, all data nodes are integrated through adaptive load factors combined with cache hit data to obtain all data nodes used for data synchronization in the distributed database (ie: synchronization node domain), which can effectively balance the load, thereby further optimizing the performance and stability of the distributed database system; finally, the synchronization trigger lock of each data node in the synchronization node domain is obtained, and then each synchronization trigger lock is used as a data lock of a trigger in the distributed database for data incremental synchronization. In summary, based on the above scheme, dynamic adaptive integration of data nodes used for cyclone working data synchronization in the distributed database can be realized, thereby improving the stability of cyclone working data synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0043] Figure 1 is an exemplary flow chart of a method for synchronizing cyclone working data based on a distributed database according to some embodiments of the present application;

[0044] Figure 2 is a schematic diagram of a process for determining read load entropy according to some embodiments of the present application;

[0045] Figure 3 It is a schematic diagram of a process of determining a synchronization node domain according to some embodiments of the present application;

[0046] Figure 4 is a schematic diagram of exemplary hardware and / or software of a data synchronization unit according to some embodiments of the present application;

[0047] Figure 5 It is a structural schematic diagram of a computer device for implementing a cyclone working data synchronization method based on a distributed database according to some embodiments of the present application. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0049] The embodiment of the present application provides a method and system for synchronizing cyclone working data based on a distributed database, the core of which is to determine the storage load of each data node based on the synchronous storage data; extract the synchronous read data from the node operation information, and then determine the read load entropy of each data node through the synchronous read data and the network load of each data node, and determine the adaptive load factor of each data node according to the read load entropy of each data node and the corresponding storage load value; integrate all data nodes through cache hit data and each adaptive load factor to obtain a synchronization node domain; when using the data nodes in the synchronization node domain for data synchronization, use the synchronization trigger lock of each data node in the synchronization node domain as the data lock of the corresponding trigger in the distributed database for data incremental synchronization. Based on the above scheme, dynamic adaptive integration of data nodes used for cyclone working data synchronization in a distributed database can be achieved.

[0050] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods. Figure 1 , which is an exemplary flow chart of a method for synchronizing cyclone working data based on a distributed database according to some embodiments of the present application. The method for synchronizing cyclone working data based on a distributed database mainly includes the following steps:

[0051] In step 101, synchronous storage data of the cyclone is obtained from a distributed database storing working data of the cyclone.

[0052] It should be noted that in the present application, the distributed database is a cluster database composed of two or more data nodes; the data node is the basic unit for storing and managing the cyclone working data in the distributed database, and each data node is responsible for processing the storage, reading and writing requests of part of the cyclone working data to realize the distributed synchronous storage and access of the cyclone working data.

[0053] In the specific implementation, the storage records of each data node in the distributed database within a preset time period are obtained from the big data resources as synchronization storage records, and the collection of all synchronization storage records can be used as synchronization storage data, wherein the synchronization storage record includes the storage operation during the synchronization process and the operation identifier of each storage operation, and the operation identifier is a sign character used to identify the node operation. It should be noted that in this application, the preset time period can be set to 10 days, which is not limited here.

[0054] In step 102, the storage load of each data node is determined based on the synchronous storage data and the disk usage information of each data node.

[0055] In specific implementation, monitoring tools at the operating system level and built-in monitoring tools in the distributed database can be used to collect disk usage information of each data node. The disk usage information includes disk usage rate and disk operation frequency.

[0056] In some embodiments, determining the storage load of each data node based on the synchronous storage data and the disk usage information of each data node can be implemented by the following steps:

[0057] For each data node, obtaining a synchronous storage record of the data node in the synchronous storage data;

[0058] Obtaining a data synchronization value of a data node in the synchronization storage record;

[0059] Determine a read-write delay ratio of a data node according to the synchronous storage record;

[0060] Determine the disk load through the disk usage information of the data node;

[0061] The storage load of the data node is determined based on the data synchronization value, the read-write delay ratio and the disk load, thereby obtaining the storage load of each data node.

[0062] It should be noted that, in the present application, the storage load refers to the degree of load on the system by the storage operation during the data synchronization process; in the specific implementation, preferably, in some embodiments, it can be implemented in the following manner, namely: first, for each data node, the synchronous storage record of the data node is obtained in the synchronous storage data; secondly, the start synchronization time and the end synchronization time of each data synchronization process in the synchronous storage record can be counted, and the difference between the end synchronization time and the start synchronization time can be used as the data synchronization value, which reflects the timeliness of the data synchronization; then, the read and write time of each data synchronization process can be obtained in the synchronous storage record, and the ratio of each read and write time to the sum of all read and write times can be used as Corresponding to the read-write delay ratio of the data synchronization process, the average of all read-write delay ratios can be used as the read-write delay ratio of the data node; further, the ratio of disk usage and disk operation frequency can be used as the disk load, which identifies the load level of the disk in the data node; finally, the sum of the data synchronization value and the read-write delay ratio can be used as the total delay, and the product of the total delay and the preset delay weight coefficient can be used as the delay load, which indicates the degree of influence of storage delay on the storage load of the data node. The sum of the delay load and the disk load can be used as the storage load of the data node, wherein the delay weight coefficient can be preset through historical experience, and the storage load of the data node can be obtained through the above method. In other embodiments, other methods can also be used for implementation, which is not limited in this application.

[0063] It should be noted that the above steps can dynamically and systematically determine and manage the storage load of each data node, thereby ensuring the efficient operation and stability of the distributed database system.

[0064] In step 103, the synchronous read data in the distributed database is obtained, and then the read load entropy of each data node is determined through the synchronous read data and the network load of each data node, and the adaptive load factor of each data node in the data synchronization process is determined according to the read load entropy of each data node and the corresponding storage load value.

[0065] In specific implementation, the reading records of each data node in the distributed database within a preset time period are obtained from the big data resource as the synchronous reading records, and the set of all synchronous reading records can be used as the synchronous reading data, wherein the synchronous reading record includes the reading operation during the synchronization process and the operation identifier of each reading operation, and the operation identifier is a flag character used to identify the node operation.

[0066] It should be noted that in the present application, network load represents the quantitative value of the network usage in the data synchronization reading process, and the network load includes bandwidth utilization and network throughput. In specific implementation, the bandwidth utilization and network throughput of each data node are obtained from the historical data of the network status, and all bandwidth utilization and network throughput can be used as the value of the network load.

[0067] In some embodiments, the read load entropy of each data node is determined by synchronously reading the data and the network load of each data node, referring to Figure 2 The figure is a schematic diagram of a process for determining read load entropy in some embodiments of the present application. In this embodiment, determining read load entropy can be implemented by the following steps:

[0068] In step 1031, for each data node, a synchronous reading record of the data node is obtained from the synchronous reading data;

[0069] In step 1032, the data read amount of the data node is determined through the synchronous reading record;

[0070] In step 1033, the network load influence degree in the data reading process is determined by the network load of the data node;

[0071] In step 1034, the read load entropy of the data node is determined based on the data read amount and the network load influence, thereby obtaining the read load entropy of each data node.

[0072] It should be noted that in the present application, read load entropy refers to the uniformity of read load distribution during data synchronization; network load impact is a quantitative value of the impact of network load on data read load during data synchronization; data read volume refers to the proportion of read requests for each data node.

[0073] In the specific implementation, first, for each data node, the synchronous read record of the data node is obtained in the synchronous read data; secondly, the number of read requests for each synchronous read record in the synchronous read data is counted, and the sum of all read request numbers is taken as the total number of read requests, and the ratio of the number of read requests of the synchronous read record corresponding to the data node to the total number of read requests can be taken as the data read amount of the data node, and the data read amount represents the degree of influence of the data reading process on the data node load; then, the sum of all bandwidth utilization rates is taken as the total bandwidth utilization value, and the ratio of the bandwidth utilization rate in the network load of the data node to the total bandwidth utilization value can be taken as the bandwidth load, and the sum of all network throughputs is taken as the total network throughput, and the ratio of the network throughput in the network load of the data node to the total network throughput can be taken as the network throughput load, and then the bandwidth load and the network throughput load can be taken as the network load influence degree in the data reading process; finally, the sum of the data read amount and the network load influence degree is taken as the comprehensive read load, and the product of the comprehensive read load and the natural logarithm of the comprehensive read load can be taken as the read load entropy of the data node, and the read load entropy of each data node can be obtained in the above manner.

[0074] It should be noted that by comprehensively analyzing the synchronous read data and network load through the above steps and calculating the read load entropy of each data node, the read load balance and performance of the distributed database system can be effectively evaluated and optimized.

[0075] In some embodiments, determining the adaptive load factor of each data node during data synchronization according to the read load entropy and the corresponding storage load value of each data node can be implemented by the following steps:

[0076] For each data node, determining a read coefficient of the data node on a read load by synchronously reading the data, and then obtaining a read load entropy of the data node;

[0077] Determine the read load of the data node during the data synchronization process according to the read load entropy and the read coefficient;

[0078] Determine the storage coefficient of the data node on the storage load through the synchronous storage data, and then obtain the storage load value of the data node;

[0079] Determine the storage load of the data node during the data synchronization process according to the storage load value and the storage coefficient;

[0080] The adaptive load factor of the data node during the data synchronization process is determined by the read load and the storage load, and then the adaptive load factor of each data node during the data synchronization process is obtained.

[0081] It should be noted that in the present application, the adaptive load factor represents the load degree of each data node affected by the storage process and the reading process during the data synchronization process. The adaptive load factor is used to dynamically adjust the load distribution of each node to ensure load balance.

[0082] In the specific implementation, first, for each data node, the read load entropy of the data node is obtained, a load model is initialized, and the synchronous read data is used as the initial data of the load model. The read coefficient on the read load is obtained through the load model. It should be noted that the read coefficient represents the weight of the read load in the load of the data synchronization process. In other embodiments, other methods can also be used to implement it, which is not limited here; secondly, the product of the read load entropy and the read coefficient can be used as the read load of the data node in the data synchronization process; then, the storage load value of the data node is obtained, a load model is initialized, and the synchronous read data is used as the initial data of the load model. The storage coefficient on the storage load is obtained through the load model. It should be noted that the storage coefficient represents the weight of the storage load in the load of the data synchronization process. In other embodiments, other methods can also be used to implement it, which is not limited here; further, the product of the storage load value and the storage coefficient can be used as the storage load of the data node in the data synchronization process; finally, the sum of the read load and the storage load can be used as the adaptive load factor of the data node in the data synchronization process. The adaptive load factor of each data node in the data synchronization process can be obtained by the above method.

[0083] It should be noted that by using the read load entropy and storage load value through the above steps to determine the adaptive load factor of each data node, the load can be effectively balanced and the performance and stability of the distributed database system can be optimized.

[0084] In step 104, all data nodes are integrated through cache hit data and various adaptive load factors in the data synchronization process to obtain a synchronization node domain of the distributed database, and then determine the synchronization trigger lock of each data node in the synchronization node domain.

[0085] It should be noted that in the present application, the cache hit rate represents the success rate of each data node in reading data from the cache, and the cache response time represents the average response time of each data node in reading data from the cache; in specific implementation, cache hit data can be collected in monitoring tools at the operating system level and monitoring tools built into distributed databases. The cache hit data includes cache hit rate and cache response time, wherein the cache hit rate represents the success rate of each data node in reading data from the cache, and the cache response time represents the average response time of each data node in reading data from the cache.

[0086] In some embodiments, all data nodes are integrated through cache hit data and each adaptive load factor during data synchronization to obtain a synchronization node domain of a distributed database. Figure 3 The figure is a schematic diagram of a process for determining a synchronization node domain in some embodiments of the present application. In this embodiment, determining the synchronization node domain can be implemented by using the following steps:

[0087] In step 1041, the cache hit rate and cache response time of each data node are determined through the cache hit data;

[0088] In step 1042, the comprehensive load of each data node is determined according to the cache hit rate of each data node, the corresponding cache response time and the corresponding adaptive load factor;

[0089] In step 1043, all data nodes are synchronously clustered according to all comprehensive loads to obtain a synchronous node domain.

[0090] It should be noted that the synchronization node domain represents the scope and strategy of a group of nodes in a distributed database coordinating work during the data synchronization process; in specific implementation, first, the cache hit rate and cache response time of each data node can be obtained from the cache hit data; then, for each data node, the ratio of the cache response time of the data node to the cache hit rate can be used as the cache load of the data node, and then the sum of the cache load of the data node and the adaptive load factor of the data node can be used as the comprehensive load of the data node. The comprehensive load of each data node can be obtained in the above manner; finally, the K-means clustering algorithm can be used to combine all the comprehensive loads to cluster all data nodes into two parts, one part of the data nodes is used to synchronize the cyclone working data in the distributed database, and the other part of the data nodes is used to read the cyclone working data. The set of all data nodes used for cyclone working data synchronization can be used as the synchronization node domain.

[0091] It should be noted that by integrating cache hit data and adaptive load factors through the above steps, the comprehensive load of each data node can be effectively determined, and then the synchronization node domain and corresponding synchronization strategy of the distributed database can be defined to achieve efficient operation and load balancing of the system.

[0092] In some embodiments, determining the synchronization trigger lock of each data node in the synchronization node domain may be implemented by the following steps:

[0093] For each data node in the synchronization node domain, obtain the comprehensive load of the data node;

[0094] A trigger lock is initialized, a trigger threshold and a trigger strategy of the trigger lock are set according to the comprehensive load, a synchronous trigger lock of the data node is obtained, and then the synchronous trigger lock of each data node is determined.

[0095] It should be noted that in the present application, a synchronous trigger lock is a trigger for controlling the data synchronization timing and data synchronization strategy of the cyclone working data in a distributed database system. The synchronous trigger lock can ensure data consistency and system performance. The synchronous trigger lock includes a time trigger lock, a load trigger lock and an event trigger lock. In specific implementation, first, for each data node in the synchronization node domain, the comprehensive load of the data node is obtained; then, an existing data synchronization model can be used to initialize a trigger lock, and the comprehensive load is used as the input load parameter in the data synchronization model. The trigger threshold and trigger strategy of the trigger lock are set using the data synchronization model, and the trigger lock after setting can be used as the synchronous trigger lock of the data node; wherein the trigger threshold is a quantitative value of the trigger adjustment of the trigger lock in the data node, and the trigger threshold includes a high load threshold, a medium load threshold and a low load threshold, wherein the synchronization interval of the high load threshold is 5 minutes, the synchronization interval of the medium load threshold is 10 minutes, and the synchronization interval of the low load threshold is 20 minutes. The trigger strategy represents the trigger execution content of the trigger lock. In the present application, the trigger strategy includes the trigger lock end condition.

[0096] It should be noted that, through the above steps, based on cache hit data and adaptive load factors, all data nodes are integrated and the synchronization trigger lock of each data node in the synchronization node domain is determined, so that efficient synchronization and load balancing of the distributed database can be achieved.

[0097] In step 105, when each data node in the synchronization node domain is used to synchronize the cyclone working data, the corresponding synchronization trigger lock is used as the data lock of the trigger in the distributed database, and then the cyclone working data is synchronized incrementally based on the data lock.

[0098] It should be noted that the data lock of the trigger is a row-level update lock. The row-level update lock is a special lock mechanism in a distributed database system, which is mainly used to prevent deadlock and improve concurrency performance. The row-level update lock only locks specific rows that need to be read and updated. In this application, the use of row-level update locks to synchronize the cyclone working data can ensure that different types of data updates do not conflict with each other.

[0099] In specific implementation, for each data node in the synchronization node domain, when using the data node to synchronize the cyclone working data, a synchronization trigger is set in the distributed database, the trigger threshold in the synchronization trigger lock corresponding to the data node is used as the trigger condition, and the trigger lock end condition in the trigger strategy is used as the end condition of the trigger lock, and the distributed database is used to perform data incremental synchronization of the cyclone working data. For example, when the trigger threshold of the trigger lock is the high load threshold: 0.65, the medium load threshold: 0.60, and the low load threshold: 0.55, if the comprehensive load is 0.659, the time trigger lock is set to perform data incremental synchronization of the cyclone working data every 5 minutes.

[0100] In the present application, when the data nodes in the synchronization node domain are used to synchronize the cyclone working data, each synchronization trigger lock can be used as a data lock of a trigger in a distributed database to synchronize data increments; first, the storage load is determined by synchronizing the storage data and disk usage information. The above steps can dynamically and systematically determine and manage the storage load of each data node to ensure the efficient operation and stability of the distributed database system during the data synchronization process; secondly, the read load entropy is determined by synchronously reading the data, and then the adaptive load factor of each data node in the synchronization process is determined by the read load entropy and the storage load value. The above steps can comprehensively consider the impact of the read data and the network load on the read load, calculate the read load entropy of each data node in real time, and effectively evaluate And optimize the read load balancing and performance of the distributed database system, so as to facilitate the load balancing of subsequent data nodes; then, all data nodes are integrated through adaptive load factors combined with cache hit data to obtain all data nodes used for data synchronization in the distributed database (ie: synchronization node domain), which can effectively balance the load, thereby further optimizing the performance and stability of the distributed database system; finally, the synchronization trigger lock of each data node in the synchronization node domain is obtained, and then each synchronization trigger lock is used as a data lock of a trigger in the distributed database for data incremental synchronization. In summary, based on the above scheme, dynamic adaptive integration of data nodes used for cyclone working data synchronization in the distributed database can be realized, thereby improving the stability of cyclone working data synchronization.

[0101] In addition, in another aspect of the present application, in some embodiments, the present application provides a cyclone working data synchronization system based on a distributed database, and the cyclone working data synchronization system based on a distributed database also includes a data synchronization unit, referring to Figure 4 , which is a schematic diagram of exemplary hardware and / or software of a data synchronization unit according to some embodiments of the present application, the data synchronization unit includes: an acquisition module 201, a processing module 202 and an execution module 203, which are described as follows:

[0102] Acquisition module 201, in this application, acquisition module 201 is mainly used to acquire synchronous storage data of the cyclone from the distributed database storing the cyclone working data;

[0103] Processing module 202, in the present application, the processing module 202 is used to determine the storage load of each data node based on the synchronous storage data and the disk usage information of each data node;

[0104] It should be noted that the processing module 202 is also used to obtain synchronous read data in the distributed database, and then determine the read load entropy of each data node through the synchronous read data and the network load of each data node, and determine the adaptive load factor of each data node in the data synchronization process according to the read load entropy of each data node and the corresponding storage load value;

[0105] In addition, the processing module 202 is also used to integrate all data nodes through cache hit data and various adaptive load factors in the data synchronization process to obtain a synchronization node domain of the distributed database, and then determine the synchronization trigger lock of each data node in the synchronization node domain;

[0106] Execution module 203, in the present application, the execution module 203 is mainly used to use each data node in the synchronization node domain to synchronize the cyclone working data, and use the corresponding synchronization trigger lock as the data lock of the trigger in the distributed database, and then perform data incremental synchronization on the cyclone working data based on the data lock.

[0107] The above describes in detail the examples of the cyclone working data synchronization method and system based on a distributed database provided in the embodiments of the present application. It can be understood that in order to realize the above functions, the corresponding device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0108] In some embodiments, the present application also provides a computer device, comprising a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the computer device executes the above-mentioned cyclone working data synchronization method based on a distributed database.

[0109] In some embodiments, reference Figure 5 , the dotted line in the figure indicates that the unit or the module is optional, and the figure is a schematic diagram of the structure of a computer device for implementing a cyclone working data synchronization method based on a distributed database according to an embodiment of the present application. The cyclone working data synchronization method based on a distributed database in the above embodiment can be achieved by Figure 5 The computer device shown in the figure is implemented, and the computer device includes at least one processor 301, a memory 302 and at least one communication unit 305. The computer device can be a terminal device, a server or a chip.

[0110] The processor 301 may be a general-purpose processor or a special-purpose processor. For example, the processor 301 may be a central processing unit (CPU), which may be used to control the computer device, execute software programs, and process data of the software programs. The computer device may also include a communication unit 305 to implement signal input (reception) and output (transmission).

[0111] For example, the computer device may be a chip, the communication unit 305 may be an input and / or output circuit of the chip, or the communication unit 305 may be a communication interface of the chip, and the chip may be a component of a terminal device, a network device, or other devices.

[0112] For another example, the computer device may be a terminal device or a server, and the communication unit 305 may be a transceiver of the terminal device or the server, or the communication unit 305 may be a transceiver circuit of the terminal device or the server.

[0113] The computer device may include one or more memories 302, on which a program 304 is stored. The program 304 can be executed by the processor 301 to generate instructions 303, so that the processor 301 performs the method described in the above method embodiment according to the instructions 303. Optionally, data (such as a target audit model) can also be stored in the memory 302. Optionally, the processor 301 can also read the data stored in the memory 302, and the data can be stored at the same storage address as the program 304, or the data can be stored at a different storage address from the program 304.

[0114] The processor 301 and the memory 302 may be provided separately or integrated together, for example, integrated on a system on chip (SOC) of the terminal device.

[0115] It should be understood that each step of the above method embodiment can be completed by a hardware-based logic circuit or software-based instructions in the processor 301. The processor 301 can be a CPU, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, such as discrete gates, transistor logic devices, or discrete hardware components.

[0116] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0117] For example, in some embodiments, the present application also provides a computer-readable storage medium, in which instructions or codes are stored. When the instructions or codes are executed on a computer, the computer implements the above-mentioned cyclone working data synchronization method based on a distributed database.

[0118] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0119] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for synchronizing cyclone working data based on a distributed database, characterized in that: The steps include: Obtain the synchronous storage data of the cyclone from the distributed database storing the working data of the cyclone; Determining the storage load of each data node based on the synchronous storage data and disk usage information of each data node; Acquire synchronous read data in a distributed database, and then determine the read load entropy of each data node through the synchronous read data and the network load of each data node, and determine the adaptive load factor of each data node in the data synchronization process according to the read load entropy of each data node and the corresponding storage load value; All data nodes are integrated through cache hit data and various adaptive load factors during data synchronization to obtain the synchronization node domain of the distributed database, and then the synchronization trigger lock of each data node in the synchronization node domain is determined; When each data node in the synchronization node domain is used to synchronize the cyclone working data, the corresponding synchronization trigger lock is used as the data lock of the trigger in the distributed database, and then the cyclone working data is synchronized incrementally based on the data lock; The step of determining the read load entropy of each data node by synchronously reading the data and the network load of each data node specifically includes: For each data node, obtaining a synchronous reading record of the data node in the synchronous reading data; Determine the data read amount of the data node through the synchronous reading record; Determine the impact of network load on data reading process through the network load of data nodes; Determine the read load entropy of the data node based on the data read amount and the network load influence, and then obtain the read load entropy of each data node; Among them, all data nodes are integrated through cache hit data and various adaptive load factors in the data synchronization process, and the synchronization node domain of the distributed database specifically includes: Determine the cache hit rate and cache response time of each data node through the cache hit data; Determine the comprehensive load of each data node according to the cache hit rate of each data node, the corresponding cache response time and the corresponding adaptive load factor; All data nodes are synchronously clustered according to all comprehensive loads to obtain a synchronous node domain.

2. The method according to claim 1, characterized in that Determining the storage load of each data node based on the synchronous storage data and the disk usage information of each data node specifically includes: For each data node, obtaining a synchronous storage record of the data node in the synchronous storage data; Obtaining a data synchronization value of a data node in the synchronization storage record; Determine a read-write delay ratio of a data node according to the synchronous storage record; Determine the disk load through the disk usage information of the data node; The storage load of the data node is determined based on the data synchronization value, the read-write delay ratio and the disk load, thereby obtaining the storage load of each data node.

3. The method according to claim 1, characterized in that Determining the synchronization trigger lock of each data node in the synchronization node domain specifically includes: For each data node in the synchronization node domain, obtain the comprehensive load of the data node; A trigger lock is initialized, a trigger threshold and a trigger strategy of the trigger lock are set according to the comprehensive load, a synchronous trigger lock of the data node is obtained, and then the synchronous trigger lock of each data node is determined.

4. The method according to claim 1, characterized in that The data lock of the trigger is a row-level update lock.

5. The method according to claim 1, characterized in that The distributed database is a cluster database composed of two or more data nodes.

6. A cyclone working data synchronization system based on a distributed database, which uses the method of any one of claims 1 to 5 to synchronize the cyclone working data, and the cyclone working data synchronization system based on a distributed database includes a data synchronization unit, characterized in that: The data synchronization unit comprises: An acquisition module, used for acquiring synchronous storage data of the cyclone from a distributed database storing working data of the cyclone; A processing module, configured to determine the storage load of each data node based on the synchronous storage data and the disk usage information of each data node; The processing module is also used to obtain synchronous read data in the distributed database, and then determine the read load entropy of each data node through the synchronous read data and the network load of each data node, and determine the adaptive load factor of each data node in the data synchronization process according to the read load entropy of each data node and the corresponding storage load value; The processing module is also used to integrate all data nodes through cache hit data and various adaptive load factors in the data synchronization process to obtain a synchronization node domain of the distributed database, and then determine the synchronization trigger lock of each data node in the synchronization node domain; The execution module is used to use each data node in the synchronization node domain to synchronize the cyclone working data, use the corresponding synchronization trigger lock as the data lock of the trigger in the distributed database, and then synchronize the cyclone working data incrementally based on the data lock.

7. A computer device, characterized in that: The computer device includes a memory and a processor, the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the computer device executes the cyclone working data synchronization method based on a distributed database as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions or codes, and when the instructions or codes are executed on a computer, the computer implements the method for synchronizing cyclone working data based on a distributed database as claimed in any one of claims 1 to 5.