A data synchronization method, device and computer equipment

Abstract

The application relates to a data synchronization method. The method comprises the following steps: when the data of a master database changes, obtaining an operation statement in a log file, the operation statement comprising a table name and a primary key value; performing a hash operation on the table name and the primary key value to obtain a hash value, performing a modulo operation on the number of threads in a thread pool according to the hash value to obtain a modulo operation value, and matching the modulo operation value with a thread number; sending the operation statement to a thread with a thread number of the modulo operation value, so that the operation statement related to the primary key value in the data table with different table names in the same thread; and updating the data in a slave database by simultaneously executing the operation statement through multiple threads. The method can improve the efficiency of data synchronization.

Description

Technical Field

[0001] This application relates to the field of database technology, and in particular to a method, apparatus, and computer device for data synchronization. Background Technology

[0002] With the development of big data, most companies' systems require data storage, which can generally be achieved through databases, files, cloud storage, and other methods. Among these, databases are the most commonly used storage method. As the number of users and the amount of data increase, master-slave configurations can be used to address the complexity of business systems.

[0003] In related technologies, when implementing master-slave data synchronization, a single-threaded execution method is used. If the amount of data affected by a certain execution statement is large, it may cause the synchronization process to be blocked, making it difficult to improve execution efficiency. Summary of the Invention

[0004] Therefore, it is necessary to provide a data synchronization method to address the aforementioned technical problems. This method can allocate related operation statements to the same thread for execution based on the result of hash modulo operation, thereby improving data synchronization efficiency.

[0005] Firstly, this application provides a data synchronization method. The method includes:

[0006] When the data in the main database changes, retrieve the operation statements from the log file. The operation statements include the table name and primary key value.

[0007] A hash operation is performed on the table name and primary key value to obtain a hash value. The number of threads in the thread pool is then moduloed based on the hash value to obtain a modulo operation value, which is matched with the thread number.

[0008] The operation statement is sent to the thread with the modulo operation value, so that the same thread contains operation statements related to the primary key value in data tables with different table names;

[0009] The operation statements are executed simultaneously by multiple threads to update the data from the database.

[0010] In one embodiment, after sending the operation statement to the thread with the thread number corresponding to the modulo operation value, the method further includes:

[0011] The operation statements in the log file are retrieved, and the execution of the operation statements is prohibited when the number of associated records of the operation statements is greater than or equal to the number of records threshold.

[0012] In one embodiment, the method further includes:

[0013] Create a separate thread and execute statements on the separate thread if the number of associated records is greater than or equal to the record number threshold.

[0014] In one embodiment, the correlation of the number of associated records is determined based on the data correlation between the operation statements. The operation statements with correlation are assigned to the same queue, the same queue corresponds to the same thread, and the operation statements in the same queue are executed sequentially through the same thread.

[0015] In one embodiment, the method further includes:

[0016] When the number of operation statements is less than the number of threads, multiple operation statements are assigned to multiple threads, and the multiple threads execute concurrently.

[0017] The operation statements are executed simultaneously by the multiple threads to update the data from the database.

[0018] Secondly, this application also provides a data synchronization device, the device comprising:

[0019] The acquisition module is used to acquire operation statements from the log file when the data in the main database changes. The operation statements include table names and primary key values.

[0020] The calculation module is used to perform a hash operation on the table name and primary key value to obtain a hash value, and to perform a modulo operation on the number of threads in the thread pool based on the hash value to obtain a modulo operation value, which is matched with the thread number.

[0021] The allocation module is used to send the operation statement to the thread with the thread number of the modulo operation value, so that the operation statements related to the primary key value in the data tables with different table names are included in the same thread;

[0022] The execution module is used to execute the operation statements simultaneously through multiple threads to update the data from the database.

[0023] In one embodiment, after sending the operation statement to the thread with the thread number corresponding to the modulo operation value, the apparatus further includes:

[0024] The operation statements in the log file are retrieved, and the execution of the operation statements is prohibited when the number of associated records of the operation statements is greater than or equal to the number of records threshold.

[0025] In one embodiment, the device further includes:

[0026] Create a separate thread and execute statements on the separate thread if the number of associated records is greater than or equal to the record number threshold.

[0027] In one embodiment, the correlation of the number of associated records is determined based on the data correlation between the operation statements. The operation statements with correlation are assigned to the same queue, the same queue corresponds to the same thread, and the operation statements in the same queue are executed sequentially through the same thread.

[0028] In one embodiment, the device further includes:

[0029] When the number of operation statements is less than the number of threads, multiple operation statements are assigned to multiple threads, and the multiple threads execute concurrently.

[0030] The operation statements are executed simultaneously by the multiple threads to update the data from the database.

[0031] Thirdly, this disclosure also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of a data synchronization method.

[0032] Fourthly, this disclosure also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of a data synchronization method.

[0033] Fifthly, this disclosure also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of a data synchronization method.

[0034] The above data synchronization method has at least the following beneficial effects:

[0035] By performing a hash modulo operation on the table name and primary key value, related operation statements can be executed by the same thread. Each thread executes operation statements simultaneously to update data in the database, thereby improving the efficiency of data synchronization.

[0036] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments or conventional technologies of this disclosure, the accompanying drawings used in the description of the embodiments or conventional technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1This is a diagram illustrating the application environment of a data synchronization method in one embodiment;

[0039] Figure 2 This is a flowchart illustrating a data synchronization method in one embodiment;

[0040] Figure 3 This is a structural block diagram of a data synchronization device in one embodiment;

[0041] Figure 4 This is an internal structural diagram of a computer device in one embodiment;

[0042] Figure 5 This is an internal structure diagram of a server in one embodiment. Detailed Implementation

[0043] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0044] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims. The terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitation, the presence of other identical or equivalent elements in a process, method, product, or apparatus that includes said elements is not excluded. For example, the use of terms such as "first," "second," etc., is to denote names and does not indicate any specific order.

[0045] This disclosure provides a data synchronization method that can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or located in the cloud or on other network servers. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0046] In some embodiments of this disclosure, such as Figure 2 As shown, a data synchronization method is provided, which can be applied to... Figure 1 The method is illustrated using the example of a server processing operation statements. It is understood that this method can be applied to a server, and also to a system including both a terminal and a server, and is implemented through interaction between the terminal and the server. In a specific embodiment, the method may include the following steps:

[0047] S202: When the data in the main database changes, retrieve the operation statements from the log file, including the table name and primary key value.

[0048] The master database can be a relational database (like MySQL), and the log file can be a binary log file (binlog). Changes to data in the master database include adding, modifying, and deleting data. When operations such as adding, modifying, and deleting data occur in the master database, the operation statements can be recorded in the log file.

[0049] Table names can be set by the user and are related to the data types stored in the table. A primary key can be used to represent a row or column in a table and can serve as an index identifier for the data. Duplicate primary keys are not allowed in a table. For example, a table named "Student Table" indicates that this table describes student attributes, and the primary key in the table could be a student ID used to uniquely identify a student.

[0050] S204: Perform a hash operation on the table name and primary key value to obtain a hash value. Then, perform a modulo operation on the number of threads in the thread pool based on the hash value to obtain a modulo operation value. The modulo operation value is matched with the thread number.

[0051] Hash operations can be performed on table names and primary key values ​​to obtain the hash value corresponding to the primary key value. Hash operations can transform inputs of arbitrary length into outputs of fixed length, making it easier to perform modulo operations on the hash value later.

[0052] The hash value is moduloed by the number of threads to obtain the modulo operation value. For example, if the user's primary key is a string, performing a hash operation on the string will give the corresponding hash value. Suppose the hash value is 17 and the number of threads is 4. Taking the modulo of the hash value with the number of threads, 17 / 4 = 1, the modulo operation result is 1. Therefore, the data corresponding to the primary key value is assigned to the thread with thread number 1 for operation.

[0053] S206: Send the operation statement to the thread with the thread number of the modulo operation value, so that the same thread contains operation statements related to the primary key value in data tables with different table names.

[0054] In some embodiments of this disclosure, if the thread pool contains 10 threads and 100 operation statements are selected, a hash operation can be performed on the table name and primary key value to obtain a hash value. This hash value is then modulo 10, and the operation statement corresponding to the modulo value can be sent to the thread with the modulo value. This ensures that operation statements related to the primary key value in data tables with different table names within the same thread are executed, improving data synchronization efficiency. Alternatively, related operation statements can be placed in the same queue, with each queue's operation statements executed by a single thread. This reduces the number of operations on the slave database, decreases resource consumption, and improves data synchronization consistency and efficiency.

[0055] S208: The operation statement is executed simultaneously by multiple threads to update the data from the database.

[0056] After retrieving multiple operation statements from the log file, the operation statements can be assigned to their respective threads using hash modulo. Each thread executes simultaneously, ensuring that all operation statements can be executed concurrently. This allows for simultaneous execution of multiple operation statements from the database, improving the stability of data synchronization.

[0057] By executing operation statements simultaneously through multiple threads, the same operations can be performed on data in the slave database as on the master database, and the data in the slave database can be updated, ensuring data consistency between the master and slave databases. The slave database can be a relational database slave.

[0058] In the above data synchronization method, hash modulo operation can be performed on the table name and primary key value, and related operation statements can be executed by the same thread. Each thread executes operation statements simultaneously to update the data in the database, thereby improving the efficiency of data synchronization.

[0059] In some embodiments of this disclosure, after sending the operation statement to the thread with the thread number of the modulo operation value, the method further includes:

[0060] The operation statements in the log file are retrieved, and the execution of the operation statements is prohibited when the number of associated records of the operation statements is greater than or equal to the number of records threshold.

[0061] By performing a hash modulo operation on the table name and primary key value, the operation statement is sent to the thread with the thread number corresponding to the modulo operation value. That is, related operation statements are executed by the same thread. If a certain thread has a large number of operation statements, the operation statement can be prohibited when the number of related operation statements is greater than or equal to the number of operation statements threshold.

[0062] In some embodiments of this disclosure, the method further includes:

[0063] Create a separate thread and execute statements on the separate thread if the number of associated records is greater than or equal to the record number threshold.

[0064] When the number of associated records exceeds the threshold, a new independent thread is created externally, and the group of operation statements will perform data synchronization operations on this independent thread.

[0065] In some embodiments of this disclosure, the correlation of the number of associated items is determined based on the data correlation between the operation statements. The operation statements with correlation are assigned to the same queue, the same queue corresponds to the same thread, and the operation statements in the same queue are executed sequentially through the same thread.

[0066] Related operation statements are assigned to the same queue, and operation statements in the same queue are assigned to the same thread. Operation statements in the same thread can be executed according to the order of addition time, data size, etc.

[0067] In some embodiments of this disclosure, the method further includes:

[0068] When the number of operation statements is less than the number of threads, multiple operation statements are assigned to multiple threads, and the multiple threads execute concurrently.

[0069] The operation statements are executed simultaneously by the multiple threads to update the data from the database.

[0070] In some embodiments of this disclosure, the maximum number of threads may be 10, and the number of operation statements may be 8. Multiple operation statements are assigned to multiple threads, that is, one operation statement corresponds to one thread, and each thread is executed synchronously. Multiple operation statements can be executed at the same time, thereby improving the efficiency of data synchronization.

[0071] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0072] Based on the same inventive concept, this disclosure also provides a data synchronization apparatus for implementing the data synchronization method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in the data synchronization apparatus embodiments provided below can be found in the limitations of the data synchronization method described above, and will not be repeated here.

[0073] The apparatus may include a system (including a distributed system), software (application), module, component, server, client, etc., that uses the methods described in the embodiments of this specification, combined with necessary hardware implementation. Based on the same innovative concept, the apparatuses in one or more embodiments provided in this disclosure are as described in the following embodiments. Since the implementation schemes and methods for solving the problem by the apparatus are similar, the implementation of the specific apparatus in the embodiments of this specification can refer to the implementation of the foregoing methods, and repeated details will not be repeated. As used below, the terms "unit" or "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the apparatuses described in the following embodiments are preferably implemented in software, hardware implementations, or a combination of software and hardware, are also possible and contemplated.

[0074] In one embodiment, such as Figure 3 As shown, a data synchronization device 300 is provided. This device can be the aforementioned server, or a module, component, device, or unit integrated into the server. The device 300 may include:

[0075] The acquisition module 302 is used to acquire operation statements from the log file when the data in the main database changes. The operation statements include table names and primary key values.

[0076] The calculation module 304 is used to perform a hash operation on the table name and primary key value to obtain a hash value, and to perform a modulo operation on the number of threads in the thread pool based on the hash value to obtain a modulo operation value, wherein the modulo operation value is matched with the thread number.

[0077] The allocation module 306 is used to send the operation statement to the thread with the thread number of the modulo operation value, so that the same thread contains operation statements related to the primary key value in data tables with different table names;

[0078] The execution module 308 is used to execute the operation statement simultaneously through multiple threads to update the data from the database.

[0079] In one embodiment, after sending the operation statement to the thread with the thread number corresponding to the modulo operation value, the apparatus further includes:

[0080] The operation statements in the log file are retrieved, and the execution of the operation statements is prohibited when the number of associated records of the operation statements is greater than or equal to the record number threshold.

[0081] In one embodiment, the device further includes:

[0082] Create a separate thread and execute statements on the separate thread if the number of associated records is greater than or equal to the record number threshold.

[0083] In one embodiment, the correlation of the number of associated records is determined based on the data correlation between the operation statements. The operation statements with correlation are assigned to the same queue, the same queue corresponds to the same thread, and the operation statements in the same queue are executed sequentially through the same thread.

[0084] In one embodiment, the device further includes:

[0085] When the number of operation statements is less than the number of threads, multiple operation statements are assigned to multiple threads, and the multiple threads execute concurrently.

[0086] The operation statements are executed simultaneously by the multiple threads to update the data from the database.

[0087] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0088] The modules described above in the data synchronization device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0089] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores operational statements. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a data synchronization method.

[0090] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. The computer program is executed by the processor to implement data synchronization methods. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0091] Those skilled in the art will understand that Figure 4 , Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present disclosure and does not constitute a limitation on the computer device to which the present disclosure is applied. A specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0092] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the methods described in any embodiment of this disclosure.

[0093] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the methods described in any embodiment of this disclosure.

[0094] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this disclosure can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this disclosure may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this disclosure may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0095] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0096] The embodiments described above are merely illustrative of several implementations of this disclosure, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent disclosure. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this disclosure, and these all fall within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the appended claims.

Claims

1. A method for data synchronization, characterized in that, The method includes: When the data in the primary database changes, the operation statements of the primary database recorded in the primary database log file are retrieved. The operation statements include table names and primary key values. A hash operation is performed on the table name and primary key value to obtain a hash value. The number of threads in the thread pool is then moduloed based on the hash value to obtain a modulo operation value, which is matched with the thread number. The operation statement is sent to the thread with the modulo operation value, so that the same thread contains operation statements related to the primary key value in data tables with different table names; Executing the operation statements simultaneously through multiple threads to update data from the database includes: Related operation statements are assigned to the same queue, and the same queue corresponds to the same thread. The operation statements in the same queue are executed sequentially through the same thread. Create an independent thread and execute statements on the independent thread that have a number of associated records greater than or equal to a threshold number of records. The correlation of the number of associated records is determined based on the data correlation between the operation statements.

2. The method according to claim 1, characterized in that, After sending the operation statement to the thread with the thread number corresponding to the modulo operation value, the method further includes: The operation statements in the log file are retrieved, and the execution of the operation statements is prohibited when the number of associated records of the operation statements is greater than or equal to the number of records threshold.

3. The method according to claim 1, characterized in that, The method further includes: When the number of operation statements is less than the number of threads, multiple operation statements are assigned to multiple threads, and the multiple threads execute concurrently. The operation statements are executed simultaneously by the multiple threads to update the data from the database.

4. A data synchronization device, characterized in that, The device includes: The acquisition module is used to acquire the operation statements of the main database recorded in the log file of the main database when the data in the main database changes. The operation statements include table names and primary key values. The calculation module is used to perform a hash operation on the table name and primary key value to obtain a hash value, and to perform a modulo operation on the number of threads in the thread pool based on the hash value to obtain a modulo operation value, which is matched with the thread number. The allocation module is used to send the operation statement to the thread with the thread number of the modulo operation value, so that the operation statements related to the primary key value in the data tables with different table names are included in the same thread; The execution module, used to execute the operation statements simultaneously through multiple threads to update data from the database, includes: Related operation statements are assigned to the same queue, and the same queue corresponds to the same thread. The operation statements in the same queue are executed sequentially through the same thread. Create an independent thread and execute statements on the independent thread that have a number of associated records greater than or equal to a threshold number of records. The correlation of the number of associated records is determined based on the data correlation between the operation statements.

5. The apparatus according to claim 4, characterized in that, After sending the operation statement to the thread with the thread number corresponding to the modulo operation value, the device further includes: The operation statements in the log file are retrieved, and the execution of the operation statements is prohibited when the number of associated records of the operation statements is greater than or equal to the number of records threshold.

6. The apparatus according to claim 4, characterized in that, The device further includes: When the number of operation statements is less than the number of threads, multiple operation statements are assigned to multiple threads, and the multiple threads execute concurrently. The operation statements are executed simultaneously by the multiple threads to update the data from the database.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 3.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 3.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 3.

Images