Data synchronization method, apparatus and electronic device

Abstract

The present application relates to the technical field of data processing, in particular to a data synchronization method, device and electronic equipment, the method comprising: obtaining and storing target synchronization data and configuration information of a plurality of target databases to be connected; creating a plurality of target threads; and synchronizing the target synchronization data to the target databases according to the configuration information of the target databases and the plurality of target threads. The target synchronization data is stored after being obtained, thereby reducing the load pressure of a source database and enhancing the system robustness. Even if the source database or the local device is down, as long as one of them is working normally, the data synchronization can be completed, thereby ensuring the reliability of data synchronization. Meanwhile, the data is synchronized by using a plurality of target threads, thereby realizing parallel synchronization of data and greatly improving the synchronization rate.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically to data synchronization methods, devices, and electronic equipment. Background Technology

[0002] For data analysis, a large amount of data is usually required. However, in the actual production process, in order to ensure the stability of the production environment, or for the needs of the project to use a database that is completely different from the source database, data operations are often not performed directly on the source database that stores production data. Instead, the data to be analyzed in the source database A is coarsely screened and then synchronized to another non-production database B before data operations are performed.

[0003] Different projects have different data requirements for source database A; that is, source database A needs to be synchronized to multiple non-production databases, such as database B, database C, and database D. For example... Figure 1 As shown, the existing data synchronization method involves retrieving the required data from the source database A via a synchronization script on an intermediate server when data synchronization is needed. The script then converts the data into data conforming to the syntax rules of non-production databases B, C, and D before inserting it into the corresponding non-production databases. This method places a significant load on the source database A when dealing with hundreds of billions of data points, potentially impacting production environments. If the source database A or the intermediate server crashes due to unforeseen circumstances, data corruption and incompatibility will occur, rendering the synchronization solution completely unusable and affecting subsequent processing. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a data synchronization method, apparatus, and electronic device to solve the problems of low reliability and efficiency in data synchronization.

[0005] According to a first aspect, embodiments of the present invention provide a data synchronization method, comprising:

[0006] Acquire and store target synchronization data and configuration information for the multiple target databases to be connected;

[0007] Create multiple target threads;

[0008] Based on the configuration information of the target database and the multiple target threads, the target synchronization data is synchronized to the target database.

[0009] The data synchronization method provided in this invention stores the target synchronization data after acquisition, thereby distributing the load pressure on the source database and enhancing its system robustness. Even if the source database or the local system crashes, as long as one of them is still working normally, data synchronization can be completed, ensuring the reliability of data synchronization. At the same time, multiple target threads are used for data synchronization to achieve parallel data synchronization, which greatly improves the synchronization rate.

[0010] In conjunction with the first aspect, in the first embodiment of the first aspect, the creation of multiple target threads includes:

[0011] Acquire local performance data, including the number of CPU cores, I / O time, and CPU time.

[0012] The number of target threads to be started is determined based on the performance data, in order to create the multiple target threads.

[0013] The data synchronization method provided in this embodiment of the invention uses local performance data to determine the number of target threads to be opened. Because the context switching overhead is very important, if too many threads are created, the CPU will spend more time switching contexts than executing the program, which may even lead to heap memory overflow errors, thus ensuring the efficiency of data synchronization.

[0014] In conjunction with the first embodiment of the first aspect, in the second embodiment of the first aspect, determining the number of target threads to be started based on the performance data to create the plurality of target threads includes:

[0015] Using the determined number of target threads, create the multiple target threads;

[0016] Create a thread pool and a counter;

[0017] The processing of the multiple target threads is controlled based on the thread pool and the counter.

[0018] The data synchronization method provided in this invention uses a thread pool and a counter to control multiple target threads, thereby ensuring the reliability of data synchronization and improving resource utilization.

[0019] In conjunction with the first aspect, in the third embodiment of the first aspect, the step of synchronizing the target synchronization data to the target database according to the configuration information of the target database and the plurality of target threads includes:

[0020] Based on the configuration information of the target database, establish connections with the multiple target databases;

[0021] After the target synchronization data is transformed using the multiple target threads, it is synchronized to the corresponding target database.

[0022] In conjunction with the third embodiment of the first aspect, in the fourth embodiment of the first aspect, the step of using the plurality of target threads to perform corresponding transformations on the target synchronization data and then synchronizing it to the corresponding target database includes:

[0023] Get the preset synchronization time;

[0024] Based on the preset synchronization time, the target synchronization data is transformed by the multiple target threads and then synchronized to the corresponding target database.

[0025] The data synchronization method provided in this embodiment of the invention performs data synchronization at a preset synchronization time, which can avoid data synchronization when the network is busy and improve the efficiency of data synchronization.

[0026] In conjunction with the first aspect, in the fifth embodiment of the first aspect, the acquisition and storage of target synchronization data includes:

[0027] Obtain and store the target synchronization data from the source database;

[0028] or,

[0029] The target synchronization data is acquired and stored from the production environment.

[0030] The data synchronization method provided in this invention directly obtains target synchronization data from the production environment and utilizes the extremely high performance of the local database to achieve data consistency.

[0031] In conjunction with the first aspect, in the sixth embodiment of the first aspect, the method further includes:

[0032] The duration during which the target synchronization data was not processed for synchronization is recorded;

[0033] When the duration exceeds a preset value, the target synchronization data is stored in the hardware storage space.

[0034] The data synchronization method provided in this embodiment of the invention persists the cached target synchronization data to the hardware storage space when it has not been called for a long time, so as to prevent excessive memory load from affecting the writing of new data. The data stored in the hardware storage space can also be used as backup data.

[0035] According to a second aspect, embodiments of the present invention also provide a data synchronization device, comprising:

[0036] The acquisition module is used to acquire and store target synchronization data and configuration information of multiple target databases to be connected;

[0037] Create a module for creating multiple target threads;

[0038] The synchronization module is used to synchronize the target synchronization data to the target database according to the configuration information of the target database and the multiple target threads.

[0039] The data synchronization device provided in this embodiment of the invention stores the target synchronization data after it is acquired, thereby distributing the load pressure on the source database and enhancing its system robustness. Even if the source database or the local system crashes, as long as one of them is still working normally, data synchronization can be completed, ensuring the reliability of data synchronization. At the same time, multiple target threads are used to perform data synchronization, realizing parallel data synchronization and greatly improving the synchronization rate.

[0040] According to a third aspect, embodiments of the present invention provide an electronic device, including: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the data synchronization method described in the first aspect or any embodiment of the first aspect.

[0041] According to a fourth aspect, embodiments of the present invention provide a computer-readable storage medium storing computer instructions for causing the computer to perform the data synchronization method described in the first aspect or any embodiment of the first aspect. Attached Figure Description

[0042] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0043] Figure 1 A schematic diagram of existing data synchronization is shown;

[0044] Figure 2 This is a flowchart of a data synchronization method according to an embodiment of the present invention;

[0045] Figure 3 This is a flowchart of a data synchronization method according to an embodiment of the present invention;

[0046] Figure 4 This is a flowchart of a data synchronization method according to an embodiment of the present invention;

[0047] Figure 5 This is a schematic diagram of data synchronization according to an embodiment of the present invention;

[0048] Figure 6 This is a structural block diagram of a data synchronization device according to an embodiment of the present invention;

[0049] Figure 7 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] According to an embodiment of the present invention, a data synchronization method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0052] This embodiment provides a data synchronization method that can be used in electronic devices, such as servers. In this embodiment, a server is used as an example; since this server is used for intermediate caching and data synchronization, it is referred to as an intermediate server. Figure 2 This is a flowchart of a data synchronization method according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps:

[0053] S11, acquire and store target synchronization data and configuration information of multiple target databases to be connected.

[0054] The target synchronization data is data generated in the production environment. Data generated in the production environment can be stored in the source database first, and then the intermediate server retrieves it from the source database; or, the data generated in the production environment can be directly sent to the intermediate server for storage. There are no restrictions on this, and the specific settings can be made according to actual needs.

[0055] After the intermediate server obtains the target synchronization data, it caches it using a local database. This not only enables data backup but also allows it to avoid retrieving data from the source database when it needs to synchronize the target synchronization data to databases on other devices. Instead, it can directly retrieve the target synchronization data from the local database for processing. The intermediate server's role is to cache the target synchronization data and synchronize it to other databases.

[0056] The configuration information is used to ensure that the intermediate server can accurately connect to the corresponding target database. For example, it includes the IP address, port number, ServiceName, username and password required to connect to the target database.

[0057] S12, create multiple target threads.

[0058] Target threads can be configured to correspond to a target database, or multiple target threads can correspond to one target database, and so on. There is no limit to the number of target threads created; the specific number can be set according to actual needs. That is, data synchronization is achieved using multithreading technology. A thread refers to a single, sequential flow of control. Multiple threads can run concurrently within a process, each executing different tasks in parallel. A process includes memory space allocated by the operating system and contains one or more threads. A thread cannot exist independently; it must be part of a process. A process runs continuously until all non-daemon threads have finished running.

[0059] The specifics of this step will be described in detail below.

[0060] S13, based on the configuration information of the target database and multiple target threads, synchronize the target synchronization data to the target database.

[0061] The intermediate server establishes a connection with the corresponding target database using configuration information, and then uses multiple target threads to synchronize the stored target synchronization data to the corresponding target database. It should be noted that the data synchronized to each target database can be the same or different; that is, data synchronization is performed according to the needs of each target database.

[0062] The specifics of this step will be described in detail below.

[0063] The data synchronization method provided in this embodiment stores the target synchronization data after it is acquired, thereby distributing the load pressure on the source database and enhancing its system robustness. Even if the source database or the local system crashes, as long as one of them is still working normally, data synchronization can be completed, ensuring the reliability of data synchronization. At the same time, multiple target threads are used to perform data synchronization, realizing parallel data synchronization and greatly improving the synchronization rate.

[0064] This embodiment provides a data synchronization method that can be used in electronic devices, such as servers. Taking an intermediate server as an example, Figure 3 This is a flowchart of a data synchronization method according to an embodiment of the present invention, such as... Figure 3 As shown, the process includes the following steps:

[0065] S21, acquire and store target synchronization data and configuration information of multiple target databases to be connected.

[0066] The intermediate server can obtain and store the target synchronization data from the source database, or it can obtain and store the target synchronization data from the production environment.

[0067] If there are extremely high requirements for concurrent data writing, data can be written to the database of an intermediate server for caching before writing to the source database in the production environment. In other words, the target synchronized data can be obtained directly from the production environment, and the high performance of the local database can be used to achieve data consistency.

[0068] Specifically, after obtaining the target synchronization data, the intermediate server stores it in a local database. For example, a Redis database, or other types of non-relational databases, can be configured according to actual needs. Redis is a non-relational database that offers extremely high performance because it runs entirely in memory. In this embodiment, the Redis database is used to store data that requires frequent reading and writing, such as configuration information and dictionary information. During data synchronization, the synchronization script can directly read from Redis for synchronization without accessing the source database.

[0069] The synchronous execution script is used to achieve data transfer between databases. Componently, it can be divided into three parts: a configuration module, a reading module, and a writing module. Specifically, the configuration module stores the configuration information of the target database and the time settings for scheduled synchronization (described below); the reading module reads the execution code from the database and also performs preliminary data filtering, retrieving only specific data. Furthermore, to prevent memory overflow, the intermediate server pulls data from the source database in batches; the writing module writes the data obtained by the reading module into the database to be synchronized. The quality of the writing method greatly affects the synchronization speed; in this embodiment, multi-threading is used for data synchronization.

[0070] Alternatively, the intermediate server can partition its local storage space, for example, using one part to store interactive data and another part to store data generated during the production process. That is, by sorting and partitioning by key, a global traversal is not required for data retrieval, enabling faster data location.

[0071] In addition, please refer to the description of configuration information. Figure 2 The relevant descriptions in S11 of the illustrated embodiment will not be repeated here.

[0072] S22, create multiple target threads.

[0073] Specifically, S22 includes:

[0074] S221, obtain local performance data.

[0075] The performance data includes the number of CPU cores, I / O time, and CPU time.

[0076] Local performance data is the same as intermediate server performance data, which can be obtained from the intermediate server's system.

[0077] S222, determine the number of target threads to be started based on performance data, in order to create multiple target threads.

[0078] Specifically, the ratio of I / O time to CPU time can be calculated, and then combined with the number of CPU cores to obtain the number N of target threads that need to be started. For example, the following formula can be used for calculation:

[0079] The number of target threads N = number of CPU cores * [the ratio of C+I / O time to CPU time].

[0080] Where C is a constant greater than or equal to 1 to ensure that the calculated result of [C + the ratio of I / O time to CPU time] is greater than 1. After calculating the number of target threads N, the intermediate server can create N target threads. Furthermore, after determining the number of target threads N, the size of the target synchronization data obtained by the reading module is divided by the number of target threads N to obtain the amount of execution data stored in each target thread.

[0081] In some optional implementations of this embodiment, S222 may include:

[0082] (1) Create multiple target threads using the determined number of target threads.

[0083] (2) Create a thread pool and a counter.

[0084] (3) Control the processing of multiple target threads based on thread pools and counters.

[0085] For example, the intermediate server uses the `ExecutorService` method to create a thread pool. The maximum number of target threads that can be started in the thread pool is the same as the number of target threads, N. Two counters are created using the `CountDownLatch` synchronization utility class to coordinate synchronization between multiple threads. The `for` loop starts the number of threads to be started, calculates the data executed by each thread, and executes the insert statements in the thread by calling the `executor.execute(mythead)` method in the thread pool. After execution, the thread pool is closed. It should be noted that when a thread finishes execution, both counters need to be decremented by one; otherwise, the thread will be suspended and occupy resources. One counter is used to count the number of times a single thread executes, and the second counter is used to determine the sequence number of a single thread in the overall thread sequence. After determining the execution sequence numbers of each thread, the execution order can be determined to ensure unified CPU allocation. The thread pool, counters, and thread calling methods are not limited to those described above; specific settings can be made according to actual needs.

[0086] Thread pools and counters are used to control multiple target threads to ensure the reliability of data synchronization and improve resource utilization.

[0087] S23, based on the configuration information of the target database and multiple target threads, synchronize the target synchronization data to the target database.

[0088] For example, inserting 10 million records into MySQL takes 12 minutes with a single thread, but only 77 seconds with multiple threads (10 threads), a 90% reduction. The reason for considering the optimal number of threads is that more threads are not always better. For instance, changing the creation of 10 threads to 20 threads would increase the time to over 100 seconds.

[0089] Please see details Figure 2 S13 of the illustrated embodiment will not be described again here.

[0090] The data synchronization method provided in this embodiment uses local performance data to determine the number of target threads to be started. Because the context switching overhead is very important, if too many threads are created, the CPU will spend more time switching contexts than executing the program, which may even lead to heap memory overflow errors, thus ensuring the efficiency of data synchronization.

[0091] This embodiment provides a data synchronization method that can be used in electronic devices, such as servers. Taking an intermediate server as an example, Figure 4 This is a flowchart of a data synchronization method according to an embodiment of the present invention, such as... Figure 4 As shown, the process includes the following steps:

[0092] S31, acquire and store target synchronization data and configuration information of multiple target databases to be connected.

[0093] Please see details Figure 3 S21 of the illustrated embodiment will not be described again here.

[0094] S32 creates multiple target threads.

[0095] Please see details Figure 3 S22 of the illustrated embodiment will not be described again here.

[0096] S33, based on the configuration information of the target database and multiple target threads, synchronizes the target synchronization data to the target database.

[0097] Specifically, S33 includes:

[0098] S331 establishes connections with multiple target databases based on the configuration information of the target database.

[0099] like Figure 5 As shown, the intermediate server is used to synchronize data from the source database A to non-production databases B, C, and D. Accordingly, the intermediate server needs to establish connections with the source database A and the non-production databases B, C, and D.

[0100] Specifically, the system connects to the corresponding target database using the IP address, port number, and ServiceName, and then logs in to the target database using the username and password to establish connections with multiple target databases.

[0101] S332 utilizes multiple target threads to transform the target synchronization data and then synchronizes it to the corresponding target database.

[0102] There are two synchronous execution scripts in the intermediate server: synchronous execution script A and synchronous execution script B. Synchronous execution script A is used to retrieve data from the source database A and save it into the Redis database on the intermediate server according to key-value pairs. This script can be set to synchronize during periods when the source database A has low load.

[0103] Synchronous execution script B is used to retrieve data from the Redis database or the disk of an intermediate server, transform the data, arrange it into the required data format, and write it to non-production databases B, C, and D. This script can be executed at any time, and multiple scripts can be run simultaneously to synchronize multiple non-production databases.

[0104] In some optional implementations of this embodiment, S332 may include:

[0105] (1) Obtain the preset synchronization time.

[0106] The preset synchronization time is used to determine the time when the intermediate server synchronizes data with the non-production database. For example, it can be done when the network is idle.

[0107] (2) Based on the preset synchronization time, the target synchronization data is transformed by multiple target threads and then synchronized to the corresponding target database.

[0108] When the preset synchronization time is reached, the intermediate server uses multiple target threads to transform the stored target synchronization data and then synchronizes it to the corresponding target database. The data synchronization method provided in this embodiment performs data synchronization at the preset synchronization time, avoiding data synchronization during periods of network congestion and improving data synchronization efficiency.

[0109] As an optional implementation of this embodiment, the data synchronization method further includes:

[0110] (1) The duration during which the target synchronization data was not processed.

[0111] (2) When the duration exceeds the preset value, the target synchronization data is stored in the hardware storage space.

[0112] Specifically, if data cached in the intermediate server is not accessed for a long time, Redis can persist it to hardware storage space, such as a disk.

[0113] When the cached target synchronization data has not been accessed for a long time, it is persisted to the hardware storage space to prevent excessive memory load from affecting the writing of new data. Storing it in the hardware storage space can also be used as backup data.

[0114] The data synchronization method provided in this invention improves the intermediate server by adding a database to it, thereby distributing the load on the source database and enhancing its system robustness. Even if either the source database A or the intermediate server fails, data synchronization can be completed as long as one of them is still functioning normally. Furthermore, since the database is entirely stored in memory, the speed is extremely fast. Further, the synchronization script is changed from single-threaded to multi-threaded, replacing the original single-threaded execution, allowing for parallel data synchronization and significantly improving the synchronization rate. In other words, the data synchronization method provided in this invention introduces multi-threading and a non-relational database, shortening database synchronization time, reducing time costs, and improving system stability.

[0115] This embodiment also provides a data synchronization device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0116] This embodiment provides a data synchronization device, such as... Figure 6 As shown, it includes:

[0117] The acquisition module 41 is used to acquire and store target synchronization data and configuration information of multiple target databases to be connected;

[0118] Create module 42 to create multiple target threads;

[0119] Synchronization module 43 is used to synchronize the target synchronization data to the target database according to the configuration information of the target database and the multiple target threads.

[0120] The data synchronization device provided in this embodiment stores the target synchronization data after it is acquired, thereby distributing the load pressure on the source database and enhancing its system robustness. Even if the source database or the local system crashes, as long as one of them is still working normally, data synchronization can be completed, ensuring the reliability of data synchronization. At the same time, multiple target threads are used for data synchronization to achieve parallel data synchronization, which greatly improves the synchronization rate.

[0121] In this embodiment, the data synchronization device is presented in the form of a functional unit. Here, a unit refers to an ASIC circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0122] Further functional descriptions of the above modules are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0123] This invention also provides an electronic device having the above-described features. Figure 6 The data synchronization device shown.

[0124] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of an electronic device provided in an optional embodiment of the present invention, such as... Figure 7As shown, the electronic device may include: at least one processor 51, such as a CPU (Central Processing Unit), at least one communication interface 53, memory 54, and at least one communication bus 52. The communication bus 52 is used to enable communication between these components. The communication interface 53 may include a display screen or a keyboard; optionally, the communication interface 53 may also include a standard wired interface or a wireless interface. The memory 54 may be high-speed RAM (Random Access Memory) or non-volatile memory, such as at least one disk storage device. Optionally, the memory 54 may also be at least one storage device located remotely from the aforementioned processor 51. The processor 51 may be combined with... Figure 6 The described apparatus has an application program stored in memory 54, and the processor 51 calls the program code stored in memory 54 to perform any of the above method steps.

[0125] The communication bus 52 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The communication bus 52 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0126] The memory 54 may include volatile memory, such as random-access memory (RAM); the memory may also include non-volatile memory, such as flash memory, hard disk drive (HDD) or solid-state drive (SSD); the memory 54 may also include a combination of the above types of memory.

[0127] The processor 51 can be a central processing unit (CPU), a network processor (NP), or a combination of CPU and NP.

[0128] The processor 51 may further include a hardware chip. This hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0129] Optionally, memory 54 is also used to store program instructions. Processor 51 can invoke program instructions to implement the functions described in this application. Figures 2 to 4 The data synchronization method shown in any of the embodiments.

[0130] This invention also provides a non-transitory computer storage medium storing computer-executable instructions that can execute the data synchronization method in any of the above-described method embodiments. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium may also include combinations of the above types of memory.

[0131] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A data synchronization method, characterized in that, Applications to intermediate servers include: Acquire and store target synchronization data and configuration information of multiple target databases to be connected. Storing target synchronization data includes storing the target synchronization data in a local non-relational database, and then reading the target synchronization data from the local non-relational database for synchronization. Create multiple target threads; the creation of multiple target threads includes: obtaining local performance data, the performance data including the number of CPU cores, I / O time, and CPU time; determining the number of target threads to be started based on the performance data, so as to create the multiple target threads; The number of target threads is calculated using the following formula: The number of target threads N = number of CPU cores * [ratio of C+I / O time to CPU time] Where C is a constant greater than or equal to 1, so that the calculated result of [C + the ratio of I / O time to CPU time] is greater than 1. After calculating the number of target threads N, the intermediate server creates N target threads. Based on the configuration information of the target database and the multiple target threads, the target synchronization data is synchronized to the target database; The step of synchronizing the target synchronization data to the target database according to the configuration information of the target database and the multiple target threads includes: establishing a connection with the multiple target databases based on the configuration information of the target database; using the multiple target threads to perform corresponding transformations on the target synchronization data and then synchronizing it to the corresponding target database; wherein, the intermediate server has two synchronization execution scripts, the first synchronization execution script is used to obtain data from the source database and save it into a local non-relational database according to key-value pairs, and the second synchronization execution script is used to obtain data from the local non-relational database, perform data transformation, and then synchronize it to the corresponding target database; The method further includes: counting the duration for which the target synchronization data has not been synchronized; and storing the target synchronization data in hardware storage space when the duration exceeds a preset value.

2. The method according to claim 1, characterized in that, The step of determining the number of target threads to be started based on the performance data, in order to create the plurality of target threads, includes: Using the determined number of target threads, create the multiple target threads; Create a thread pool and a counter; The processing of the multiple target threads is controlled based on the thread pool and the counter.

3. The method according to claim 1, characterized in that, The step of using the multiple target threads to perform corresponding transformations on the target synchronization data and then synchronizing it to the corresponding target database includes: Get the preset synchronization time; Based on the preset synchronization time, the target synchronization data is transformed by the multiple target threads and then synchronized to the corresponding target database.

4. The method according to claim 1, characterized in that, The acquisition and storage of target synchronization data includes: Obtain and store the target synchronization data from the source database; or, The target synchronization data is acquired and stored from the production environment.

5. A data synchronization device, characterized in that, Applications to intermediate servers include: The acquisition module is used to acquire and store target synchronization data and configuration information of multiple target databases to be connected. Storing target synchronization data includes: storing target synchronization data in a local non-relational database, and reading target synchronization data from the local non-relational database for synchronization. A creation module is used to create multiple target threads; the creation of multiple target threads includes: obtaining local performance data, the performance data including the number of CPU cores, I / O time, and CPU time; determining the number of target threads to be started based on the performance data, so as to create the multiple target threads; The number of target threads is calculated using the following formula: The number of target threads N = number of CPU cores * [ratio of C+I / O time to CPU time] Where C is a constant greater than or equal to 1, so that the calculated result of [C + the ratio of I / O time to CPU time] is greater than 1. After calculating the number of target threads N, the intermediate server creates N target threads. The synchronization module is used to synchronize the target synchronization data to the target database according to the configuration information of the target database and the multiple target threads; The step of synchronizing the target synchronization data to the target database according to the configuration information of the target database and the multiple target threads includes: establishing a connection with the multiple target databases based on the configuration information of the target database; using the multiple target threads to perform corresponding transformations on the target synchronization data and then synchronizing it to the corresponding target database; wherein, the intermediate server has two synchronization execution scripts, the first synchronization execution script is used to obtain data from the source database and save it into a local non-relational database according to key-value pairs, and the second synchronization execution script is used to obtain data from the local non-relational database, perform data transformation, and then synchronize it to the corresponding target database; The device further includes: counting the duration for which the target synchronization data has not been synchronized; and storing the target synchronization data in hardware storage space when the duration exceeds a preset value.

6. An electronic device, characterized in that, include: A memory and a processor are communicatively connected, the memory stores computer instructions, and the processor executes the data synchronization method of any one of claims 1-4 by executing the computer instructions.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the data synchronization method according to any one of claims 1-4.

Images