Data synchronization method and apparatus

Abstract

The application provides a data synchronization method and device, which comprises the following steps: querying the total number of data to be synchronized in a source database; querying the source database according to the total number of data to be synchronized and the number of completed synchronization data, so as to synchronize the data between the number of completed synchronization data and the total number of data; updating the number of completed synchronization data based on the number of successfully synchronized data in response to the successful synchronization of at least part of the data. In the data synchronization process, the number of completed synchronization data is recorded, and the number of completed synchronization data is updated based on the number of completed synchronization data, so that the breakpoint continuation transmission can be realized through two database queries. Compared with the related art, the number of queries is reduced, and the breakpoint continuation transmission efficiency is improved. The number of completed synchronization data (i.e. the breakpoint position) is updated only after the data synchronization is successful, so that the data that fails to be written is not recorded as the breakpoint position and is rewritten, thereby solving the problem that the data may be lost.

Description

Technical Field

[0001] This invention relates to the field of big data technology, and in particular to a data synchronization method and apparatus. Background Technology

[0002] As technology continues to evolve and iterate, the need for algorithms to process data increases, and the amount of data used to train these algorithms grows ever larger. This has created a demand for users to build data warehouses and data lakes. The process of building a data warehouse or data lake requires the synchronous collection of large amounts of data.

[0003] If data synchronization is interrupted due to network or service issues, restarting the data synchronization process each time will undoubtedly increase time costs and resource consumption. Summary of the Invention

[0004] The present invention aims to at least partially solve one of the technical problems in the related art.

[0005] Therefore, the first objective of this invention is to provide a data synchronization method.

[0006] The second objective of this invention is to provide a data synchronization device.

[0007] The third objective of this invention is to provide an electronic device.

[0008] A fourth objective of this invention is to provide a non-transitory computer-readable storage medium.

[0009] The fifth objective of this invention is to provide a computer program product.

[0010] To achieve the above objectives, a first aspect of the present invention provides a data synchronization method, comprising:

[0011] Query the total number of data entries that need to be synchronized in the source database, wherein the data in the source database is sorted in a preset order;

[0012] Based on the count of completed synchronized data entries and the total number of entries queried, the source database is queried to synchronize the data sorted between the count of completed synchronized data entries and the total number of entries;

[0013] In response to the successful synchronization of at least some data, the number of data items that have been successfully synchronized is updated based on the number of data items that have been successfully synchronized.

[0014] To achieve the above objectives, a second aspect of the present invention provides a data synchronization device, comprising:

[0015] The first query module is used to query the total number of data entries that need to be synchronized in the source database, wherein the data in the source database is sorted in a preset order;

[0016] The second query module is used to query the source database based on the number of data entries that have been synchronized and the total number of data entries to be queried, so as to synchronize the data sorted between the number of data entries that have been synchronized and the total number of data entries.

[0017] The first processing module is used to update the number of data items that have been successfully synchronized based on the number of successfully synchronized data items in response to the successful synchronization of at least some data items.

[0018] The data synchronization method and apparatus provided in this invention, when the data in the source database is sorted in a preset order, query the total number of data entries to be synchronized in the source database. Based on the statistically calculated number of data entries that have been synchronized and the total number of data entries queried, the source database is queried to synchronize data sorted between the number of data entries that have been synchronized and the total number of data entries. In response to successfully synchronizing at least some data, the number of data entries that have been synchronized is updated based on the number of successfully synchronized data entries. During the data synchronization process, by recording the number of data entries that have been synchronized, and based on this number, resuming interrupted data transmission can be achieved through two database queries. Compared with related technologies, this reduces the number of queries and improves the efficiency of resuming interrupted data transmission. In addition, the solution described in this application only updates the number of data entries that have been synchronized (i.e., updates the breakpoint position) after successful data synchronization. Compared with related technologies, data that fails to be written is not recorded as a breakpoint position and is rewritten, thus preventing data loss and solving the problem of potential data loss.

[0019] To achieve the above objectives, a third aspect of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements the method as described in the first aspect.

[0020] To achieve the above objectives, a fourth aspect of the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the method described in the first aspect.

[0021] To achieve the above objectives, a fifth aspect of the present invention provides a computer program product that, when executed by an instruction processor in the computer program product, performs the method described in the first aspect.

[0022] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0023] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0024] Figure 1 This is a flowchart illustrating a data synchronization method provided in an embodiment of the present invention;

[0025] Figure 2 This is a flowchart illustrating a data synchronization method in one scenario.

[0026] Figure 3 This is a flowchart illustrating another data synchronization method provided in an embodiment of the present invention;

[0027] Figure 4 This is a flowchart illustrating another data synchronization method provided in an embodiment of the present invention;

[0028] Figure 5 A schematic diagram of the structure of a data synchronization device provided in an embodiment of the present invention; and

[0029] Figure 6 A block diagram of an exemplary computer device suitable for implementing embodiments of the present application is shown. Detailed Implementation

[0030] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0031] The data synchronization method and apparatus of the present invention are described below with reference to the accompanying drawings.

[0032] In related technologies, pagination has been used to implement breakpoint resume, but in scenarios with heterogeneous data, if the destination database fails to write, data loss will occur. Furthermore, breakpoint resume based on pagination requires multiple queries from the source database, which is inefficient.

[0033] In view of this, embodiments of the present invention provide a data synchronization method to achieve breakpoint resume.

[0034] Figure 1 This is a schematic flowchart illustrating a data synchronization method provided in an embodiment of the present invention. Figure 1 As shown, the method includes the following steps:

[0035] Step 101: Query the total number of data entries that need to be synchronized in the source database, where the data in the source database is sorted in a preset order.

[0036] Each piece of data in the source database is sorted according to a preset order. In some methods, a sorting field can be specified when initiating a data synchronization task, so that the source database can sort the data according to the sorting field in response to the data synchronization task, thereby ensuring that the order of the data obtained each time is consistent.

[0037] In one possible implementation, when data needs to be added to the source database, it is added incrementally according to a set sorting, and the newly added data will not affect the sorting of the original data in the source database.

[0038] When data synchronization of source data is required, a data synchronization task is executed. In one scenario, if the data synchronization task specifies query conditions, the maximum number of records in the source database can be retrieved based on the specified query conditions; that is, the total number of records that need to be synchronized that meet the query conditions.

[0039] In another scenario, if the data synchronization task does not specify query conditions, it queries the total number of records in the source database.

[0040] It should be noted that querying the total number of data entries to be synchronized in this step is to determine how many data entries are pending synchronization. This is so that after the synchronization process, it can be confirmed whether all data has been synchronized successfully.

[0041] It should also be noted that since the data in the source database is also incrementally changing, each time a data synchronization task is executed, it is necessary to re-query the total number of data entries to be synchronized in the source database.

[0042] Step 102: Based on the statistically calculated number of synchronized data entries and the total number of data entries queried, query the source database to synchronize the data sorted between the number of synchronized data entries and the total number of data entries.

[0043] This query is based on the total number of records retrieved previously and the number of locally stored records that have been synchronized. The query statement is generated to retrieve and synchronize the data sorted between the number of synchronized records and the total number of records.

[0044] Step 103: In response to the successful synchronization of at least some data, update the number of data items that have been successfully synchronized based on the number of data items that have been successfully synchronized.

[0045] If at least some data is successfully synchronized, it means that the number of data records that have been synchronized in the source database has been updated. The number of data records that have been successfully synchronized this time is added to the original number of data records that have been synchronized in step 102 to obtain the updated number of data records that have been synchronized.

[0046] Furthermore, in response to the absence of the stated number of synchronized data entries, the source database is queried to synchronize the data sorted between the first entry and the total number of entries. For example... Figure 2 The diagram illustrates a data synchronization method in one scenario:

[0047] As shown in the figure, it is divided into two branches. One branch is the data synchronization process, which is executed by the input plugin; the other branch is the breakpoint information update process, which is executed by the output plugin.

[0048] Specifically, the data synchronization process is executed via an input plugin:

[0049] Step 201: Determine whether interrupted downloads need to be resumed.

[0050] Step 202: If a breakpoint resume is required, query whether there are stored records of data that have been synchronized (i.e., breakpoint information).

[0051] Step 203: If the number of data entries that have been synchronized is stored, generate a query statement based on the number of data entries that have been synchronized and the total number of queries, and perform data synchronization based on the generated query statement.

[0052] The following describes a data synchronization process from MySQL to Oracle as an example. In some possible implementations, the first SQL statement, `select count(*) from table`, retrieves the total number of rows in the source database as parameter `m`. Combined with the record breakpoint information—the number of rows that have been synchronized—as parameter `n`, the second SQL statement, `select * from table where limit n,m`, is then used. It should be noted that SQL is a standard computer language used for accessing and manipulating databases.

[0053] For example: If the total number of records in the source database is 10,000 (m = 10,000) and the breakpoint information n = 0, then the second SQL statement executed in this task is `select * from table where limit 0,10000`.

[0054] For databases similar to Oracle that support row numbers (rownum), the first SQL statement can be omitted. The executed SQL statement would then be: `select * from (select rownum no, * from table) where no>n`. This means that data querying and synchronization can be completed with just one SQL statement.

[0055] Step 204: If the number of completed data entries is not stored, or if resuming interrupted data uploads is not required, execute a full query statement. It should be noted that the full query statement mentioned here is used to retrieve all data to be synchronized for data synchronization.

[0056] The breakpoint information update process is executed via the output plugin:

[0057] Step 205: Determine whether the data synchronization process requires subsequent breakpoint resume. If so, proceed to step 206; otherwise, terminate the plugin directly.

[0058] Step 206: If necessary, update the number of data items that have been successfully synchronized (i.e., breakpoint information) based on the number of data items successfully synchronized during the aforementioned data synchronization process.

[0059] After a data synchronization task is executed once, regardless of whether it succeeds or fails, a program can be used to record the number of successfully synchronized data entries. For example, if n is recorded as 5000 before step 203 fails, then a breakpoint of 5000 will be generated for this task; if n is recorded as 6000 after step 203 succeeds, then a breakpoint of 6000 will be generated for this task.

[0060] In this embodiment, with the data in the source database sorted in a preset order, the total number of data entries to be synchronized in the source database is queried. Based on the count of completed synchronized data entries and the queried total number of entries, the source database is queried to synchronize data sorted between the number of completed synchronized data entries and the total number of entries. In response to successful synchronization of at least some data, the number of completed synchronized data entries is updated based on the number of successfully synchronized data entries. During the data synchronization process, by recording the number of completed synchronized data entries and performing two database queries based on this number, breakpoint resumption can be achieved. Compared with related technologies, this reduces the number of queries and improves the efficiency of breakpoint resumption. In addition, the solution described in this application only updates the number of completed synchronized data entries (i.e., updates the breakpoint position) after successful data synchronization. Compared with related technologies, data that fails to be written is not recorded as a breakpoint position and is rewritten, thus preventing data loss and solving the problem of potential data loss.

[0061] Building upon the previous embodiment, the data in the source database also has corresponding timestamps, which indicate the time when the corresponding data was stored in the source database. Accordingly, the preset sorting refers to the chronological order in which the data is stored in the source database.

[0062] Based on this, this embodiment provides another data synchronization method. Figure 3 This is a flowchart illustrating another data synchronization method provided in an embodiment of the present invention.

[0063] like Figure 3 As shown, the method may include the following steps:

[0064] Step 301: Query the total number of data entries that need to be synchronized in the source database, wherein the data in the source database is sorted in a preset order.

[0065] Please refer to the relevant descriptions in the foregoing embodiments for details, which will not be repeated in this embodiment.

[0066] Step 302: Obtain the target timestamp of the last synchronized data and the number of synchronized data items in the most recent synchronization count.

[0067] To enable breakpoint resume, the device used for data synchronization locally stores the number of data entries that have been synchronized during the most recent synchronization, as well as the target timestamp of the last synchronized data entry.

[0068] Step 303: Based on the statistically calculated number of synchronized data entries and the total number of data entries queried, query the source database to find data whose synchronization timestamp is greater than or equal to the target timestamp and sorted between the number of synchronized data entries and the total number of data entries.

[0069] By storing both the number of synchronized data entries and the target timestamp locally on the device performing the data synchronization, double assurance is ensured for successful resuming of interrupted data transmission. This avoids synchronization errors caused by sorting anomalies in the source database. Adding a target timestamp further helps identify such situations and prevents synchronization errors from occurring.

[0070] Step 304: In response to the successful synchronization of at least some data, update the target timestamp based on the timestamp of the last successfully synchronized data, and update the number of data items that have been successfully synchronized based on the number of successfully synchronized data items.

[0071] In this embodiment, with the data in the source database sorted in a preset order, the total number of data entries to be synchronized in the source database is queried. Based on the count of completed synchronized data entries and the queried total number of entries, the source database is queried to synchronize data sorted between the number of completed synchronized data entries and the total number of entries. In response to successful synchronization of at least some data, the number of completed synchronized data entries is updated based on the number of successfully synchronized data entries. During the data synchronization process, by recording the number of completed synchronized data entries and performing two database queries based on this number, breakpoint resumption can be achieved. Compared with related technologies, this reduces the number of queries and improves the efficiency of breakpoint resumption. In addition, the solution described in this application only updates the number of completed synchronized data entries (i.e., updates the breakpoint position) after successful data synchronization. Compared with related technologies, data that fails to be written is not recorded as a breakpoint position and is rewritten, thus preventing data loss and solving the problem of potential data loss. Furthermore, by storing the number of completed synchronized data entries and the target timestamp locally on the device performing data synchronization, double insurance is provided to ensure the normal implementation of breakpoint resumption.

[0072] During data synchronization, memory overflow is a common problem. To avoid memory overflow, this invention provides another data synchronization method. Figure 4 This is a flowchart illustrating another data synchronization method provided in an embodiment of the present invention, as shown below. Figure 4 As shown, it includes:

[0073] Step 401: Query the total number of data entries that need to be synchronized in the source database, wherein the data in the source database is sorted in a preset order.

[0074] Step 402: Query the data from the source database that is sorted between the number of synchronized data entries and the total number of data entries.

[0075] For details of steps 401 and 402, please refer to the relevant descriptions in the foregoing embodiments. They will not be repeated in this embodiment.

[0076] Step 403: The first thread retrieves the queried data from the source database and writes it into a preset data queue, so that the second thread reads the data from the preset data queue and writes it into the destination database.

[0077] To address memory issues during data synchronization and prevent application memory overflow (OOM), a pre-defined data queue is used. For example, a data queue with a set cache size is configured. During the process of writing data from the source database to the destination database, data reading and writing are handled by two threads: a first thread and a second thread. These two threads transfer data through the data queue, thus avoiding memory overflow issues.

[0078] In one possible scenario, the pre-defined data queue includes, but is not limited to, a blocking queue. The blocking queue has the following functions: firstly, when the blocking queue is empty, operations to retrieve elements from the blocking queue will be blocked; secondly, when the blocking queue is full, operations to add elements to the blocking queue will be blocked. This makes the data synchronization process more reliable.

[0079] In this embodiment, with the data in the source database sorted in a preset order, the total number of data entries to be synchronized in the source database is queried. Based on the count of completed synchronized data entries and the queried total number of entries, the source database is queried to synchronize data sorted between the number of completed synchronized data entries and the total number of entries. In response to successful synchronization of at least some data, the number of completed synchronized data entries is updated based on the number of successfully synchronized data entries. During the data synchronization process, by recording the number of completed synchronized data entries and performing two database queries based on this number, breakpoint resumption can be achieved. Compared with related technologies, this reduces the number of queries and improves the efficiency of breakpoint resumption. In addition, the solution described in this application only updates the number of completed synchronized data entries (i.e., updates the breakpoint position) after successful data synchronization. Compared with related technologies, data that fails to be written is not recorded as a breakpoint position and is rewritten, thus preventing data loss and solving the problem of potential data loss. In addition, the first thread retrieves the queried data from the source database and writes it to a preset data queue, while the second thread reads the data from the preset data queue and writes it to the destination database. By transferring data between the two threads through the data queue, the problem of memory overflow is avoided.

[0080] To achieve the above embodiments, the present invention also proposes a data synchronization device.

[0081] Figure 5 This is a schematic diagram of the structure of a data synchronization device 50 provided in an embodiment of the present invention.

[0082] like Figure 5 As shown, the data synchronization device includes: a first query module 51, a second query module 52, and a first processing module 53.

[0083] The first query module 51 is used to query the total number of data entries that need to be synchronized in the source database, wherein the data in the source database is sorted in a preset order.

[0084] The second query module 52 is used to query the source database based on the statistically calculated number of synchronized data entries and the total number of data entries, so as to synchronize the data sorted between the number of synchronized data entries and the total number of data entries.

[0085] The first processing module 53 is used to update the number of data entries that have been successfully synchronized based on the number of data entries that have been successfully synchronized in response to the successful synchronization of at least some data.

[0086] Furthermore, in one possible implementation of this invention, the data in the source database also has a corresponding timestamp, which is used to indicate the time when the corresponding data is stored in the source database; the preset sorting refers to the time order in which the data is stored in the source database.

[0087] Based on this, the second query module 52 is used for:

[0088] 0. Get the target timestamp of the last piece of synchronized data;

[0089] Based on the number of data entries that have been synchronized and the total number of data entries queried, the source database is queried to find data whose synchronization timestamp is greater than or equal to the target timestamp and is sorted between the number of data entries that have been synchronized and the total number of data entries.

[0090] Update the target timestamp based on the timestamp of the last successfully synchronized data.

[0091] 5. The data synchronization device also includes:

[0092] The second processing module is used to query the source database in response to the absence of the number of synchronized data entries, so as to synchronize the data sorted between the first entry and the total number of entries.

[0093] Furthermore, in one possible implementation of this embodiment, the second query module 52 is used for:

[0094] 0. Query the source database and sort the data based on the number of synchronized data entries and the total number of data entries.

[0095] Data between;

[0096] A first thread retrieves data from the source database and writes it to a preset data queue. A second thread then reads data from the preset data queue and writes it to the destination database. The preset data queue includes a blocking queue.

[0097] Furthermore, in one possible implementation of this invention, when the source database is a preset type database, the total number of data entries to be synchronized in the source database is queried, and the database is queried based on the statistically calculated number of synchronized data entries and the total number of data entries queried, so that the statements that synchronize the data sorted between the number of synchronized data entries and the total number of data entries belong to the same SQL statement.

[0098] It should be noted that the foregoing explanation of the data synchronization method embodiment also applies to the data synchronization device of this embodiment, and will not be repeated here.

[0099] In this embodiment of the invention, when the data in the source database is sorted in a preset order, the total number of data entries to be synchronized in the source database is queried. Based on the statistically calculated number of data entries that have been synchronized and the queried total number of entries, the source database is queried to synchronize data sorted between the number of data entries that have been synchronized and the total number of entries. In response to the successful synchronization of at least some data, the number of data entries that have been synchronized is updated based on the number of successfully synchronized data entries. During the data synchronization process, by recording the number of data entries that have been synchronized, and based on this number of data entries, breakpoint resumption can be achieved through two database queries. Compared with related technologies, this reduces the number of queries and improves the efficiency of breakpoint resumption. In addition, the solution described in this application only updates the number of data entries that have been synchronized (i.e., updates the breakpoint position) after the data synchronization is successful. Compared with related technologies, data that fails to be written is not recorded as a breakpoint position and will be rewritten, thus preventing data loss and solving the problem of potential data loss.

[0100] To implement the above embodiments, the present invention also proposes an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements the data synchronization method as described in the foregoing embodiments.

[0101] To implement the above embodiments, the present invention also proposes a non-transitory computer-readable storage medium storing a computer program thereon, characterized in that the program, when executed by a processor, implements the data synchronization method as described in the foregoing embodiments.

[0102] To implement the above embodiments, the present invention also proposes a computer program product that, when the instruction processor in the computer program product is executed, performs the data synchronization method as described in the foregoing embodiments.

[0103] Figure 6 A block diagram of an exemplary computer device suitable for implementing embodiments of the present application is shown. Figure 6 The computer device 12 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0104] like Figure 6 As shown, the computer device 12 is represented in the form of a general-purpose computing device. The components of the computer device 12 may include, but are not limited to: one or more processors or processing units 16, system memory 28, and a bus 18 connecting different system components (including system memory 28 and processing unit 16).

[0105] Bus 18 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0106] Computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0107] Memory 28 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM) 30 and / or cache memory 32. Computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media (…). Figure 6 Not shown; usually referred to as a "hard drive"). Although Figure 6 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disc drive for reading and writing to a removable non-volatile optical disc (e.g., a compact disc read-only memory (CD-ROM), a digital video disc read-only memory (DVD-ROM), or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. Memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this application.

[0108] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 42 typically perform the functions and / or methods described in the embodiments of this application.

[0109] Computer device 12 can also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), and with one or more devices that enable a user to interact with the computer system / server 12, and / or with any device that enables the computer system / server 12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 22. Furthermore, computer device 12 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 20. As shown, network adapter 20 communicates with other modules of computer device 12 via bus 18. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with computer device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0110] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing the methods mentioned in the foregoing embodiments.

[0111] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0112] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0113] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of the invention pertain.

[0114] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0115] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any of the following techniques known in the art, or a combination thereof: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0116] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0117] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0118] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. A data synchronization method, characterized in that, include: Query the total number of data entries that need to be synchronized in the source database, wherein the data in the source database is sorted in a preset order; Based on the count of completed synchronized data entries and the total number of entries queried, the source database is queried to synchronize the data sorted between the count of completed synchronized data entries and the total number of entries; In response to the successful synchronization of at least some data, the number of data items that have been successfully synchronized is updated based on the number of data items that have been successfully synchronized; In the case where the source database is a preset type database, a single SQL statement is used to query the total number of data entries that need to be synchronized in the source database, and the database is queried based on the statistically calculated number of data entries that have been synchronized and the total number of data entries queried, so as to synchronize the data sorted between the number of data entries that have been synchronized and the total number of data entries. The step of querying the source database based on the statistically calculated number of synchronized data entries and the total number of data entries retrieved, to synchronize data sorted between the number of synchronized data entries and the total number of data entries, includes: Query the source database for data sorted between the number of synchronized data entries and the total number of data entries; The first thread retrieves the queried data from the source database and writes it to a preset data queue, while the second thread reads the data from the preset data queue and writes it to the destination database. The preset data queue includes a blocking queue.

2. The method according to claim 1, characterized in that, The data in the source database also has a corresponding timestamp, which indicates the time when the corresponding data was stored in the source database; the preset order sorting refers to the chronological order in which the data is stored in the source database. The step of querying the source database based on the statistically calculated number of synchronized data entries and the total number of data entries retrieved, to synchronize data sorted between the number of synchronized data entries and the total number of data entries, includes: Get the target timestamp of the last piece of synchronized data; Based on the number of data entries that have been synchronized and the total number of data entries queried, the source database is queried to find data whose synchronization timestamp is greater than or equal to the target timestamp and is sorted between the number of data entries that have been synchronized and the total number of data entries. Update the target timestamp based on the timestamp of the last successfully synchronized data.

3. The method according to claim 1 or 2, characterized in that, The method further includes: In response to the absence of the number of synchronized data entries, the source database is queried to synchronize the data sorted between the first entry and the total number of entries.

4. A data synchronization device, characterized in that, include: The first query module is used to query the total number of data entries that need to be synchronized in the source database, wherein the data in the source database is sorted in a preset order; The second query module is used to query the source database based on the number of data entries that have been synchronized and the total number of data entries to be queried, so as to synchronize the data sorted between the number of data entries that have been synchronized and the total number of data entries. The first processing module is configured to update the number of data items that have been successfully synchronized based on the number of successfully synchronized data items in response to the successful synchronization of at least some data items. In the case where the source database is a preset type database, the statement that queries the total number of data entries to be synchronized in the source database, and queries the database based on the statistical number of data entries that have been synchronized and the total number of data entries queried, so that the data sorted between the number of data entries that have been synchronized and the total number of data entries belongs to the same SQL statement; The second query module is used for: Query the source database for data sorted between the number of synchronized data entries and the total number of data entries; The first thread retrieves the queried data from the source database and writes it to a preset data queue, while the second thread reads the data from the preset data queue and writes it to the destination database. The preset data queue includes a blocking queue.

5. The apparatus according to claim 4, characterized in that, The data in the source database also has a corresponding timestamp, which indicates the time when the corresponding data was stored in the source database; the preset order sorting refers to the chronological order in which the data is stored in the source database. The second query module is used for: Get the target timestamp of the last piece of synchronized data; Based on the number of data entries that have been synchronized and the total number of data entries queried, the source database is queried to find data whose synchronization timestamp is greater than or equal to the target timestamp and is sorted between the number of data entries that have been synchronized and the total number of data entries. Update the target timestamp based on the timestamp of the last successfully synchronized data.

6. The apparatus according to claim 4 or 5, characterized in that, The device further includes: The second processing module is used to query the source database in response to the absence of the number of synchronized data entries, so as to synchronize the data sorted between the first entry and the total number of entries.

7. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements the method as described in any one of claims 1-3.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-3.

Images