Data synchronization method, storage medium and device

Abstract

The present invention provides a data synchronization method, storage medium, and device. The data synchronization method includes: obtaining data to be synchronized; writing the data to be synchronized into a buffer queue; determining whether the data to be synchronized meets preset transmission conditions; if so, writing the data in the buffer queue into a temporary file; verifying the temporary file, and copying the data in the temporary file to a target database. The present invention has the advantage of being able to transmit data in file form, meeting data synchronization requirements across isolated devices, while ensuring the performance and efficiency of data synchronization.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a data synchronization method, storage medium and device. Background Art

[0002] When using data synchronization tools for real-time data synchronization, it is divided into three stages. The first stage is to initialize the loading of existing data to obtain the basic point for data synchronization; the second stage is to perform incremental data synchronization based on the synchronization basic point established by the initial data loading; the third stage is to regularly compare and verify the source data and target data of the data synchronization to confirm that no data is lost during the data synchronization process.

[0003] In the second phase of incremental synchronization, the source synchronization software and the target software directly transmit data through the TCP protocol. Specifically, it is divided into three steps. Figure 1 , the target end initiates a connection to the source end socket service; through the agreed handshake protocol, the target end sends the breakpoint of the processing event id to the source end; the source end starts sending data from the event breakpoint id, and the target end receives the data and writes it into the local file cache.

[0004] The above approach is suitable for some environments with low security requirements. In some systems with higher security levels, during the second phase of incremental synchronization, refer to Figure 2 ,There is a cross-isolation device between the source-side synchronization software and the target-side software. Data cannot be directly transferred from the source-side area to the target-side area, that is, the application cannot be directly transmitted through the socket, which seriously affects the performance and efficiency of data synchronization. Summary of the Invention

[0005] An object of the present invention is to provide a data synchronization method, storage medium and device that overcome the above problems or at least partially solve the above problems.

[0006] A further object of the present invention is to ensure the integrity of data copies.

[0007] In particular, the present invention provides a data synchronization method, comprising:

[0008] Get the data to be synchronized;

[0009] Writing the data to be synchronized into a buffer queue;

[0010] Determining whether the data to be synchronized meets a preset transmission condition;

[0011] If so, write the data in the buffer queue into a temporary file;

[0012] The temporary file is verified, and the data of the temporary file is copied to the target database.

[0013] Optionally, the preset transmission condition includes:

[0014] The usage capacity of the buffer queue has reached the upper limit; or

[0015] The preset time has passed since the last data synchronization.

[0016] Optionally, the step of writing the data in the buffer queue into a temporary file includes:

[0017] Saving the data into a transmission file according to a preset data format;

[0018] Extracting data information, and generating a file name for the transmission file according to the data information;

[0019] The transmission file is written into the temporary file.

[0020] Optionally, the preset data format includes a first storage column and a second storage column, the first storage column is used to store the serial number of the data, and the second storage column is used to store the actual text of the data.

[0021] Optionally, the file name of the transmission file includes multiple setting fields, and the multiple setting fields include: a prefix, a sequence number, a fragment number, a last fragment identifier, a hash value, and a suffix.

[0022] Optionally, after the step of writing the data in the buffer queue into a temporary file, the method further includes:

[0023] Calculating a hash value of the temporary file;

[0024] Generate a final file name based on the serial number and fragment number of the first transferred file in the temporary file and the hash value of the temporary file;

[0025] The file name of the temporary file is modified to the final file name.

[0026] Optionally, the step of verifying the temporary file includes:

[0027] Recalculating the hash value of the temporary file;

[0028] Determine whether the recalculated hash value is consistent with the hash value in the final file name;

[0029] If so, copy the data of the temporary file to the target database.

[0030] Optionally, the data synchronization method further includes:

[0031] Get the event that the program stops midway;

[0032] Locate the transfer file where the data breakpoint is located;

[0033] Restart synchronization starting from data greater than the data breakpoint.

[0034] According to another aspect of the present invention, the present invention further provides a machine-readable storage medium having a machine-executable program stored thereon, wherein the machine-executable program, when executed by a processor, implements any one of the above-mentioned data synchronization methods.

[0035] According to another aspect of the present invention, the present invention also provides a computer device, comprising a memory, a processor, and a machine executable program stored in the memory and running on the processor, and the processor can implement any of the above-mentioned data synchronization methods when executing the machine executable program.

[0036] The data synchronization method of the present invention first obtains the data to be synchronized, then writes the data to a buffer queue. It then determines whether the data to be synchronized meets preset transmission conditions. If so, the data in the buffer queue is written to a temporary file. Finally, the temporary file is verified and the data in the temporary file is copied to the target database. In other words, the present invention transmits data in file form, enabling data synchronization across isolated devices while ensuring the performance and efficiency of data synchronization.

[0037] Furthermore, the data synchronization method of the present invention calculates a hash value of the temporary file after writing the data in the buffer queue to the temporary file, generates a final file name based on the sequence number of the first transferred file in the temporary file, the shard number, and the hash value of the temporary file, and modifies the temporary file name to the final file name. By introducing the hash value and final file name of the temporary file, the data in the temporary file can be prevented from being copied and transmitted to the target database while the data is being written to the temporary file, thereby avoiding data incompleteness.

[0038] Based on the following detailed description of specific embodiments of the present invention in conjunction with the accompanying drawings, those skilled in the art will become more aware of the above and other objects, advantages and features of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present invention. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0040] Figure 1 This is a schematic architecture diagram of conventional data synchronization in the background technology of the present invention;

[0041] Figure 2 This is a schematic architecture diagram of data synchronization across isolated devices in the background technology of the present invention;

[0042] Figure 3 is a schematic flow chart of a data synchronization method according to an embodiment of the present invention;

[0043] Figure 4 is a schematic flow chart of writing data in a buffer queue into a temporary file in a data synchronization method according to an embodiment of the present invention;

[0044] Figure 5 is a schematic flow chart of a data synchronization method according to an embodiment of the present invention after writing data in a buffer queue into a temporary file;

[0045] Figure 6 is a schematic flow chart of a data synchronization method according to an embodiment of the present invention;

[0046] Figure 7 is a schematic diagram of a machine-readable storage medium according to one embodiment of the present invention;

[0047] Figure 8 is a schematic diagram of a computer device according to one embodiment of the present invention. DETAILED DESCRIPTION

[0048] It should be understood by those skilled in the art that the embodiments described below are only some embodiments of the present invention, rather than all embodiments of the present invention, and that these embodiments are intended to explain the technical principles of the present invention and are not intended to limit the scope of protection of the present invention. Based on the embodiments provided by the present invention, all other embodiments obtained by those skilled in the art without creative effort should still fall within the scope of protection of the present invention.

[0049] It should be noted that the logic and / or steps represented in the flowchart or described in other ways herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or equipment (such as a computer-based system, a system including a processor or other system that can fetch instructions from an instruction execution system, device or equipment and execute instructions), or used in combination with these instruction execution systems, devices or equipment.

[0050] Figure 3 is a flow chart of a data synchronization method according to an optional embodiment of the present invention, referring to Figure 3 The data synchronization method at least includes the following steps S302 to S310.

[0051] Step S302: Acquire data to be synchronized.

[0052] Step S304: write the data to be synchronized into the buffer queue.

[0053] Step S306: Determine whether the data to be synchronized meets the preset transmission conditions.

[0054] Step S308: If yes, write the data in the buffer queue into a temporary file.

[0055] Step S310: verify the temporary file and copy the data of the temporary file to the target database.

[0056] The data synchronization method of the present invention first obtains the data to be synchronized, then writes it into a buffer queue. A determination is then made as to whether the data to be synchronized meets preset transmission conditions. If so, the data in the buffer queue is written to a temporary file. Finally, the temporary file is verified and the data in the temporary file is copied to the target database. This method, by transmitting data in file format, can achieve data synchronization across isolated devices while ensuring the performance and efficiency of data synchronization.

[0057] In step S306 above, the preset transmission condition can be that the buffer queue has reached its upper limit, or that the current data synchronization has reached a preset time interval since the last data synchronization. The preset time interval can be configured according to actual needs. In this way, while ensuring the real-time nature of data transmission, the use of batch data storage can reduce file transmission bandwidth.

[0058] In step S308 above, refer to Figure 4 Writing the data in the buffer queue into a temporary file may include the following steps S402 to S406.

[0059] Step S402: Save the data into a transmission file according to a preset data format.

[0060] Step S404: extract the data information and generate a file name for the transmission file based on the data information.

[0061] Step S406: writing the transmission file into a temporary file.

[0062] In the above step S402, the preset data format may include a first storage column and a second storage column, the first storage column is used to store the serial number of the data, and the second storage column is used to store the actual text of the data.

[0063] In a specific embodiment of the present invention, after the data is saved to the transmission file according to the preset data format, the first line of the file is the header information of the data, including two columns of serial number (id) and real data (data), with <tab>Separated by . Starting from the second line of the file, each line stores one event. The first column is the event sequence number, and the second column is the serialized binary data of the event. This is then converted into text format using a base64 encoding method, which is the actual text of the data. The file can have 1 to N lines of data.

[0064] In the above step S404, the file name of the transmission file may include multiple setting fields, and the multiple setting fields may include: a prefix, a sequence number, a fragment number, a last fragment identifier, a hash value, and a suffix.

[0065] In a specific embodiment of the present invention, the file name of the transmission file may be: kfs_$seqid_$fragid_lastfrag_md5.kmf, where:

[0066] kfs and .kmf are the prefix and suffix of the file name respectively;

[0067] seqid is the sequence number of the first event in the transmission file;

[0068] fragid is the fragment number of the first event in the transmission file. The data of a large event may be split into multiple fragments;

[0069] lastfrag is the last fragment identifier of the first event in the transfer file. If the last fragment identifier is 0, it means that the fragment is not the last fragment of the first event. If the last fragment identifier is 1, it means that the fragment is the last fragment of the first event.

[0070] md5 is the MD5 value of the temporary file.

[0071] In an optional embodiment of the present invention, referring to Figure 5 After writing the data in the buffer queue into the temporary file, the following steps S502 to S506 may also be included.

[0072] Step S502: Calculate the hash value of the temporary file.

[0073] Step S504: Generate a final file name according to the serial number of the first transmission file in the temporary file, the fragment number, and the hash value of the temporary file.

[0074] Step S506 changes the file name of the temporary file to the final file name.

[0075] As you can understand, after batch writing data to a temporary file, introducing the temporary file's hash value and final file name prevents the data in the temporary file from being copied and transmitted to the target database via the isolation device while the data is being written to the temporary file. In other words, only after the batch writing of the data to the temporary file is complete and the temporary file's file name is changed to the final file name can the data in the temporary file be copied to the target database, ensuring the integrity of the data copy and preventing data loss.

[0076] In the above step S506, the file name of the temporary file may be modified by an atomic copy method, which can ensure that the atomic properties of each data remain unchanged after the file name is modified.

[0077] In step S310 above, the step of verifying the temporary file may be to recalculate the hash value of the temporary file, and then determine whether the recalculated hash value is consistent with the hash value in the final file name. If they are consistent, it means that the data in the temporary file is complete at this time, and the data of the temporary file can be copied to the target database, further ensuring the reliability of the data copy.

[0078] In an optional embodiment of the present invention, referring to Figure 6 The data synchronization method may further include the following steps S602 to S606.

[0079] Step S602: Acquire an event in which the program stops midway.

[0080] Step S604: Locate the transmission file where the data breakpoint is located.

[0081] Step S606: Restart synchronization from data greater than the data breakpoint.

[0082] It should be noted that steps S602 to S606 are breakpoint resuming steps, which can be executed at any stage without affecting the execution of steps S302 to S310.

[0083] In actual applications, program interruptions may be caused by network instability, database anomalies, operating system anomalies, and synchronization software anomalies. When the target software is restarted, it can automatically identify and restart synchronization from the data breakpoint.

[0084] In step S604 above, first use the breakpoint of the warehousing record (the serial number of the last processed event + the fragment number when warehousing) to locate the breakpoint file in the target directory of the target software. Specifically, first obtain the file names of all the transfer files in the directory, and parse out the serial numbers and fragment numbers corresponding to the file names of all the transfer files. Then sort all the file names in the numerical order of the serial number and fragment number, and traverse each file name in turn until the transfer file where the data breakpoint is located is found. If the transfer file where the data breakpoint is located is not found at present, it means that the transfer file where the data breakpoint is located is missing at this time, and you can try to search again after a certain period of time.

[0085] After finding the transfer file where the data breakpoint is located, read and parse the data starting from the data line of the transfer file. After reading each file, calculate the seqid and fragid of the next expected file to be read using the seqid, fragid, and lastfrag of the last line of event data. That is, when lastfrag = 1, the expected seqid = seqid + 1 and the expected fragid = 0; when lastfrag = 0, the expected seqid = seqid and the expected fragid = fragid + 1. This way, the seqid and fragid of the data breakpoint can be found.

[0086] When restarting synchronization, skip data less than or equal to the breakpoint's seqid & fragid. Before reading file data, verify that the file's MD5 matches the MD5 in the file name to prevent the file from being copied. When reading each line of data, parse the Base64 data, deserialize it into an event object, and write it to the target database.

[0087] It can be understood that the data synchronization method of the present invention uses the file name as a sparse index to save the transmitted data. While meeting the isolation device verification, it effectively ensures the accuracy of the breakpoints on the source and target ends, avoids data loss, and meets the high-level security requirements of data transmission.

[0088] This embodiment also provides a machine-readable storage medium 10 and a computer device 20 . Figure 7 is a schematic diagram of a machine-readable storage medium 10 according to one embodiment of the present invention. Figure 8 is a schematic diagram of a computer device 20 according to one embodiment of the present invention.

[0089] The machine-readable storage medium 10 stores a machine-executable program 11 , which, when executed by a processor, implements the data synchronization method of any of the above embodiments.

[0090] The computer device 20 may include a memory 210, a processor 220, and a machine executable program 11 stored in the memory 210 and running on the processor 220, and the processor 220 implements the data synchronization method of any of the above embodiments when executing the machine executable program 11.

[0091] It should be noted that the logic and / or steps represented in the flowchart or described in other ways herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any machine-readable storage medium for use by an instruction execution system, device or equipment (such as a computer-based system, a system including a processor or other system that can fetch instructions from an instruction execution system, device or equipment and execute instructions), or used in combination with these instruction execution systems, devices or equipment.

[0092] For the purposes of the description of this embodiment, the machine-readable storage medium 10 can be any device that can contain, store, communicate, propagate, or transmit a program for use with an instruction execution system, device, or apparatus, or in conjunction with such instruction execution system, device, or apparatus. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection portion having one or more wirings (electronic device), a portable computer disk cartridge (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the machine-readable storage medium 10 can even be paper or other suitable medium on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, and then editing, interpreting, or processing in other suitable ways as necessary, and then stored in a computer memory.

[0093] It should be understood that each part of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system.

[0094] The computer device 20 can be, for example, a server, a desktop computer, a laptop computer, a tablet computer, or a smartphone. In some examples, the computer device 20 can be a cloud computing node. The computer device 20 can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc. that perform specific tasks or implement specific abstract data types. The computer device 50 can be implemented in a distributed cloud computing environment where remote processing devices linked through a communication network perform tasks. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media, including storage devices.

[0095] The computer device 20 may include a processor 220 adapted to execute stored instructions, and a memory 210 that provides temporary storage for the instructions during operation. The processor 220 may be a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations. The memory 210 may include random access memory (RAM), read-only memory, flash memory, or any other suitable storage system.

[0096] The processor 220 can be connected to an I / O interface (input / output interface) suitable for connecting the computer device 20 to one or more I / O devices (input / output devices) through a system interconnect (e.g., PCI, PCI-Express, etc.). The I / O devices may include, for example, a keyboard and a pointing device, wherein the pointing device may include a touchpad or a touch screen, etc. The I / O devices may be built-in components of the computer device 20, or may be devices externally connected to the computing device.

[0097] The processor 220 may also be linked to a display interface suitable for connecting the computer device 20 to a display device via a system interconnect. The display device may include a display screen as a built-in component of the computer device 20. The display device may also include a computer monitor, a television, or a projector, etc., externally connected to the computer device 20. In addition, a network interface controller (NIC) may be suitable for connecting the computer device 20 to a network via a system interconnect. In some embodiments, the NIC may use any suitable interface or protocol (such as an Internet Small Computer System Interface, etc.) to transmit data. The network may be a cellular network, a radio network, a wide area network (WAN), a local area network (LAN), or the Internet, etc. A remote device may be connected to the computing device via a network.

[0098] The flowchart provided in this embodiment is not intended to indicate that the operations of the method will be performed in any particular order, or that all operations of the method are included in all every case. In addition, the method may include additional operations. Within the scope of the technical ideas provided by the method of this embodiment, additional changes can be made to the above method.

[0099] At this point, those skilled in the art will recognize that, although a number of exemplary embodiments of the present invention have been shown and described in detail herein, many other variations or modifications consistent with the principles of the present invention may be directly determined or derived from the disclosure of the present invention without departing from the spirit and scope of the present invention. Therefore, the scope of the present invention should be understood and deemed to cover all such other variations or modifications.< / tab>

Claims

1. A data synchronization method, comprising: Get the data to be synchronized; Writing the data to be synchronized into a buffer queue; Determining whether the data to be synchronized meets a preset transmission condition; If so, write the data in the buffer queue into a temporary file; Verify the temporary file and copy the data of the temporary file to the target database; The step of writing the data in the buffer queue into a temporary file includes: Saving the data into a transmission file according to a preset data format; Extracting data information, and generating a file name for the transmission file according to the data information; Writing the transmission file into the temporary file; The file name of the transmission file includes multiple setting fields, and the multiple setting fields include: a prefix, a serial number of the first event in the transmission file, a fragment number of the first event in the transmission file, an end fragment identifier of the first event in the transmission file, a hash value of the temporary file, and a suffix.

2. The data synchronization method according to claim 1, wherein: The preset transmission conditions include: The usage capacity of the buffer queue has reached the upper limit; or The preset time has passed since the last data synchronization.

3. The data synchronization method according to claim 1, wherein: The preset data format includes a first storage column and a second storage column, the first storage column is used to store the serial number of the data, and the second storage column is used to store the actual text of the data.

4. The data synchronization method according to claim 1, wherein: After the step of writing the data in the buffer queue into a temporary file, the method further includes: Calculating a hash value of the temporary file; Generate a final file name based on the serial number of the first transferred file in the temporary file, the fragment number and the hash value of the temporary file; The file name of the temporary file is modified to the final file name.

5. The data synchronization method according to claim 4, wherein: The step of verifying the temporary file includes: Recalculating the hash value of the temporary file; Determine whether the recalculated hash value is consistent with the hash value in the final file name; If so, copy the data of the temporary file to the target database.

6. The data synchronization method according to claim 1, further comprising: Get the event that the program stops midway; Locate the transfer file where the data breakpoint is located; Restart synchronization starting from data greater than the data breakpoint.

7. A machine-readable storage medium having a machine-executable program stored thereon, wherein the machine-executable program, when executed by a processor, implements the data synchronization method according to any one of claims 1 to 6.

8. A computer device comprising a memory, a processor, and a machine executable program stored in the memory and running on the processor, wherein the processor implements the data synchronization method according to any one of claims 1 to 6 when executing the machine executable program.

Images