Data synchronization method, apparatus, device, storage medium, and product

Abstract

The application discloses a data synchronization method, device, equipment, storage medium and product. The method comprises the following steps: receiving a first to-be-synchronized file list sent by a synchronization control server, wherein the first to-be-synchronized file list comprises a plurality of file information; selecting first file information from the plurality of file information; sending a first request carrying the first file information to a cache server; in the case that the synchronization state of the first file fed back based on the first request is that the first file is not synchronized by other synchronization servers except the first synchronization server among a plurality of synchronization servers, synchronizing the first file to a file server; in the case that the first file is successfully synchronized, returning to the step of selecting the first file information from the plurality of file information until the files corresponding to the plurality of file information are all synchronized to the file server. The application embodiment improves the synchronization efficiency of a large number of files.

Description

Technical Field

[0001] This application belongs to the field of data processing technology, and in particular relates to a data synchronization method, apparatus, device, storage medium and product. Background Technology

[0002] Existing technologies can use two Redis (Remote Dictionary Server) caching servers for data synchronization. One Redis server records file information from the backup server, and the other records file information from the file server. A Redis comparison program checks for file differences and notifies the file synchronization program of these differences, which then synchronizes the differing files. This approach allows for incremental synchronization with each iteration, preventing redundant synchronization from wasting bandwidth and resources. However, this existing technology is a single-node model, resulting in low synchronization efficiency for large-scale file synchronization. Summary of the Invention

[0003] This application provides a data synchronization method, apparatus, device, storage medium, and product to solve the problem of low synchronization efficiency for large-scale file synchronization.

[0004] In a first aspect, embodiments of this application provide a data synchronization method applied to a first synchronization server, wherein the first synchronization server is any one of a plurality of synchronization servers in a data synchronization system, the data synchronization system further comprising a synchronization control server and a cache server, the method comprising:

[0005] Receive a first list of files to be synchronized sent by the synchronization control server, the first list of files to be synchronized including information on multiple files;

[0006] Select the first file information from the plurality of file information;

[0007] Send a first request carrying first file information to the cache server. The first request is used to obtain the synchronization status of the first file corresponding to the first file information.

[0008] If the first file is not synchronized by any of the other synchronization servers besides the first synchronization server, based on the feedback of the first request, the first file is synchronized to the file server.

[0009] If the first file is successfully synchronized, return to the step of selecting the first file information from the multiple file information, until all files corresponding to the multiple file information are synchronized to the file server.

[0010] Secondly, embodiments of this application provide a data synchronization method applied to a cache server, wherein the cache server exists in a data synchronization system, and the data synchronization system further includes a synchronization control server and multiple synchronization servers, the method comprising:

[0011] Receive a first request sent by the first synchronization server, which carries information about the first file;

[0012] Based on the second file to be synchronized, the synchronization status of the first file corresponding to the first file information is determined, and the first synchronization server is any one of the plurality of synchronization servers.

[0013] Send the synchronization status of the first file to the first synchronization server.

[0014] Thirdly, embodiments of this application provide a data processing apparatus, the apparatus comprising:

[0015] The first receiving module is used to receive a first list of files to be synchronized sent by the synchronization control server. The first list of files to be synchronized includes information on multiple files.

[0016] The selection module is used to select the first file information from the plurality of file information;

[0017] The first sending module is used to send a first request carrying first file information to the cache server, wherein the first request is used to obtain the synchronization status of the first file corresponding to the first file information;

[0018] The second sending module is used to synchronize the first file to the file server when it receives feedback based on the first request that the synchronization status of the first file is not being synchronized by any of the multiple synchronization servers other than the first synchronization server.

[0019] The execution module is used to return to the step of selecting the first file information from the multiple file information if the first file is successfully synchronized, until all files corresponding to the multiple file information are synchronized to the file server.

[0020] Fourthly, embodiments of this application provide a data processing apparatus, the apparatus comprising:

[0021] The second receiving module is used to receive a first request sent by the first synchronization server, which carries first file information;

[0022] The first determining module is used to determine the synchronization status of the first file corresponding to the first file information based on the second file to be synchronized, wherein the first synchronization server is any one of the plurality of synchronization servers.

[0023] The third sending module is used to send the synchronization status of the first file to the first synchronization server.

[0024] Fifthly, embodiments of this application provide a terminal device, the device including: a processor and a memory storing computer program instructions;

[0025] When the processor executes computer program instructions, it implements a data synchronization method as described in the first aspect, or a data synchronization method as described in the second aspect.

[0026] Sixthly, embodiments of this application provide a computer storage medium storing computer program instructions, which, when executed by a processor, implement the data synchronization method as described in the first aspect, or the data synchronization method as described in the second aspect.

[0027] In a seventh aspect, embodiments of this application provide a computer program product in which instructions, when executed by a processor of an electronic device, cause the electronic device to perform the data synchronization method of the first aspect or implement the data synchronization method of the second aspect.

[0028] The data synchronization method provided in this application is applied to any first synchronization server among multiple synchronization servers in a data synchronization system. When the first synchronization server receives a first list of files to be synchronized sent by the synchronization control server, it synchronizes the files corresponding to the file information in the first list to the file server. This allows each synchronization server to work independently and handle different file synchronization tasks, achieving multi-node file synchronization. Furthermore, by first checking the synchronization status of the first file, duplicate synchronization of files that have already been synchronized by other synchronization servers is avoided, thereby improving the system's parallel processing capability and synchronization efficiency for large batches of files. Attached Figure Description

[0029] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a schematic diagram of the structure of the data synchronization system provided in the embodiments of this application;

[0031] Figure 2 This is a flowchart illustrating the data synchronization method provided in an embodiment of this application;

[0032] Figure 3This is a schematic diagram of the structure of the data processing apparatus provided in the embodiments of this application;

[0033] Figure 4 This is another structural schematic diagram of the data processing apparatus provided in the embodiments of this application;

[0034] Figure 5 This is a schematic diagram of the structure of the terminal device provided in the embodiments of this application. Detailed Implementation

[0035] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0036] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0037] To address the problems in the prior art, embodiments of this application provide a data synchronization method, apparatus, device, storage medium, and product.

[0038] The data synchronization method provided in this application is applied to a data synchronization system. First, the data synchronization system is introduced.

[0039] Figure 1 A schematic diagram of the data synchronization system is shown. Figure 1 As shown, the system includes:

[0040] The system includes a synchronization control server, a synchronization server, a cache server, and a file server. The synchronization control server is connected to both the cache server and the synchronization server; the cache server is also connected to the synchronization server; and the synchronization server is also connected to the file server.

[0041] There are multiple synchronization servers, each with the same functional logic, representing different instances of the same service. The synchronization servers primarily handle file synchronization logic, receiving synchronization initiation signals from the synchronization control server, interacting with the caching server, locking files being processed in real-time to prevent them from being processed by other synchronization services, and notifying the caching server of the synchronization results. Finally, the synchronization service uploads the files to the file server.

[0042] The caching server primarily caches the list of files to be synchronized and the list of files that failed to synchronize, while also providing related operations for distributed locks on files during each process.

[0043] The file server primarily accepts and stores files uploaded synchronously. This is often a standalone SFTP / FTP server.

[0044] The synchronization control server primarily generates files to be synchronized and controls the file synchronization logic, notifying the synchronization service to begin the synchronization operation. It also receives the results of the synchronization tasks from the cache server. The synchronization control server and the synchronization server are separate servers, thus improving system availability and preventing the failure of one synchronization server from affecting the entire synchronization process.

[0045] Based on the data synchronization system in the above embodiments, the data synchronization method provided in the embodiments of this application will be described below.

[0046] Figure 2 A flowchart illustrating a data synchronization method provided in one embodiment of this application is shown. Figure 2 As shown, the method includes:

[0047] Step 201: The first synchronization server receives a first list of files to be synchronized sent by the synchronization control server. The first list of files to be synchronized includes information on multiple files.

[0048] In this embodiment, the first synchronization server can be any one of multiple synchronization servers. That is, each of the multiple synchronization servers receives the first list of files to be synchronized, and then performs subsequent operations.

[0049] The first list of files to be synchronized is obtained by the synchronization control server based on the corresponding business service. Specifically, the synchronization control server compresses all daily settlement files provided by the business service that need to be synchronized, generating a compressed file of the source files; it also writes the source file names of the daily settlement files into a .check file to generate a data file confirmation file; and it generates the MD5 hash of the source files of the daily settlement files and writes it into a .md5 checksum file. The three types of files corresponding to the daily settlement files generated are as follows, with different suffixes:

[0050] fileName.gz: A compressed file containing encrypted data.

[0051] fileName.check: Data file confirmation file, where the filename containing the data content is written.

[0052] fileName.md5: Verify the file, where the MD5 hash of the data file is written.

[0053] The synchronization control server generates a first list of files to be synchronized based on the three types of files mentioned above. Each file has a corresponding file path. Specifically, the first list of files to be synchronized is generated by combining the file paths, filenames, and corresponding MD5 hashes of all daily settlement files. The format of each daily settlement file in the list is as follows:

[0054] [{fileName1,{path1.gz1,path1.check,path1.md5},md5},{fileName2,{path1.gz1,path1.check,path1.md5},md5}...]

[0055] As shown in the cached data format above, fileName1 is the filename, path1.gz1 is the file path corresponding to the compressed file, path1.check is the file path corresponding to the data file confirmation file, path1.md5 is the file path corresponding to the verification file, and md5 is the MD5 value used for verification.

[0056] Each set of file information in the first list of files to be synchronized above includes the file name, file path, and corresponding MD5 hash.

[0057] Step 202: The first synchronization server selects the first file information from the plurality of file information;

[0058] The first synchronization server can randomly select the first file information from multiple file information.

[0059] Step 203: The first synchronization server sends a first request carrying first file information to the cache server. The first request is used to obtain the synchronization status of the first file corresponding to the first file information.

[0060] Synchronization status can be understood as transmission status. In this embodiment, the first synchronization server carries randomly selected first file information in the first request and sends it to the cache server to obtain the synchronization status of the first file. It should be noted that the first file is the aforementioned daily settlement file.

[0061] Step 204: The cache server receives a first request from the first synchronization server, which carries information about the first file.

[0062] Step 205: The cache server determines the synchronization status of the first file corresponding to the first file information based on the second file to be synchronized, wherein the first synchronization server is any one of the plurality of synchronization servers.

[0063] In this embodiment, after generating the first file to be synchronized, the synchronization control server sends the first file to be synchronized to both the first synchronization control server and the cache server. The first file to be synchronized in the cache server is the second file to be synchronized. Therefore, the second file to be synchronized has the same file information as the first file to be synchronized.

[0064] The second list of files to be synchronized records the synchronization status of the files, including those that are not synchronized and those that are being synchronized. The synchronization status can be recorded by adding a distributed lock to the file information in the second list of files to be synchronized.

[0065] Step 206: The cache server sends the synchronization status of the first file to the first synchronization server.

[0066] Step 207: When the first synchronization server receives feedback based on the first request that the synchronization status of the first file is not being synchronized by any of the other synchronization servers among the plurality of synchronization servers besides the first synchronization server, the first synchronization server synchronizes the first file with the file server.

[0067] In this embodiment, since there are multiple synchronization servers, each server performs file synchronization based on the file information in the first file to be synchronized. Therefore, when the first synchronization server needs to synchronize the first file, if the first file is being synchronized by other synchronization servers, synchronization cannot be performed. Synchronization can only be performed on the first file when it is in an unsynchronized state. This avoids conflicts during the synchronization process.

[0068] The first file includes the three types of files mentioned above: a compressed file, a data file confirmation file, and a verification file. When uploading the first file to the file server, since the first file information includes the file paths of these three types of files, they can be directly uploaded to the file server based on these file paths.

[0069] Step 208: If the first file is successfully synchronized, the first synchronization server returns to the step of selecting the first file information from the multiple file information, until all files corresponding to the multiple file information are synchronized to the file server.

[0070] If the first file is successfully synchronized to the file server, the first synchronization server will randomly select the first file from multiple file information and start the synchronization process again. This continues until all files in the first list of files to be synchronized have been synchronized to the file server.

[0071] In one embodiment, when the first synchronization server receives feedback based on the first request that the synchronization status of the first file is being synchronized by other synchronization servers among the plurality of synchronization servers besides the first synchronization server, it returns to the step of selecting the first file information from the plurality of file information until the first file corresponding to the first file information is not synchronized by other synchronization servers among the plurality of synchronization servers besides the first synchronization server.

[0072] In this embodiment, if the synchronization status of the first file is "synchronizing", the first synchronization server randomly selects the first file information from multiple file information and restarts the synchronization process. This continues until all files corresponding to all file information in the first file list to be synchronized are synchronized to the file server.

[0073] In this embodiment, the data synchronization method is applied to any first synchronization server among multiple synchronization servers in the data synchronization system. When the first synchronization server receives a first list of files to be synchronized sent by the synchronization control server, it synchronizes the files corresponding to the file information in the first list to the file server. This allows each synchronization server to work independently and handle different file synchronization tasks, achieving multi-node file synchronization. Furthermore, by first checking the synchronization status of the first file, duplicate synchronization of files that have already been synchronized by other synchronization servers is avoided, thereby improving the system's parallel processing capability and synchronization efficiency for large batches of files.

[0074] In one embodiment of this application, the method further includes:

[0075] During the synchronization of the first file, the first synchronization server locks the second file information in the second file list to be synchronized in the cache server, so that the synchronization status of the file corresponding to the second file information is modified to be synchronized, and the second file information is the same as the first file information.

[0076] In this embodiment, the second list of files to be synchronized is sent from the synchronization control server to the cache server, and the second list of files to be synchronized and the first list of files to be synchronized have the same file information.

[0077] During the synchronization of the first file, the first synchronization server needs to apply a distributed lock to the second file information that is identical to the first file information in the cache server. This prevents the first file from being uploaded repeatedly by other synchronization servers. Additionally, the timeout for the distributed lock needs to be set during the locking process to prevent indefinite lock holding and deadlocks. By using locking, potential redundancy and deadlocks that may occur when multiple nodes simultaneously handle file uploads are controlled, ensuring the normal upload of files.

[0078] In one embodiment of this application, the step of returning to the step of selecting the first file information from the plurality of file information when the first file is successfully synchronized includes:

[0079] If the first file is successfully synchronized, retrieve the first verification file from the successfully synchronized first file on the file server;

[0080] Compare the first verification file with the second verification file of the first file recorded in the first synchronization server;

[0081] If the first verification file is the same as the second verification file, a synchronization success instruction is sent to the cache server, and the process returns to the step of selecting the first file information from the multiple file information.

[0082] In this embodiment, in order to prevent the first file from being tampered with during the synchronization process, it is necessary to verify it when synchronizing the first file to the file server.

[0083] Specifically, the first verification file and the second verification file are the verification files MD5 in the above embodiments. The first verification file is the verification file after successful upload to the file server, while the second verification file is the verification file recorded by the first synchronization server before upload to the file server. If the first verification file and the second verification file are identical, it indicates that the first file has not been tampered with during the upload process. This signifies that the first file synchronization is complete, and a synchronization success instruction is sent to the cache server. The first file information is then reselected for upload.

[0084] In this embodiment, the integrity and security of the file are ensured by comparing the second verification file before the first file is successfully uploaded with the first verification file after the first file is successfully uploaded.

[0085] In one embodiment of this application, the method further includes:

[0086] Upon receiving a synchronization success instruction from the first synchronization server indicating successful verification of the first file, the cache server deletes the second file information that is identical to the first file information from the second file to be synchronized.

[0087] In this embodiment, upon receiving a successful synchronization instruction, the cache server deletes second file information that is identical to the first file information from the second file to be synchronized, thereby ensuring that the same file will not be repeatedly uploaded during subsequent file synchronization processes and improving file synchronization efficiency.

[0088] In one embodiment of this application, after synchronizing the first file with the file server, the method further includes:

[0089] If the synchronization of the first file fails, record the breakpoint information of the first file;

[0090] Send a synchronization failure instruction containing the breakpoint information to the cache server.

[0091] In this embodiment, breakpoint information refers to file information after the first file transfer fails, including the breakpoint location, filename, and verification information. The breakpoint information is shown below:

[0092] [{fileName1,{{path1-1,point1},{path1-2,point2},{path1-3,point3}},md5}...].

[0093] Among them, {path1-1,point1},{path1-2,point2},{path1-3,point3} are the breakpoint locations.

[0094] After determining the breakpoint information, it is sent to the cache server along with the synchronization failure instruction.

[0095] In one embodiment of this application, after sending a synchronization failure instruction containing the breakpoint information to the cache server, the method further includes:

[0096] The system receives a failure handling list sent by the synchronization control server. The failure handling list is generated by the cache server based on the synchronization failure instruction and sent to the synchronization control server.

[0097] Based on the breakpoint information in the failure handling list, determine the breakpoint position of the first file, where the breakpoint position is the location of the interrupted synchronization in the first file;

[0098] At the breakpoint location, synchronize the first file with the file server.

[0099] In this embodiment, the failure handling list includes breakpoint information for all files that failed to synchronize. After receiving the failure handling list, the first synchronization server can randomly read the breakpoint information in the list, determine the breakpoint position, and resynchronize the corresponding first file to the server from the breakpoint position. This enables resume upload in case of first file upload failure, improving file synchronization efficiency.

[0100] As an example, suppose the first file has 10 lines of data. If the upload of the first file fails, the breakpoint might be a byte in the 5th line of data. When re-uploading, the first file can be re-synchronized to the file server at that byte.

[0101] In addition, if the first file fails to upload again during the re-upload process, a second failure handling list can be created. After all files in the first failure handling list have been processed, the second failure handling list can be uploaded again.

[0102] In one embodiment of this application, the method further includes:

[0103] The cache server receives a synchronization failure instruction sent by the first synchronization server, the synchronization failure instruction including the breakpoint information of the first file;

[0104] The cache server records the breakpoint information in the failure handling list;

[0105] The cache server sends the failure handling list to the synchronization control server.

[0106] In one embodiment, after sending the failure handling list to the synchronization control server, the method further includes:

[0107] If the cache server does not have file information in the second file to be synchronized, it sends a second request to the synchronization control server, so that the synchronization control server sends the failure handling list to the first synchronization server.

[0108] In this embodiment, after receiving a synchronization failure instruction, the cache server records the breakpoint information carried by the synchronization instruction into the failure handling list. After clearing the file information in the second file list to be synchronized, the cache server sends the failure handling list to the synchronization control server. This allows the synchronization control server to continue resuming the transmission of the files in the failure handling list after all files in the first file list to be synchronized have been synchronized, ensuring that all files that need to be synchronized can be synchronized to the file server and guaranteeing file integrity.

[0109] like Figure 3As shown in the illustration, this application also provides a data processing device 300, applied to a first synchronization server. The data processing device 300 includes:

[0110] The first receiving module 301 is used to receive a first list of files to be synchronized sent by the synchronization control server, the first list of files to be synchronized including multiple file information;

[0111] Selection module 302 is used to select first file information from the plurality of file information;

[0112] The first sending module 303 is used to send a first request carrying first file information to the cache server. The first request is used to obtain the synchronization status of the first file corresponding to the first file information.

[0113] The second sending module 304 is used to synchronize the first file to the file server when it receives feedback based on the first request that the synchronization status of the first file is not synchronized by other synchronization servers among the plurality of synchronization servers other than the first synchronization server.

[0114] The execution module 305 is used to return to the step of selecting the first file information from the multiple file information if the first file is successfully synchronized, until all the files corresponding to the multiple file information are synchronized to the file server.

[0115] Optionally, the data processing device 300 is specifically used for:

[0116] If the system receives feedback that the synchronization status of the first file based on the first request is that it is being synchronized by other synchronization servers among the plurality of synchronization servers besides the first synchronization server, the system returns to the step of selecting the first file information from the plurality of file information until the first file corresponding to the first file information is not synchronized by other synchronization servers among the plurality of synchronization servers besides the first synchronization server.

[0117] Optionally, the data processing device 300 is specifically used for:

[0118] During the synchronization of the first file, the second file information in the second file list to be synchronized in the cache server is locked so that the synchronization status of the file corresponding to the second file information is changed to synchronized, and the second file information is the same as the first file information.

[0119] The second list of files to be synchronized is sent from the synchronization control server to the cache server, and the second list of files to be synchronized and the first list of files to be synchronized have the same file information.

[0120] Optionally, execution module 305 includes:

[0121] The acquisition submodule is used to acquire the first verification file from the first file that has been successfully synchronized in the file server, provided that the first file has been successfully synchronized.

[0122] The comparison submodule is used to compare the first verification file with the second verification file of the first file recorded in the first synchronization server;

[0123] The first sending submodule is used to send a synchronization success instruction to the cache server when the first verification file is the same as the second verification file, and return to execute the step of selecting the first file information from the multiple file information.

[0124] Optionally, the data processing device 300 includes:

[0125] The first recording module is used to record the breakpoint information of the first file in the event that the synchronization of the first file fails.

[0126] The fourth sending module is used to send a synchronization failure instruction containing the breakpoint information to the cache server.

[0127] Optionally, the data processing device 300 includes:

[0128] The third receiving module is used to receive the failure handling list sent by the synchronization control server. The failure handling list is generated by the cache server based on the synchronization failure instruction and sent to the synchronization control server.

[0129] The second determining module is used to determine the breakpoint position of the first file based on the breakpoint information in the failure handling list, wherein the breakpoint position is the position of the interrupted synchronization in the first file;

[0130] A synchronization module is used to synchronize the first file with the file server at the breakpoint location.

[0131] Optionally, the data processing device 300 is specifically used for:

[0132] Upon receiving a distributed lock based on the first request, the synchronization status of the first file is determined to be that it has not been synchronized by any of the other synchronization servers among the plurality of synchronization servers, except for the first synchronization server.

[0133] It should be noted that the data processing device 300 is a device corresponding to the data synchronization method applied to the first synchronization server described above. All implementation methods in the above method embodiments are applicable to the embodiments of this device and can achieve the same technical effect.

[0134] like Figure 4 As shown in the figure, this application embodiment also provides a data processing device 400, applied to a cache server, the data processing device 400 including:

[0135] The second receiving module is used to receive a first request sent by the first synchronization server, which carries first file information;

[0136] The first determining module is used to determine the synchronization status of the first file corresponding to the first file information based on the second file to be synchronized, wherein the first synchronization server is any one of the plurality of synchronization servers.

[0137] The third sending module is used to send the synchronization status of the first file to the first synchronization server.

[0138] Optionally, the data processing device 400 is specifically used for:

[0139] Upon receiving a synchronization success instruction from the first synchronization server indicating successful verification of the first file, the second file information identical to the first file information is deleted from the second file to be synchronized.

[0140] Optionally, the data processing device 400 further includes:

[0141] The fourth receiving module is used to receive a synchronization failure instruction sent by the first synchronization server, wherein the synchronization failure instruction includes the breakpoint information of the first file;

[0142] The second recording module is used to record the breakpoint information to the failure handling list;

[0143] The fifth sending module is used to send the failure handling list to the synchronization control server.

[0144] Optionally, the data processing device 400 is specifically used for:

[0145] If no file information is found in the second file to be synchronized, a second request is sent to the synchronization control server, so that the synchronization control server sends the failure handling list to the first synchronization server.

[0146] It should be noted that the data processing device 400 is a device corresponding to the data synchronization method applied to the cache server described above. All implementation methods in the above method embodiments are applicable to the embodiments of this device and can achieve the same technical effect.

[0147] Figure 5 A schematic diagram of the hardware structure of the terminal device provided in an embodiment of this application is shown.

[0148] The terminal device may include a processor 501 and a memory 502 storing computer program instructions.

[0149] Specifically, the processor 501 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0150] Memory 502 may include mass storage for data or instructions. For example, and not limitingly, memory 502 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 502 may include removable or non-removable (or fixed) media. Where appropriate, memory 502 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 502 is non-volatile solid-state memory.

[0151] In certain embodiments, memory 502 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Thus, generally, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of this disclosure.

[0152] The processor 501 implements any of the data synchronization methods described in the above embodiments by reading and executing computer program instructions stored in the memory 502.

[0153] In one example, the terminal device may also include a communication interface 503 and a bus 510. Wherein, for example... Figure 5 As shown, the processor 501, memory 502, and communication interface 503 are connected through bus 510 and complete communication with each other.

[0154] The communication interface 503 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0155] Bus 510 includes hardware, software, or both, that couples components of an online data traffic metering device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper-Synchronous (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 510 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.

[0156] Furthermore, in conjunction with the data synchronization methods in the above embodiments, this application embodiment can provide a computer storage medium for implementation. This computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the data synchronization methods in the above embodiments.

[0157] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0158] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. The programs or code segments can be stored on a machine-readable medium or transmitted synchronously via data signals carried on a carrier wave on a synchronization medium or communication link. "Machine-readable medium" can include any medium capable of storing or synchronizing information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0159] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0160] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0161] The above are merely specific embodiments of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A data synchronization method, characterized in that, The method is applied to a first synchronization server, which is any one of multiple synchronization servers in a data synchronization system. The data synchronization system further includes a synchronization control server and a cache server. Receive a first list of files to be synchronized sent by the synchronization control server, the first list of files to be synchronized including information on multiple files; Select the first file information from the plurality of file information; A first request carrying first file information is sent to the cache server so that the cache server can determine the synchronization status of the first file corresponding to the first file information based on the second file to be synchronized. The second file to be synchronized is the first file to be synchronized in the cache server. The second file to be synchronized has the same file information as the first file to be synchronized. The first request is used to obtain the synchronization status of the first file corresponding to the first file information. If the first file is not synchronized by any of the other synchronization servers besides the first synchronization server, based on the feedback of the first request, the first file is synchronized to the file server. If the first file is successfully synchronized, return to the step of selecting the first file information from the multiple file information, until all files corresponding to the multiple file information are synchronized to the file server; The method further includes: During the synchronization of the first file, the second file information in the second file list to be synchronized in the cache server is locked according to a preset time, so that the synchronization status of the file corresponding to the second file information is modified to be synchronized, and the second file information is the same as the first file information. The second list of files to be synchronized is sent from the synchronization control server to the cache server, and the second list of files to be synchronized and the first list of files to be synchronized have the same file information.

2. The data synchronization method as described in claim 1, characterized in that, After sending the first request carrying the first file information to the cache server, the method further includes: If the system receives feedback that the synchronization status of the first file based on the first request is that it is being synchronized by other synchronization servers among the plurality of synchronization servers besides the first synchronization server, the system returns to the step of selecting the first file information from the plurality of file information until the first file corresponding to the first file information is not synchronized by other synchronization servers among the plurality of synchronization servers besides the first synchronization server.

3. The data synchronization method as described in claim 1, characterized in that, If the first file is successfully synchronized, the step of returning to the step of selecting the first file information from the plurality of file information includes: If the first file is successfully synchronized, retrieve the first verification file from the successfully synchronized first file on the file server; Compare the first verification file with the second verification file of the first file recorded in the first synchronization server; If the first verification file is the same as the second verification file, a synchronization success instruction is sent to the cache server, and the process returns to the step of selecting the first file information from the multiple file information.

4. The data synchronization method as described in claim 1, characterized in that, After synchronizing the first file with the file server, the method further includes: If the synchronization of the first file fails, record the breakpoint information of the first file; Send a synchronization failure instruction containing the breakpoint information to the cache server.

5. The data synchronization method as described in claim 4, characterized in that, After sending a synchronization failure instruction containing the breakpoint information to the cache server, the method further includes: The system receives a failure handling list sent by the synchronization control server. The failure handling list is generated by the cache server based on the synchronization failure instruction and sent to the synchronization control server. Based on the breakpoint information in the failure handling list, determine the breakpoint position of the first file, where the breakpoint position is the location of the interrupted synchronization in the first file; At the breakpoint location, synchronize the first file with the file server.

6. The data synchronization method as described in claim 1, characterized in that, Upon receiving a distributed lock based on the first request, the synchronization status of the first file is determined to be that it has not been synchronized by any of the other synchronization servers among the plurality of synchronization servers, except for the first synchronization server.

7. A data synchronization method, characterized in that, The method, applied to a cache server residing in a data synchronization system, which further includes a synchronization control server and multiple synchronization servers, comprises: The system receives a first request sent by a first synchronization server, which carries first file information. The first request is used to obtain the synchronization status of the first file corresponding to the first file information. The first file information is the file information selected by the first synchronization server from multiple file information in the first file list to be synchronized. The first file list to be synchronized is the file list sent by the synchronization control server to the first synchronization server. Based on the second file to be synchronized, the synchronization status of the first file corresponding to the first file information is determined. The first synchronization server is any one of the plurality of synchronization servers. The second file to be synchronized is the first file to be synchronized in the cache server. The second file to be synchronized has the same file information as the first file to be synchronized. The synchronization status of the first file is sent to the first synchronization server, so that if the first synchronization server receives feedback based on the first request that the synchronization status of the first file is not being synchronized by any of the multiple synchronization servers other than the first synchronization server, it will synchronize the first file to the file server; if the first file is successfully synchronized, the process returns to the step of selecting the first file information from the multiple file information until all files corresponding to the multiple file information are synchronized to the file server; during the synchronization of the first file, the second file information in the second list of files to be synchronized in the cache server is locked for a preset time, so that the synchronization status of the file corresponding to the second file information is modified to synchronized, and the second file information is the same as the first file information; wherein, the second list of files to be synchronized is sent to the cache server by the synchronization control server, and the second list of files to be synchronized and the first list of files to be synchronized have the same file information.

8. The data synchronization method as described in claim 7, characterized in that, The method further includes: Upon receiving a synchronization success instruction from the first synchronization server indicating successful verification of the first file, the second file information identical to the first file information is deleted from the second file to be synchronized.

9. The data synchronization method as described in claim 7, characterized in that, The method further includes: Receive a synchronization failure instruction sent by the first synchronization server, wherein the synchronization failure instruction includes breakpoint information of the first file; Record the breakpoint information in the failure handling list; Send the failure handling list to the synchronization control server.

10. The data synchronization method as described in claim 9, characterized in that, After sending the failure handling list to the synchronization control server, the method further includes: If no file information is found in the second file to be synchronized, a second request is sent to the synchronization control server, so that the synchronization control server sends the failure handling list to the first synchronization server.

11. A data synchronization device, characterized in that, The device includes: The first receiving module is used to receive a first list of files to be synchronized sent by the synchronization control server. The first list of files to be synchronized includes information on multiple files. The selection module is used to select the first file information from the plurality of file information; The first sending module is used to send a first request carrying first file information to the cache server, so that the cache server determines the synchronization status of the first file corresponding to the first file information based on the second file to be synchronized. The second file to be synchronized is the first file to be synchronized in the cache server. The second file to be synchronized has the same file information as the first file to be synchronized. The first request is used to obtain the synchronization status of the first file corresponding to the first file information. The second sending module is used to synchronize the first file to the file server when it receives feedback based on the first request that the synchronization status of the first file is not being synchronized by any of the multiple synchronization servers other than the first synchronization server. The execution module is used to return to the step of selecting the first file information from the plurality of file information if the first file is successfully synchronized, until all files corresponding to the plurality of file information are synchronized to the file server; The device is specifically used to lock the second file information in the second file list to be synchronized in the cache server according to a preset time during the process of synchronizing the first file, so that the synchronization status of the file corresponding to the second file information is modified to synchronized, and the second file information is the same as the first file information; wherein, the second file list to be synchronized is sent to the cache server by the synchronization control server, and the second file list to be synchronized and the first file list to be synchronized have the same file information.

12. A data synchronization device, characterized in that, The device includes: The second receiving module is used to receive a first request sent by the first synchronization server carrying first file information. The first request is used to obtain the synchronization status of the first file corresponding to the first file information. The first file information is the file information selected by the first synchronization server from multiple file information included in the first file list to be synchronized. The first file list to be synchronized is the file list sent by the synchronization control server to the first synchronization server. The first determining module is used to determine the synchronization status of the first file corresponding to the first file information based on the second file to be synchronized. The first synchronization server is any one of the plurality of synchronization servers, the second file to be synchronized is the first file to be synchronized in the cache server, and the second file to be synchronized has the same file information as the first file to be synchronized. The third sending module is used to send the synchronization status of the first file to the first synchronization server, so that if the first synchronization server receives feedback based on the first request that the synchronization status of the first file is not synchronized by any of the multiple synchronization servers other than the first synchronization server, it will synchronize the first file to the file server; if the first file is successfully synchronized, it will return to the step of selecting the first file information from the multiple file information until all the files corresponding to the multiple file information are synchronized to the file server; during the synchronization of the first file, the second file information in the second file list to be synchronized in the cache server is locked according to a preset time, so that the synchronization status of the file corresponding to the second file information is modified to synchronized, and the second file information is the same as the first file information; wherein, the second file list to be synchronized is sent to the cache server by the synchronization control server, and the second file list to be synchronized and the first file list to be synchronized have the same file information.

13. A terminal device, characterized in that, The device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements the data synchronization method as described in any one of claims 1-10.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the data synchronization method as described in any one of claims 1-10.

15. A computer program product, characterized in that, The computer product includes a computer program that, when executed by a processor, implements the data synchronization method according to any one of claims 1-10.

Images