Data synchronization method, device, electronic device and readable storage medium
By installing a synchronization client on the local device, monitoring file changes and automatically performing synchronization operations, the problem of real-time synchronization in the prior art is solved, and the timeliness and reliability of data synchronization is improved.
Patent Information
- Application Number
- CN202110154892.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-02-04
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2041-02-04
AI Technical Summary
The existing method of synchronizing local data to the cloud cannot achieve real-time synchronization, and requires manual operations, wastes human resources and cannot update data in a timely manner.
By installing a synchronization client on the local device, monitoring file changes, comparing local files with cloud files, and automatically performing synchronization operations, including uploading, deleting and sharding uploading, syncing according to synchronization type and error handling.
Real-time synchronization of local data to the cloud is achieved, reducing manual operations, improving the timeliness and reliability of synchronization, and avoiding resource waste.
Smart Images

Figure CN112948340B_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present disclosure relate to the technical field of data synchronization, and more specifically, to a data synchronization method, device, electronic device, and readable storage medium. Background Art
[0002] At present, there are two main methods for synchronizing local data to the cloud on the market: offline data copying is to copy local data through a disk and then synchronize it to the cloud data center to achieve synchronization between local data and the cloud, or to manually synchronize local data with the cloud in real time through online migration tools. However, due to the long offline data copying time, the offline data migration is not timely, and the online migration tool must manually submit synchronization tasks to achieve data synchronization, which wastes human resources and cannot achieve real-time synchronization of local data to the cloud. Therefore, it is necessary to provide a technical solution that can achieve real-time synchronization of local data to the cloud. Summary of the invention
[0003] One purpose of the embodiments of the present disclosure is to provide a new technical solution for a data synchronization method.
[0004] According to a first aspect of the present disclosure, a data synchronization method is provided, which is executed by a synchronization client installed on a local device, and the method includes:
[0005] Get the local file list and at least one preset synchronization type;
[0006] Monitoring whether a file in the local file list has undergone a change corresponding to any of the synchronization types, and if so, determining the file as a target file;
[0007] Compare the target file with the file in the cloud server to determine whether the target file is a file to be synchronized;
[0008] A corresponding synchronization operation is performed according to the synchronization type of the file to be synchronized, so as to realize synchronization of the file to be synchronized by the cloud server.
[0009] Optionally, performing a corresponding synchronization operation according to the synchronization type of the file to be synchronized to achieve synchronization of the file to be synchronized by the cloud server includes:
[0010] When the synchronization type of the file to be synchronized is a new type or a modified type, calling an upload interface to upload the file to be synchronized to the cloud server;
[0011] When the synchronization type of the file to be synchronized is a deletion type, a deletion interface is called to delete the file corresponding to the file to be synchronized in the cloud server.
[0012] Optionally, when the synchronization type of the file to be synchronized is a new type or a modified type, calling an upload interface to upload the file to be synchronized to the cloud server includes:
[0013] When the synchronization type of the file to be synchronized is a new type or a modified type, creating an upload task corresponding to the file to be synchronized and adding it to the upload task queue;
[0014] The upload interface is called to execute the upload task in the upload task queue in an asynchronous manner.
[0015] Optionally, performing a corresponding synchronization operation according to the synchronization type of the file to be synchronized to achieve synchronization of the file to be synchronized by the cloud server includes:
[0016] When the synchronization type of the file to be synchronized is a new type or a modified type, determining whether the size of the file to be synchronized is greater than a preset threshold;
[0017] When the size of the file to be synchronized is greater than the threshold, the file to be synchronized is fragmented to obtain a plurality of file fragments that are less than or equal to the threshold;
[0018] Uploading the plurality of file fragments to the cloud server through upload requests respectively; wherein the request header of the upload request contains the fragment number and hash value of the file fragment, so that the cloud server can verify whether the hash value of the received file fragment is consistent with the hash value in the request header and respond with an upload failure if they are inconsistent;
[0019] When receiving an upload failure response from the cloud server, the fragment number is obtained from the upload failure response as the retransmission fragment number, and the file segment corresponding to the retransmission fragment number is uploaded to the cloud server through an upload request.
[0020] Optionally, performing a corresponding synchronization operation according to the synchronization type of the file to be synchronized to achieve synchronization of the file to be synchronized by the cloud server includes:
[0021] When the synchronization type of the file to be synchronized is a new type or a modified type, determining whether the size of the file to be synchronized is greater than a preset threshold;
[0022] In the case where the size of the file to be synchronized is less than or equal to the threshold, an upload request is initiated to upload the file to be synchronized to the cloud server; wherein the request header of the upload request contains a hash value of the file to be synchronized, so that the cloud server can verify whether the hash value of the received file is consistent with the hash value in the request header, and respond with an upload failure if they are inconsistent;
[0023] When receiving an upload failure response from the cloud server, an upload request is initiated again to upload the file to be synchronized to the cloud server.
[0024] Optionally, the method further comprises:
[0025] generating an error log in the event that the synchronization operation fails to execute;
[0026] Determine the cause of failure according to the error log;
[0027] If the failure reason is insufficient local resources, prompt the user to close unnecessary processes and re-perform the synchronization operation after the user closes the unnecessary processes;
[0028] or, when the failure reason is that the cloud server installed is unavailable, re-performing the synchronization operation after a preset time interval;
[0029] Or, when the failure reason is network jitter, the synchronization operation is performed again.
[0030] Optionally, before obtaining the local file list, the method further includes:
[0031] receiving a configuration instruction, wherein the configuration instruction includes a file screening rule;
[0032] The local file list is generated according to the file screening rule.
[0033] According to a second aspect of the present disclosure, a data synchronization device is provided, the device comprising:
[0034] A monitoring unit, used to monitor whether a file in the local file list has undergone a change corresponding to any preset synchronization type, and if so, determine the file as a target file;
[0035] A comparison unit, used to compare the target file with the file in the cloud server to determine whether the target file is a file to be synchronized;
[0036] The synchronization unit is used to perform a corresponding synchronization operation according to the synchronization type of the file to be synchronized, so as to realize synchronization of the file to be synchronized by the cloud server.
[0037] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the data synchronization method described in the first aspect when executed by the processor.
[0038] According to a fourth aspect of the present disclosure, a readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the data synchronization method as described in any one of the first aspects are implemented.
[0039] One beneficial effect of the disclosed embodiments is that the local file list is monitored by a synchronization client installed on a local device, and when it is found that a file in the local file list has undergone a change corresponding to any preset synchronization type, it is further compared with the file in the cloud server to determine the files to be synchronized, and then the corresponding synchronization operation is performed according to the synchronization type of the files to be synchronized. Using this method, after the user sets the synchronization type according to his needs, the files to be synchronized can be automatically detected and the corresponding synchronization operation can be performed, which can reduce manual operations, avoid wasting human resources and improve the timeliness of synchronization.
[0040] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.
[0042] Figure 1 It is a schematic diagram of the implementation environment of the data synchronization method of the embodiment of the present disclosure and the system composition structure capable of implementing the method;
[0043] Figure 2 is a flowchart of a data synchronization method according to an embodiment of the present disclosure;
[0044] Figure 3 is a structural diagram of a data synchronization device according to an embodiment of the present disclosure;
[0045] Figure 4 It is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION
[0046] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.
[0047] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.
[0048] Technologies, methods and equipment known to persons of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods and equipment should be considered part of the specification.
[0049] In all examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.
[0050] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.
[0051] <Method Example>
[0052] like Figure 1 As shown, the data synchronization method of this embodiment involves a cloud server 2000, a local device 1000 and a network 3000. The local device 1000 can communicate with the cloud server 2000 through the communication network 3000.
[0053] The cloud server 2000 may be, for example, a blade server, a rack server, etc. deployed in the cloud. The cloud server 2000 may also be a server cluster deployed in the cloud, which is not limited here.
[0054] like Figure 1 As shown, the specific configuration of the cloud server 2000 may include but is not limited to a processor 2100, a memory 2200, an interface device 2300, a communication device 2400, a display device 2500 and an input device 2600. The processor 2100 may be, for example, a central processing unit CPU, etc. The memory 2200 may include, for example, a ROM (read-only memory), a RAM (random access memory), a non-volatile memory such as a hard disk, etc. The interface device 2300 may include, for example, a USB interface, a serial interface, etc. The communication device 2400 may be capable of wired or wireless communication, for example. The display device 2500 may be, for example, a liquid crystal display screen. The input device 2600 may include, for example, a touch screen, a keyboard, etc.
[0055] As used in the embodiment of the present invention, the memory 2200 of the cloud server 2000 is used to store instructions, and the instructions are used to control the processor 2100 to operate to support the implementation of the method according to any embodiment of the present invention. A technician can design instructions according to the scheme disclosed in the present invention. How instructions control the processor to operate is well known in the art, so it will not be described in detail here.
[0056] In one example, the cloud server 2000 is a storage server that provides object storage services. Object storage, also known as object-based storage, can be applied to more complex storage scenarios.
[0057] It should be understood by those skilled in the art that although Figure 1 Multiple devices of the cloud server 2000 are shown in the figure, but the cloud server 2000 of the embodiment of the present invention may only involve some of the devices, for example, only the processor 2100, the memory 2200 and the communication device 2400.
[0058] like Figure 1 As shown, the local device 1000 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, an input device 1600, a speaker 1700, a microphone 1800, etc. Among them, the processor 1100 may be a central processing unit CPU, a graphics processing unit GPU, a microprocessor MCU, etc., for executing a computer program, and the computer program may be written in an instruction set of an architecture such as x86, Arm, RISC, MIPS, SSE, etc. The memory 1200 includes, for example, a ROM (read-only memory), a RAM (random access memory), a non-volatile memory such as a hard disk, etc. The interface device 1300 includes, for example, a USB interface, a headphone interface, etc. The communication device 1400 can, for example, use an optical fiber or cable for wired communication, or wireless communication, and specifically can include WiFi communication, Bluetooth communication, 2G / 3G / 4G / 5G communication, etc. The display device 1500 is, for example, a liquid crystal display screen, a touch display screen, etc. The input device 1600 may include, for example, a touch screen, a keyboard, somatosensory input, etc. The user may input / output voice information through the speaker 1700 and the microphone 1800 .
[0059] The local device 1000 may be a server, a desktop computer, a portable computer, a tablet computer, a smart phone, etc., which is not limited here.
[0060] In the embodiment of the present invention, the memory 1200 of the local device 1000 is used to store instructions, and the instructions are used to control the processor 1100 to operate to support the implementation of the data synchronization method according to any embodiment of the present invention. A technician can design instructions according to the scheme disclosed in the present invention. How instructions control the processor to operate is well known in the art, so it will not be described in detail here.
[0061] It should be understood by those skilled in the art that although Figure 1Multiple devices of the local device 1000 are shown in the figure, but the local device 1000 of the embodiment of the present invention may only involve some of the devices, for example, only the processor 1100, the memory 1200, the display device 1500, the input device 1600, the communication device 1400, etc.
[0062] Figure 2 1 is a flow chart of a data synchronization method according to an embodiment of the present disclosure. The method can be executed by a local device.
[0063] In some embodiments, the method can be performed by a synchronization client installed on a local device. The synchronization client can be implemented based on a file monitoring tool (such as a WatchService tool) provided by the local operating system, so that the synchronization client can run in the form of a native application of the operating system, thereby improving the processing capability and processing speed of the synchronization client.
[0064] The method may include the following steps S102 to S105.
[0065] Step S102: Obtain a local file list and at least one preset synchronization type.
[0066] Before step S102, the method may further include step S101. Step S101, receiving a configuration instruction, and configuring synchronization-related parameters according to the configuration instruction. When the method is executed by a synchronization client installed on a local device, configuring synchronization-related parameters is to configure working parameters of the synchronization client.
[0067] In some embodiments, a local file list is specified in the configuration instruction. In one example, the local file list may be a disk of a local device, such as "disk C" or "disk D". The local file list may also be a folder in the disk, such as "folder A" in disk C. The text file list may also be a single file, such as "file B" in disk C, which is not limited in the embodiments of the present disclosure.
[0068] In some embodiments, the configuration instruction specifies a file filtering rule, and a local file list can be generated according to the file filtering rule. In one example, up to 50 file filtering rules can be set to generate a preset file list, and the number of file filtering rules and specific filtering rules can be set according to the actual needs of the user. In one example, the file filtering rule can be, for example, a regular expression. A regular expression is also called a regular expression (the full name in English is RegularExpression), which is used in the disclosed embodiments to retrieve local text that meets a certain rule.
[0069] In some embodiments, the synchronization type may be preset, such as one or more of a new type, a modification type, and a deletion type. The new type corresponds to a new file in the local file list, the modification type corresponds to a modification of an existing file in the local file list, and the deletion type corresponds to a deletion of a file in the local file list. In some embodiments, in order to ensure the data security of the cloud server and prevent files from being accidentally deleted, the default configuration is that the synchronization type includes a creation type and a modification type.
[0070] Step S103: monitor whether a file in the local file list has undergone a change corresponding to any synchronization type, and if so, determine the file as a target file.
[0071] In some embodiments, a variety of situations may cause a file to change, such as creating a new file, modifying a file, or deleting a file, which may all cause a file change. In other words, the changed file may be a newly added file, a deleted file, or a modified file. Among them, modifying a file may include modifying the file name of the file and modifying the file content.
[0072] For example, if the local file list is "Disk C", changing the file name from "File A" to "File B" can be considered as modifying the file name; changing the word "knowledge" in the content of File A to the word "property rights" can be considered as modifying the file content, which is not limited here.
[0073] For a file that has changed in the local folder, if the change mode thereof corresponds to any preset synchronization type, the changed file is determined as a target file.
[0074] For example, the preset synchronization types include a new type and a modified type, then the newly created file and the modified file are both target files, and the corresponding synchronization types are the new type and the modified type, respectively, while the deleted file is not a target file.
[0075] For example, the preset synchronization types include new creation type, modification type, and deletion type. Then, newly created files, modified files, and deleted files are all target files, and the corresponding synchronization types are new creation type, modification type, and deletion type, respectively.
[0076] Step S104: compare the target file with the file in the cloud server to determine whether the target file is a file to be synchronized.
[0077] In some embodiments, the hash value of the target file is compared with the hash value of the file in the cloud server to determine whether the target file is the file to be synchronized. The hash value of a file or data can be calculated by a hash function. A hash function is a method of creating a small digital "fingerprint" from data, usually a string of random letters and numbers. If the hash values of two files or data are the same, it can usually be considered that the two files or data are exactly the same.
[0078] In one example, the MD5 algorithm is executed on the target file to obtain an encrypted 128-bit binary array, and then the binary data is Base64 encoded to obtain the Content-MD5 value as the hash value of the target file. Based on the same method, the cloud server can obtain the hash value of each file stored in the cloud server.
[0079] In some embodiments, when the target file is a deleted file, if after comparison, it is found that there is a file in the cloud server that has the same file name and metadata information as the deleted file, then the target file is determined as a file to be synchronized, and the corresponding synchronization type is the deletion type. The metadata information may include the last modification time of the file, the file size, the hash value of the file, etc. In this way, the accuracy of the deletion can be fully guaranteed by comparing the file name and the file metadata information.
[0080] For example, after file A in the C drive of the local device is deleted by the user, it is detected whether there is a file A* in the cloud server that is consistent with the file name and metadata information of file A (the last modification time of file A, the file size and hash value of file A, etc.). If file A* exists in the cloud server, it means that file A* needs to be deleted to achieve synchronization, and the deleted file A is the file to be synchronized.
[0081] In some embodiments, when the target file is a newly added file or a modified file, if there is no file with the same file name as the target file in the cloud server, the target file is determined to be a file to be synchronized, and the corresponding synchronization type is a new type or a modified type.
[0082] For example, after a file B is added to the C drive of the local device, it is detected whether there is a file B* with the same file name as file B in the cloud server. If the file B* does not exist, it means that there is no file with the same file name as the target file in the cloud server, and the target file is the file to be synchronized, that is, file B is the file to be synchronized.
[0083] For example, if file C in the C drive of the local device is modified, check whether there is a file C* with the same file name as file C in the cloud server. If the file C* does not exist, it means that there is no file with the same file name as the target file in the cloud server, and the target file is the file to be synchronized, that is, file C is the file to be synchronized.
[0084] In some embodiments, when the target file is a newly added file or a modified file, if there is a file with the same file name as the target file in the cloud server, it is further determined whether the hash value of the file with the same file name is different from the hash value of the target file. If they are different, the target file is determined to be a file to be synchronized.
[0085] For example, when file E in the local device is a newly added file or a modified file, if there is a file E* with the same file name as file E in the cloud server, it is necessary to obtain the hash values of file E and file E* respectively. If the hash values of file E and file E* are different, it is determined that file E is the file to be synchronized. Although the file name of file E* in the cloud server is the same as the file name of file E, it cannot be guaranteed that the content in file E is the same as the content in file E*, which means that file E and file E* may not be the same file, and it is necessary to obtain the hash values of file E and file E* to further confirm whether file E is the file to be synchronized.
[0086] In some embodiments, when the target file is a newly added file or a modified file, if there is a file with the same file name as the target file in the cloud server, it is further determined whether the last modification time of the file with the same file name is different from the last modification time of the target file. If they are different, the target file is determined to be a file to be synchronized.
[0087] For example, when file E in the local device is a newly added file or a modified file, if there is a file E* with the same file name as file E in the cloud server, it is necessary to obtain the last modification time of file E and file E* respectively (which can be obtained from the metadata of the files). If the last modification time of file E and file E* is different, file E is determined to be the file to be synchronized.
[0088] Step S105: Execute corresponding synchronization operations according to the synchronization type of the file to be synchronized, so as to realize synchronization of the file to be synchronized by the cloud server.
[0089] When the synchronization type of the file to be synchronized is the deletion type, the deletion interface is called to delete the file corresponding to the file to be synchronized in the cloud server.
[0090] In one example, a deletion task queue based on a serial execution mechanism is created in advance. When the synchronization type of the file to be synchronized is the deletion type, a deletion task corresponding to the file to be synchronized is created and added to the deletion task queue, and the deletion interface is called to execute the deletion task in the deletion task queue in a serial manner, that is, the deletion tasks in the deletion task queue are executed sequentially in the order in which the deletion tasks are added to the deletion task queue.
[0091] Alternatively, a deletion task queue based on an asynchronous execution mechanism is created in advance. When the synchronization type of the file to be synchronized is the deletion type, a deletion task corresponding to the file to be synchronized is created and added to the deletion task queue, and the deletion interface is called to execute the deletion task in the deletion task queue asynchronously.
[0092] When the synchronization type of the to-be-synchronized file is a newly added type or a modified type, the upload interface is called to upload the to-be-synchronized file to the cloud server. In the application scenario of object storage, it is difficult for a local device to modify files in the cloud server. In the disclosed embodiment, for the modified type of to-be-synchronized file, the cloud server synchronizes the to-be-synchronized file by uploading and overwriting the original file.
[0093] In one example, an upload task queue based on a serial execution mechanism is created in advance. When the synchronization type of the file to be synchronized is a new addition type or a modified type, an upload task corresponding to the file to be synchronized is created and added to the upload task queue, and the upload interface is called to execute the upload tasks in the upload task queue in a serial manner, that is, the upload tasks in the upload task queue are executed sequentially in the order in which the upload tasks are added to the upload task queue.
[0094] Alternatively, an upload task queue based on an asynchronous execution mechanism is created in advance. When the synchronization type of the file to be synchronized is a new addition type or a modified type, an upload task corresponding to the file to be synchronized is created and added to the upload task queue, and the upload interface is called to execute the upload task in the upload task queue asynchronously.
[0095] That is to say, in the disclosed embodiment, synchronous tasks can be processed based on the asynchronous execution mechanism, that is, different tasks in the task queue can be executed alternately or simultaneously, and whether to start the execution process of the task does not depend on the completion of the previous task. The disclosed embodiment utilizes the concurrent execution task capability and asynchronous callback capability of the asynchronous execution mechanism to reduce the occurrence of task blocking, which is conducive to improving the reliability of the synchronization scheme.
[0096] In one example, when the file to be synchronized is a newly added file or a modified file, after synchronizing the file to be synchronized to the cloud server, it is necessary to obtain the hash value of the synchronization file corresponding to the file to be synchronized in the cloud server, and compare the hash value of the synchronization file with the hash value of the file to be synchronized to determine whether the hash value of the synchronization file is the same as the hash value of the file to be synchronized. If the hash value of the synchronization file is different from the hash value of the file to be synchronized, it means that the previous synchronization was not successful, and the file to be synchronized needs to be synchronized to the cloud server again. In this way, the success of synchronization can be fully guaranteed and the synchronization success rate can be improved.
[0097] In an example, when the synchronization type of the file to be synchronized is a new addition type or a modified type, it can be determined whether to transmit the file to be synchronized in segments according to the size of the file to be synchronized.
[0098] First, when the synchronization type of the file to be synchronized is a new addition type or a modified type, it is determined whether the size of the file to be synchronized is greater than a preset threshold.
[0099] When the size of the file to be synchronized is greater than the threshold, the synchronization process includes steps S1051 - S1053 .
[0100] Step S1051: Segment the file to be synchronized to obtain a plurality of file segments whose size is less than or equal to a threshold.
[0101] Step S1052: Upload the multiple file segments to the cloud server through upload requests respectively.
[0102] The request header of the upload request contains the fragment number and hash value of the file fragment. After receiving the file fragment, the cloud server verifies whether the hash value of the received file fragment is consistent with the hash value in the request header. If they are inconsistent, it means that the file fragment received by the cloud server is incomplete or an error has occurred, and an upload failure response is issued to notify the local device to re-upload the file fragment.
[0103] Step S1053: upon receiving an upload failure response from the cloud server, obtaining a shard number from the upload failure response as a retransmission shard number, and uploading the file segment corresponding to the retransmission shard number to the cloud server through an upload request.
[0104] After receiving all the file fragments and passing the consistency check, the cloud server can merge the file fragments into a complete file according to the fragment number.
[0105] When the size of the file to be synchronized is less than or equal to the threshold, the synchronization process includes steps S1054-S1055.
[0106] Step S1054: Initiate an upload request to upload the file to be synchronized to the cloud server.
[0107] The request header of the upload request contains the hash value of the file to be synchronized, so that the cloud server can verify whether the hash value of the received file is consistent with the hash value in the request header and respond with an upload failure if they are inconsistent. If they are inconsistent, it means that the file received by the cloud server is incomplete or an error has occurred, and an upload failure response is issued to notify the local device to re-upload the file to be synchronized.
[0108] Step S1055: upon receiving an upload failure response from the cloud server, initiating an upload request again to upload the file to be synchronized to the cloud server.
[0109] In the embodiment of the present disclosure, when the synchronization type of the file to be synchronized is a new addition type or a modified type, when the file to be synchronized is relatively large (that is, larger than a preset threshold), the segmented transmission mode is selected for uploading. When the segmented upload fails, it is only necessary to re-upload the corresponding file fragment, and there is no need to re-upload the entire file to be synchronized, so as to prevent wasting bandwidth resources.
[0110] In one example, the method further includes: generating an error log when the synchronization operation fails to be executed, and determining the cause of the failure according to the error log.
[0111] When the failure reason is insufficient local resources, the user is prompted to close non-essential processes and the synchronization operation is performed again after the user closes the non-essential processes.
[0112] When the failure reason is that the synchronization server installed on the cloud server is unavailable, the synchronization operation is performed again after a preset time interval.
[0113] If the failure is caused by network jitter, re-perform the synchronization operation.
[0114] If the cause of the failure cannot be determined, restart the sync client and try the sync operation again after the restart.
[0115] That is to say, the data synchronization method of the embodiment of the present disclosure has an automatic error correction and remediation mechanism, which can try to solve the problem causing the failure and re-upload when the synchronization operation fails, thereby improving the synchronization reliability.
[0116] In one example, each synchronization operation and the hash value of the file to be synchronized are recorded and stored in the local operation record database for subsequent search and use of synchronization records. After the consistency check passes, the cloud server records the hash value of the synchronized file for subsequent search or comparison.
[0117] In some embodiments, the cloud storage server may belong to a third-party cloud vendor, and the configuration instructions also include configurations for the third-party cloud vendor.
[0118] In some embodiments, according to configuration instructions, different or the same maximum bandwidth within a preset time range can be set for different third-party cloud vendors to implement speed limit settings to make reasonable use of network resources.
[0119] In some embodiments, according to configuration instructions, different or the same maximum number of connections can be set for different third-party cloud vendors. By setting the maximum number of connections, dynamic adjustment of the real-time synchronization tool can be achieved, and the number of connections can be determined based on the usage and performance of the local machine.
[0120] In some embodiments, according to the configuration instructions, a signature key of a third-party cloud vendor may also be set for user identification to ensure data security during the synchronization of files to be synchronized.
[0121] In some embodiments, according to the configuration instructions, the synchronous transmission can also be set to use http or https transmission mode. Generally, https transmission mode is used by default.
[0122] In some embodiments, according to the configuration instructions, it can also be set whether the synchronous transmission needs to be encrypted. In order to ensure the security of the files to be synchronized during the synchronous transmission, the encrypted synchronous transmission setting is generally adopted by default.
[0123] In some embodiments, according to the configuration instructions, the transmission speed and transmission mode can also be set, and when synchronizing the to-be-synchronized file to the cloud server, the to-be-synchronized file is synchronized to the cloud server according to the transmission speed and transmission mode. The transmission mode can include, for example, fragmented transmission, serial transmission, or asynchronous transmission.
[0124] The data synchronization method of the disclosed embodiment monitors the local file list through a synchronization client installed on a local device. When it is found that a file in the local file list has undergone a change corresponding to any preset synchronization type, it is further compared with the file in the cloud server to determine the to-be-synchronized files, and then the corresponding synchronization operation is performed according to the synchronization type of the to-be-synchronized files. Using this method, after the user sets the synchronization type according to his or her needs, the files to be synchronized can be automatically detected and the corresponding synchronization operation can be performed, which can reduce manual operations, avoid wasting human resources and improve the timeliness of synchronization.
[0125] <Device Example>
[0126] Figure 3 Schematic diagram of a data synchronization device to which the present disclosure embodiment can be applied. Figure 3As shown, the data synchronization device includes a monitoring unit 301 , a comparison unit 302 and a synchronization unit 303 .
[0127] A monitoring unit 301 is used to monitor whether a file in the local file list has undergone a change corresponding to any preset synchronization type, and if so, determine the file as a target file;
[0128] A comparison unit 302 is used to compare the target file with the file in the cloud server to determine whether the target file is a file to be synchronized;
[0129] The synchronization unit 303 is used to perform corresponding synchronization operations according to the synchronization type of the file to be synchronized, so as to realize synchronization of the file to be synchronized by the cloud server.
[0130] In an example, the preset synchronization types include new creation type, modification type, and deletion type. Then, newly created files, modified files, and deleted files are all target files, and the corresponding synchronization types are new creation type, modification type, and deletion type, respectively.
[0131] In one example, the data synchronization device further includes a configuration management module. The configuration management module is used to receive a configuration instruction, wherein the configuration instruction includes a file screening rule; and generate a local file list according to the file screening rule. Alternatively, the configuration instruction specifies a local file list. In one example, the configuration instruction also includes other configuration information mentioned in the foregoing content, and the configuration management module is used to control the synchronization process according to the configuration information, which will not be repeated here.
[0132] In one example, the synchronization unit includes a first synchronization unit and a second synchronization unit.
[0133] The first synchronization unit is used to call a deletion interface to delete a file corresponding to the file to be synchronized in the cloud server when the synchronization type of the file to be synchronized is a deletion type.
[0134] The second synchronization unit is used to call the upload interface to upload the file to be synchronized to the cloud server when the synchronization type of the file to be synchronized is a new type or a modified type.
[0135] In one example, the first synchronization unit includes a deletion task creation submodule and a deletion task execution submodule.
[0136] The deletion task creation submodule is used to create a deletion task corresponding to the file to be synchronized and add it to the deletion task queue when the synchronization type of the file to be synchronized is the deletion type.
[0137] The deletion task execution submodule is used to call the deletion interface to execute the upload task in the deletion task queue asynchronously.
[0138] In one example, the second synchronization unit includes an upload task creation module and an upload task execution submodule.
[0139] The upload task creation submodule is used to create an upload task corresponding to the file to be synchronized and add it to the upload task queue when the synchronization type of the file to be synchronized is a new type or a modified type.
[0140] The upload interface execution submodule is used to call the upload interface to execute the upload task in the upload task queue in an asynchronous manner.
[0141] The data synchronization device of the disclosed embodiment can process synchronization tasks based on an asynchronous execution mechanism. Different tasks in the task queue can be executed alternately or simultaneously. Whether to start the execution process of the task does not depend on the completion of the previous task. The ability of concurrent execution of tasks and asynchronous callback of the asynchronous execution mechanism can reduce the occurrence of task blocking, which is conducive to improving synchronization reliability.
[0142] In one example, the second synchronization unit includes a comparison submodule, a slicing submodule, a first uploading submodule, and a second uploading submodule.
[0143] The comparison module is used to determine whether the size of the file to be synchronized is greater than a preset threshold when the synchronization type of the file to be synchronized is a new type or a modified type.
[0144] The fragmentation module is used to fragment the file to be synchronized to obtain multiple file fragments less than or equal to the threshold value when the size of the file to be synchronized is greater than a threshold value.
[0145] The first upload submodule is used to upload multiple file fragments to the cloud server through upload requests respectively; wherein, the request header of the upload request contains the fragment number and hash value of the file fragment, so that the cloud server can verify whether the hash value of the received file fragment is consistent with the hash value in the request header and respond to the upload failure if they are inconsistent; and when receiving the upload failure response from the cloud server, the fragment number is obtained from the upload failure response as the retransmission fragment number, and the file fragment corresponding to the retransmission fragment number is uploaded to the cloud server through the upload request.
[0146] The second upload submodule is used to initiate an upload request to upload the file to be synchronized to the cloud server when the size of the file to be synchronized is less than or equal to a threshold value; wherein the request header of the upload request contains a hash value of the file to be synchronized, so that the cloud server can verify whether the hash value of the received file is consistent with the hash value in the request header and respond with an upload failure if they are inconsistent; and, upon receiving an upload failure response from the cloud server, initiate an upload request again to upload the file to be synchronized to the cloud server.
[0147] The data synchronization device of the disclosed embodiment selects the segmented transmission mode for uploading newly added or modified files to be synchronized when the files to be synchronized are relatively large. When the segmented upload fails, it is only necessary to re-upload the corresponding file fragments, and there is no need to re-upload the entire file to be synchronized, so as to prevent wasting bandwidth resources.
[0148] In one example, the data synchronization device further includes a log generation module, a log analysis module, and an exception handling module;
[0149] A log generation module is used to generate error logs when the synchronization operation fails.
[0150] Log analysis module, used to determine the cause of failure based on error logs;
[0151] The exception handling module is used to prompt the user to close non-essential processes and re-synchronize the operation after the user closes the non-essential processes when the reason for failure is insufficient local resources; or, when the reason for failure is that the cloud server installed is unavailable, re-synchronize the operation after a preset time interval; or, when the reason for failure is network jitter, re-synchronize the operation.
[0152] The data synchronization device of the disclosed embodiment has an automatic error correction and remediation mechanism, which can try to solve the problem causing the failure and re-upload when the synchronization operation fails, thereby improving the synchronization reliability.
[0153] The data synchronization device of the disclosed embodiment can automatically monitor the local file list. When it is found that the files in the local file list have changed corresponding to any preset synchronization type, it is further compared with the files in the cloud server to determine the files to be synchronized, and then the corresponding synchronization operations are performed according to the synchronization type of the files to be synchronized. Using this method, after the user sets the synchronization type according to his needs, the files to be synchronized can be automatically detected and the corresponding synchronization operations can be performed, which can reduce manual operations, avoid wasting human resources and improve the timeliness of synchronization.
[0154] The data synchronization device of the disclosed embodiment modularizes the monitoring function, the comparison function, and the synchronous execution function, thereby reducing the coupling between the various functions and facilitating users to expand according to their needs.
[0155] The data synchronization device of the disclosed embodiment can support synchronization management of cloud servers of different third-party cloud vendors.
[0156] Figure 4 Schematic diagram of an electronic device structure to which the present disclosure embodiment can be applied. Figure 4 As shown, the electronic device 501 includes: a memory 5011, a processor 5012, and a computer program stored in the memory 5011 and executable on the processor 5012, wherein when the computer program is executed by the processor 5012, the steps of any data synchronization method in any of the above method embodiments are implemented.
[0157] In an embodiment of the present invention, a readable storage medium is further provided, characterized in that a computer program is stored on the readable storage medium, and when the computer program is executed by a processor, the steps of any data synchronization method in any of the above method embodiments are implemented.
[0158] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.
[0159] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.
[0160] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.
[0161] Here, various aspects of the present invention are described with reference to the flow chart and / or block diagram of the method, system and computer program product according to the embodiment of the present invention. It should be understood that each square frame of the flow chart and / or block diagram and the combination of each square frame in the flow chart and / or block diagram can be realized by computer readable program instructions.
[0162] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0163] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0164] The flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of an instruction, and a part of the module, program segment or instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of the boxes in the block diagram and / or the flowchart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.
[0165] Embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terms used herein are selected to best explain the principles of the embodiments, practical applications, or technical improvements in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.
Claims
1. A data synchronization method, characterized in that: The method is executed by a synchronization client installed on a local device, and includes: Get the local file list and at least one preset synchronization type; Monitoring whether a file in the local file list has undergone a change corresponding to any of the synchronization types, and if so, determining the file as a target file; Compare the target file with the file in the cloud server to determine whether the target file is a file to be synchronized; Executing a corresponding synchronization operation according to the synchronization type of the file to be synchronized, so as to realize synchronization of the file to be synchronized by the cloud server; The step of comparing the target file with the file in the cloud server to determine whether the target file is a file to be synchronized includes: When the target file is a newly added file or a modified file, and there is no file with the same file name as the target file in the cloud server, the target file is determined to be a file to be synchronized, and the corresponding synchronization type is a new type or a modified type; When the target file is a deleted file and there is a file in the cloud server that has the same file name and metadata information as the deleted file, the target file is determined to be a file to be synchronized, and the corresponding synchronization type is a deletion type; Before obtaining the local file list, the method further includes: receiving a configuration instruction, wherein the configuration instruction includes a file screening rule; The local file list is generated according to the file screening rule.
2. The method according to claim 1, characterized in that: The performing a corresponding synchronization operation according to the synchronization type of the file to be synchronized, so as to realize synchronization of the file to be synchronized by the cloud server, includes: When the synchronization type of the file to be synchronized is a new type or a modified type, calling an upload interface to upload the file to be synchronized to the cloud server; When the synchronization type of the file to be synchronized is a deletion type, a deletion interface is called to delete the file corresponding to the file to be synchronized in the cloud server.
3. The method according to claim 2, characterized in that When the synchronization type of the file to be synchronized is a new type or a modified type, calling an upload interface to upload the file to be synchronized to the cloud server includes: When the synchronization type of the file to be synchronized is a new type or a modified type, creating an upload task corresponding to the file to be synchronized and adding it to the upload task queue; The upload interface is called to execute the upload task in the upload task queue in an asynchronous manner.
4. The method according to claim 1, characterized in that: The performing a corresponding synchronization operation according to the synchronization type of the file to be synchronized, so as to realize synchronization of the file to be synchronized by the cloud server, includes: When the synchronization type of the file to be synchronized is a new type or a modified type, determining whether the size of the file to be synchronized is greater than a preset threshold; When the size of the file to be synchronized is greater than the threshold, the file to be synchronized is fragmented to obtain a plurality of file fragments that are less than or equal to the threshold; Uploading the plurality of file fragments to the cloud server through upload requests respectively; wherein the request header of the upload request contains the fragment number and hash value of the file fragment, so that the cloud server can verify whether the hash value of the received file fragment is consistent with the hash value in the request header and respond with an upload failure if they are inconsistent; When receiving an upload failure response from the cloud server, the fragment number is obtained from the upload failure response as the retransmission fragment number, and the file segment corresponding to the retransmission fragment number is uploaded to the cloud server through an upload request.
5. The method according to claim 1, characterized in that The performing a corresponding synchronization operation according to the synchronization type of the file to be synchronized, so as to realize synchronization of the file to be synchronized by the cloud server, includes: When the synchronization type of the file to be synchronized is a new type or a modified type, determining whether the size of the file to be synchronized is greater than a preset threshold; In the case where the size of the file to be synchronized is less than or equal to the threshold, an upload request is initiated to upload the file to be synchronized to the cloud server; wherein the request header of the upload request contains a hash value of the file to be synchronized, so that the cloud server can verify whether the hash value of the received file is consistent with the hash value in the request header, and respond with an upload failure if they are inconsistent; When receiving an upload failure response from the cloud server, an upload request is initiated again to upload the file to be synchronized to the cloud server.
6. The method according to claim 1, characterized in that The method further comprises: Generate error logs in case of failure in synchronization operations; Determine the cause of failure according to the error log; If the failure reason is insufficient local resources, prompt the user to close unnecessary processes and re-perform the synchronization operation after the user closes the unnecessary processes; or, when the failure reason is that the cloud server installed is unavailable, re-performing the synchronization operation after a preset time interval; Or, when the failure reason is network jitter, the synchronization operation is performed again.
7. A data synchronization device, characterized in that: The device comprises: A monitoring unit, used to monitor whether a file in the local file list has undergone a change corresponding to any preset synchronization type, and if so, determine the file as a target file; A comparison unit, used to compare the target file with the file in the cloud server to determine whether the target file is a file to be synchronized; A synchronization unit, configured to perform a corresponding synchronization operation according to the synchronization type of the file to be synchronized, so as to realize synchronization of the file to be synchronized by the cloud server; A configuration management module, configured to receive configuration instructions, wherein the configuration instructions include file screening rules; and generate a local file list according to the file screening rules; The comparison unit is further configured to determine that the target file is a file to be synchronized and the corresponding synchronization type is a new type or a modified type when the target file is a newly added file or a modified file and there is no file with the same file name as the target file in the cloud server; When the target file is a deleted file and there is a file in the cloud server that has the same file name and metadata information as the deleted file, the target file is determined to be a file to be synchronized, and the corresponding synchronization type is a deletion type.
8. An electronic device, characterized in that: The electronic device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the data synchronization method according to any one of claims 1 to 6 when executed by the processor.
9. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data synchronization method according to any one of claims 1 to 6 are implemented.
