File synchronization method and device, equipment, storage medium and program product
By traversing file transfer records and creating transfer tasks, files are synchronized from the original system to the new system, solving the problems of time-consuming, labor-intensive, and error-prone data synchronization during system migration, and achieving efficient and accurate data transmission.
Patent Information
- Application Number
- CN202511664456.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-13
- Publication Date
- 2026-02-27
AI Technical Summary
During system migration, existing technologies are time-consuming and labor-intensive when synchronizing large amounts of data files to the new system, and are prone to omissions and errors, affecting data consistency and integrity.
By iterating through the file transfer records within a set time period, parsing the file synchronization status, creating transfer tasks and synchronizing the files to the new system, updating the synchronization status, and so on, the processing of all records is completed.
It improves the speed and accuracy of file synchronization, ensuring data consistency and integrity.
Smart Images
Figure CN121585654A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of file synchronization technology, and in particular to a file synchronization method, apparatus, device, storage medium and program product. Background Technology
[0002] With the changing global technological environment and increasing security threats, and adhering to the principle of security and controllability, it has become a trend for fintech companies to migrate their existing systems to more secure and reliable new systems. For systems that provide data processing services, synchronizing a large number of data files to the new system during the migration process is a complex task. The commonly used solution is to copy the files from the old system to the new system during a shutdown and service switchover. However, files transferred from upstream to the old system during the switchover will be affected, which is not only time-consuming and labor-intensive but also prone to omissions and errors. Summary of the Invention
[0003] This invention provides a file synchronization method, apparatus, device, storage medium, and program product, which can improve the speed and accuracy of file synchronization and ensure data consistency and integrity.
[0004] In a first aspect, embodiments of the present invention provide a file synchronization method, including:
[0005] When the file synchronization task is started, it iterates through the file transfer records generated within a set time period; wherein, the file transfer records are records of files transferred from the upstream system to the original system within the set time period; wherein, the file transfer records include a unique identifier for the file; the unique identifier includes at least one of the following: upstream system identifier ID, file generation timestamp, file checksum, and file name;
[0006] Parse the traversed file transfer records to determine the synchronization status of the files in the file transfer records; wherein, the synchronization status includes any of the following: not synchronized, synchronizing, synchronization completed, synchronization failed, and pending retransmission;
[0007] If the file in the file transfer record is in a state of unsynchronized or pending retransmission, a transfer task is created based on the unique identifier.
[0008] Based on the transmission task, the files in the file transfer record are synchronized to the new system, and the synchronization status of the files in the file transfer record is updated to "synchronizing".
[0009] Based on the synchronization results fed back by the new system, the synchronization status of the files in the file transfer record is updated, and the next file transfer record is traversed until all file transfer records generated within the set time period are traversed.
[0010] Secondly, embodiments of the present invention also provide a file synchronization device, comprising:
[0011] The file transfer record traversal module is used to traverse the file transfer records generated within a set time period when the file synchronization task is started; wherein, the file transfer record is the record of the file transferred from the upstream system to the original system within the set time period; wherein, the file transfer record includes a unique identifier of the file; the unique identifier includes at least one of the following: upstream system identifier ID, file generation timestamp, file checksum, and file name;
[0012] The synchronization status determination module is used to parse the traversed file transfer records and determine the synchronization status of the files in the file transfer records; wherein, the synchronization status includes any one of the following: not synchronized, synchronizing, synchronization completed, synchronization failed, and pending retransmission;
[0013] The transmission task creation module is used to create a transmission task based on the unique identifier if the synchronization status of the file in the file transmission record is out of synchronization or pending retransmission.
[0014] The file synchronization module is used to synchronize the files in the file transfer record to the new system based on the transfer task, and update the synchronization status of the files in the file transfer record to "synchronizing".
[0015] The synchronization status update module is used to continue updating the synchronization status of the files in the file transfer record based on the synchronization results fed back by the new system, and to continue traversing the next file transfer record until all file transfer records generated within the set time period have been traversed.
[0016] Thirdly, embodiments of the present invention also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the file synchronization method as described in any of the embodiments of the present invention.
[0017] Fourthly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the file synchronization method as described in any of the embodiments of the present invention.
[0018] Fifthly, embodiments of the present invention also provide a computer program product, including a computer program that, when executed by a processor, implements the file synchronization method as described in any of the embodiments of the present invention.
[0019] This invention discloses a file synchronization method, apparatus, device, storage medium, and program product. When a file synchronization task is started, it iterates through file transfer records generated within a set time period. Each file transfer record is a record of files transferred from an upstream system to an original system within the set time period. Each file transfer record includes a unique identifier for the file. The unique identifier includes at least one of the following: upstream system identifier ID, file generation timestamp, file checksum, and file name. The iterated file transfer records are parsed to determine the synchronization status of the files within them. The synchronization status includes any one of the following: not synchronized, synchronizing, synchronized, synchronized, synchronized failed, and pending retransmission. If the synchronization status of a file in a file transfer record is not synchronized or pending retransmission, a transfer task is created based on the unique identifier. The files in the file transfer records are synchronized to the new system based on the transfer task, and the synchronization status of the files in the file transfer records is updated to synchronizing. Based on the synchronization results fed back by the new system, the synchronization status of the files in the file transfer records is continuously updated, and the next file transfer record is traversed until all file transfer records generated within the set time period are traversed. The file synchronization method provided in this invention creates a transmission task based on a unique identifier when the file synchronization status in the file transfer record is unsynchronized or pending retransmission, so as to synchronize the file from the original system to the new system. This can improve the speed and accuracy of file synchronization, and ensure the consistency and integrity of the data. Attached Figure Description
[0020] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0021] Figure 1 A flowchart of a file synchronization method provided in an embodiment of the present invention;
[0022] Figure 2 A flowchart of a file synchronization method provided in an embodiment of the present invention;
[0023] Figure 3 This is a schematic diagram of a file synchronization device provided in an embodiment of the present invention;
[0024] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation
[0025] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.
[0026] It should be noted that similar reference numerals and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this invention, terms such as "first" and "second" are used only to distinguish descriptions and should not be construed as indicating or implying relative importance. The acquisition, storage, use, and processing of data in the technical solutions of this application comply with relevant national laws and regulations. It should be noted that in the embodiments of this application, certain existing industry solutions such as software, components, and models may be mentioned. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solutions of this application, and do not imply that the applicant has already used or necessarily used such solutions.
[0027] Figure 1 This is a flowchart of a file synchronization method provided by an embodiment of the present invention. This method is applicable to the process of synchronizing files from an original system to a new system. The method can be executed by a file synchronization device, which can be implemented in software and / or hardware, optionally through an electronic device, such as a mobile terminal, PC, or server. Figure 1 As shown, the method includes:
[0028] S110: When the file synchronization task starts, iterate through the file transfer records generated within the set time period.
[0029] The file transfer record is a record of files transferred from the upstream system to the original system within a set time period. This set time period can be a recent historical period, such as the last 3 days. The file synchronization task is started periodically, for example, every 5 minutes. The file transfer record includes a unique identifier for the file, which includes at least one of the following: upstream system identifier ID, file generation timestamp, file checksum, and file name. The file checksum can be represented by the file's MD5 value. The unique identifier for the file is generated by the upstream system that sent the file. In this embodiment, the multi-level unique identifier ensures that files do not conflict and are traceable.
[0030] Specifically, when a file synchronization task is started, the method for iterating through the file transfer records generated within a set time period can be: starting the file synchronization task every set time interval; or iterating through the file transfer records generated within a set time period when the file synchronization task is started.
[0031] In this embodiment, a file synchronization task can be started periodically using a command-line tool (such as crontab) that can be set to execute tasks periodically. When the file synchronization task starts, the file transfer records generated within the set time period are extracted based on the reception time in the file transfer records, and these file transfer records are traversed sequentially.
[0032] S120, parse the traversed file transfer records and determine the synchronization status of the files in the file transfer records.
[0033] The synchronization status includes any of the following: not synchronized, synchronizing, synchronizing completed, synchronizing failed, and waiting to be retransmitted.
[0034] One way to determine the synchronization status of a file in a file transfer record is to read the synchronization status corresponding to the file in the file transfer record stored in the synchronization status table.
[0035] Optionally, the method further includes the following steps: pre-establishing a synchronization state table; recording the synchronization state of files in the file transfer record into the synchronization state table.
[0036] S130, if the file synchronization status in the file transfer record is out of synchronization or pending retransmission, then a transfer task is created based on the unique identifier.
[0037] Specifically, the methods for creating a transfer task based on a unique identifier can be: obtaining the file transfer information from the file transfer record; and creating a transfer task based on the unique identifier and the transfer information.
[0038] The transmitted information includes the original system path, the new system path, the original system nodes, and the new system nodes.
[0039] In this embodiment, the process of creating a transmission task based on a unique identifier and transmission information can be as follows: create a file transmission process based on the unique identifier and transmission information, and execute the transmission task through this process.
[0040] Optionally, the method for creating a transmission task based on a unique identifier can be as follows: if the synchronization status of the file in the file transmission record is pending retransmission, then obtain the time interval between the last transmission of the file and the current time in the file transmission record; if the time interval is greater than a set threshold, then create a transmission task based on the unique identifier.
[0041] In this embodiment, to avoid repeated retransmissions of files in a short period of time, which would increase the network load, the retransmission interval between two adjacent retransmissions of the same file must be greater than a set threshold. If the time interval between the last file transmission and the current time is less than or equal to the set threshold, the file transmission record is skipped, and the process continues to traverse the next file transmission record. If the time interval between the last file transmission and the current time is greater than the set threshold, a transmission task is created based on a unique identifier to retransmit the file.
[0042] S140, based on the transfer task, synchronize the files in the file transfer record to the new system, and update the synchronization status of the files in the file transfer record to synchronized.
[0043] In this embodiment, when the files in the file transfer record are synchronized to the new system, the synchronization status of the files in the file transfer record is updated to "synchronizing" to avoid duplicate transfers.
[0044] Optionally, the method for synchronizing files in the file transfer record to the new system based on the transfer task can be: adding the transfer task to the synchronization queue according to the file priority; when a transfer task is read from the synchronization queue, retrieving the file based on the unique identifier, and synchronizing the retrieved file to the new system.
[0045] File priority can be categorized according to file importance, for example: core business files (such as order data) > ordinary business files (such as logs) > redundant files (such as backup copies). Adding transmission tasks to the synchronization queue according to file priority can be done by adding the task before tasks with lower priority and after tasks with higher priority. In other words, output tasks in the synchronization queue are arranged from highest to lowest priority. The advantage of this approach is that high-priority tasks are synchronized first, preventing ordinary files from consuming resources and causing delays in core files.
[0046] Optionally, the method for synchronizing files in the file transfer record to the new system based on the transfer task can be: splitting the files in the file transfer record into multiple fragments and assigning numbers to the multiple fragments; synchronizing the multiple fragments to the new system based on the transfer task.
[0047] If the file size exceeds a set value, the file in the file transfer record is split into multiple fragments, and each fragment is assigned a number according to its position in the file. In this embodiment, during the process of synchronizing multiple fragments to the new system, the numbers of the fragments that have been synchronized can be recorded.
[0048] Optionally, the following steps may also be included: if the file synchronization status in the file transfer record is pending retransmission, then obtain the fragments that have not been synchronized; synchronize the fragments that have not been synchronized to the new system.
[0049] In this embodiment, fragments that have not been synchronized can be determined based on the recorded fragment numbers that have been synchronized. In this way, fragments that have not been synchronized can be synchronized to the new system, and the entire file can be wirelessly retransmitted, reducing bandwidth waste.
[0050] S150, based on the synchronization results fed back by the new system, continue to update the synchronization status of the files in the file transfer record, and continue to traverse the next file transfer record until the file transfer records generated within the set time period are traversed.
[0051] The synchronization results reported by the new system include whether synchronization is complete or failed.
[0052] Optionally, the method for continuing to update the synchronization status of files in the file transfer record based on the synchronization result fed back by the new system can be as follows: if the synchronization result fed back by the new system is synchronization complete, then update the synchronization status of the files in the file transfer record to synchronization complete; if the synchronization result fed back by the new system is synchronization failure or no synchronization result fed back by the new system is received within a set time, then determine the number of file retransmissions in the file transfer record; if the number of retransmissions is less than the set number, then update the synchronization status of the files in the file transfer record to pending retransmission; if the number of retransmissions is greater than or equal to the set number, then update the synchronization status of the files in the file transfer record to synchronization failure.
[0053] In this embodiment, if the synchronization result reported by the new system is a synchronization failure or no synchronization result is received from the new system within a set time period, it is considered that the synchronization has failed. In this case, the retransmission mechanism for the file that failed to synchronize can be automatically started based on the set number of retransmissions.
[0054] Optionally, the method for continuing to update the synchronization status of files in the file transfer record based on the synchronization results fed back by the new system can be as follows: if the synchronization result fed back by the new system is synchronization failure, then determine the reason for the synchronization failure; if the reason for the synchronization failure is caused by a recoverable exception, then update the synchronization status of the files in the file transfer record to pending retransmission; if the reason for the synchronization failure is caused by an unrecoverable exception, then update the synchronization status of the files in the file transfer record to synchronization failure.
[0055] Among these, recoverable anomalies include network disconnection and / or system crash; unrecoverable causes include file corruption and / or format errors. In this embodiment, if the synchronization failure is caused by recoverable anomalies such as network disconnection and / or system crash, the file's synchronization status in the file transfer record is updated to "Pending Retransmission" to automatically initiate file retransmission. If the synchronization failure is caused by file corruption and / or format errors, the file's synchronization status in the file transfer record is updated to "Synchronization Failed," requiring further investigation and resolution of the cause of the failure.
[0056] Optionally, after updating the file synchronization status in the file transfer record to synchronization failure, the following steps are also included: generating a prompt message and recording the reason for the synchronization failure.
[0057] In this embodiment, recording the reason for synchronization failure can help users quickly locate and resolve the problem, thereby increasing the probability of successful synchronization next time.
[0058] In this embodiment, when a file synchronization task is started, the file transfer records generated within a set time period are traversed. Each file transfer record is a record of files transferred from an upstream system to the original system within that time period. Each file transfer record includes a unique identifier for the file. The unique identifier includes at least one of the following: upstream system identifier ID, file generation timestamp, file checksum, and file name. The traversed file transfer records are parsed to determine the synchronization status of the files within them. The synchronization status includes any one of the following: not synchronized, synchronizing, synchronized, synchronized, synchronized failed, and pending retransmission. If the synchronization status of a file in a file transfer record is not synchronized or pending retransmission, a transfer task is created based on the unique identifier. The files in the file transfer records are synchronized to the new system based on the transfer task, and the synchronization status of the files in the file transfer records is updated to synchronizing. Based on the synchronization results fed back by the new system, the synchronization status of the files in the file transfer records is continuously updated, and the next file transfer record is traversed until all file transfer records generated within the set time period are traversed. The file synchronization method provided in this invention creates a transmission task based on a unique identifier when the file synchronization status in the file transfer record is unsynchronized or pending retransmission, so as to synchronize the file from the original system to the new system. This can improve the speed and accuracy of file synchronization, and ensure the consistency and integrity of the data.
[0059] Based on the above embodiments, Figure 2 This is a flowchart of a file synchronization method provided in an embodiment of the present invention, such as... Figure 2 As shown, the method includes the following steps:
[0060] S201, start the file synchronization task at set intervals. When the file synchronization task starts, iterate through the file transfer records generated within the set time period.
[0061] S202, parse the traversed file transfer records, determine the synchronization status of the files in the file transfer records. If the synchronization status is not synchronized, then execute S204. If the synchronization status is pending retransmission, then execute S203.
[0062] S203, obtain the time interval between the last file transfer and the current time in the file transfer record; if the time interval is greater than the set threshold, then execute S204.
[0063] S204, retrieve the file transfer information from the file transfer record; create a transfer task based on the unique identifier and transfer information.
[0064] S205, add the transfer task to the synchronization queue according to file priority.
[0065] S206, when a transmission task is read from the synchronization queue, the file is obtained based on the unique identifier; the file in the file transmission record is split into multiple fragments and numbered; the multiple fragments are synchronized to the new system based on the transmission task.
[0066] S207: Obtain the synchronization result from the new system. If the synchronization result is "synchronization completed", proceed to S208. If the synchronization result is "synchronization failed", proceed to S209.
[0067] S208, update the synchronization status of the file in the file transfer record to synchronization complete.
[0068] S209, determine the number of file retransmissions in the file transfer record. If the number of retransmissions is less than the set number, update the file synchronization status in the file transfer record to pending retransmission; if the number of retransmissions is greater than or equal to the set number, update the file synchronization status in the file transfer record to synchronization failed.
[0069] Figure 3 This is a schematic diagram of the structure of a file synchronization device provided in an embodiment of the present invention, as shown below. Figure 3 As shown, the device includes:
[0070] The file transfer record traversal module 310 is used to traverse the file transfer records generated within a set time period when the file synchronization task is started; wherein, the file transfer record is a record of files transferred from the upstream system to the original system within the set time period; wherein, the file transfer record includes a unique identifier of the file; the unique identifier includes at least one of the following: upstream system identifier ID, file generation timestamp, file checksum, and file name;
[0071] The synchronization status determination module 320 is used to parse the traversed file transfer records and determine the synchronization status of the files in the file transfer records; wherein, the synchronization status includes any one of the following: not synchronized, synchronizing, synchronizing completed, synchronizing failed, and waiting to be retransmitted;
[0072] The transmission task creation module 330 is used to create a transmission task based on the unique identifier if the synchronization status of the file in the file transmission record is out of synchronization or pending retransmission.
[0073] The file synchronization module 340 is used to synchronize the files in the file transfer record to the new system based on the transfer task, and update the synchronization status of the files in the file transfer record to "synchronizing".
[0074] The synchronization status update module 350 is used to continue updating the synchronization status of the files in the file transfer record based on the synchronization results fed back by the new system, and to continue traversing the next file transfer record until the file transfer records generated within the set time period are traversed.
[0075] Optionally, the transfer task creation module 330 is also used for:
[0076] If the synchronization status of the file in the file transfer record is pending retransmission, then obtain the time interval between the last file transfer and the current time in the file transfer record;
[0077] If the time interval is greater than a set threshold, a transmission task is created based on the unique identifier.
[0078] Optionally, the synchronization state update module 350 is also used for:
[0079] If the new system reports a synchronization result of "synchronization complete", then the synchronization status of the file in the file transfer record will be updated to "synchronization complete".
[0080] If the synchronization result reported by the new system is a synchronization failure or no synchronization result is received from the new system within the set time, the number of file retransmissions in the file transfer record is determined. If the number of retransmissions is less than the set number, the synchronization status of the file in the file transfer record is updated to pending retransmission; if the number of retransmissions is greater than or equal to the set number, the synchronization status of the file in the file transfer record is updated to synchronization failure.
[0081] Optionally, the synchronization state update module 350 is also used for:
[0082] If the new system reports a synchronization failure, then determine the reason for the synchronization failure.
[0083] If the synchronization failure is caused by a recoverable anomaly, the synchronization status of the file in the file transfer record will be updated to pending retransmission; wherein, the recoverable anomaly includes network disconnection and / or new system crash;
[0084] If the synchronization failure is caused by an unrecoverable anomaly, the synchronization status of the file in the file transfer record will be updated to synchronization failure; wherein, the unrecoverable cause includes file corruption and / or format error.
[0085] Optionally, it also includes: a prompt message generation module, used to: generate prompt messages and record the reasons for synchronization failure.
[0086] Optionally, the file synchronization module 340 is also used for:
[0087] The transmission tasks are added to the synchronization queue according to file priority;
[0088] When the transmission task is read from the synchronization queue, the file is retrieved based on the unique identifier, and the retrieved file is synchronized to the new system.
[0089] Optionally, it also includes: a synchronization status table establishment module, used to: pre-establish a synchronization status table; and record the synchronization status of files in the file transfer record into the synchronization status table.
[0090] Optionally, the file synchronization module 340 is also used for:
[0091] The file in the file transfer record is split into multiple fragments, and the multiple fragments are assigned numbers;
[0092] The multiple shards are synchronized to the new system based on the transmission task.
[0093] Optionally, the file synchronization module 340 is also used for:
[0094] If the file synchronization status in the file transfer record is pending retransmission, then the fragments that have not been synchronized are obtained.
[0095] Synchronize the unsynchronized fragments to the new system.
[0096] Optionally, the transfer task creation module 330 is also used for:
[0097] Obtain the file transfer information from the file transfer record; wherein, the transfer information includes the original system path, the new system path, the original system node, and the new system node;
[0098] A transmission task is created based on the unique identifier and the transmission information.
[0099] Optionally, the file transfer record traversal module 310 is also used for:
[0100] Start the file synchronization task at set intervals;
[0101] When the file synchronization task starts, it iterates through the file transfer records generated within the set time period.
[0102] The above-described apparatus can execute the methods provided in all the foregoing embodiments of the present invention, and has the corresponding functional modules and beneficial effects for executing the above methods. Technical details not described in detail in this embodiment can be found in the methods provided in all the foregoing embodiments of the present invention.
[0103] Figure 4A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.
[0104] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.
[0105] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0106] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as file synchronization methods.
[0107] In some embodiments, the file synchronization method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the file synchronization method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the file synchronization method by any other suitable means (e.g., by means of firmware).
[0108] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.
[0109] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.
[0110] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.
[0111] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).
[0112] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.
[0113] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.
[0114] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the file synchronization method provided in any embodiment of this application.
[0115] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages or a combination thereof. Programming languages include object-oriented programming languages such as Java, Smalltalk, and C++, as well as conventional procedural programming languages such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0116] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.
Claims
1. A file synchronization method, characterized by, The method comprises the following steps: when a file synchronization task is started, traversing file transmission records generated within a set time period; wherein the file transmission records are records of file transmission from an upstream system to an original system within the set time period; wherein the file transmission records comprise a unique identifier of a file; the unique identifier comprises at least one of the following: an upstream system identifier ID, a file generation timestamp, a file check code, and a file name; parsing the traversed file transmission records to determine the synchronization state of the file in the file transmission records; wherein the synchronization state comprises any one of the following: not synchronized, in synchronization, synchronization completed, synchronization failed, and to be retransmitted; if the synchronization state of the file in the file transmission records is not synchronized or to be retransmitted, creating a transmission task according to the unique identifier; synchronizing the file in the file transmission records to a new system based on the transmission task, and updating the synchronization state of the file in the file transmission records to in synchronization; continuing to update the synchronization state of the file in the file transmission records based on the synchronization result fed back by the new system, and continuing to traverse the next file transmission record until the file transmission records generated within the set time period are traversed.
2. The method of claim 1, wherein, The method for creating a transmission task according to the unique identifier comprises the following steps: if the synchronization state of the file in the file transmission records is to be retransmitted, obtaining the time interval between the last transmission of the file in the file transmission records and the current time; if the time interval is greater than a set threshold, creating a transmission task according to the unique identifier.
3. The method of claim 1, wherein, The method for continuing to update the synchronization state of the file in the file transmission records based on the synchronization result fed back by the new system comprises the following steps: if the synchronization result fed back by the new system is synchronization completed, updating the synchronization state of the file in the file transmission records to synchronization completed; if the synchronization result fed back by the new system is synchronization failed or no synchronization result fed back by the new system is received within a set time length, judging the retransmission number of the file in the file transmission records, if the retransmission number is less than a set number, updating the synchronization state of the file in the file transmission records to to be retransmitted, if the retransmission number is greater than or equal to the set number, updating the synchronization state of the file in the file transmission records to synchronization failed.
4. The method of claim 1, wherein, The method for continuing to update the synchronization state of the file in the file transmission records based on the synchronization result fed back by the new system comprises the following steps: if the synchronization result fed back by the new system is synchronization failed, determining the synchronization failure reason; if the synchronization failure reason is caused by a recoverable exception, updating the synchronization state of the file in the file transmission records to to be retransmitted; wherein the recoverable exception comprises network disconnection and / or new system downtime; if the synchronization failure reason is caused by an unrecoverable exception, updating the synchronization state of the file in the file transmission records to synchronization failed; wherein the unrecoverable reason comprises file damage and / or format error.
5. The method according to claim 3 or 4, characterized in that, After updating the synchronization state of the file in the file transmission records to synchronization failed, the method further comprises the following steps: generating prompt information and recording the synchronization failure reason.
6. The method of claim 1, wherein, The method for synchronizing the file in the file transmission records to the new system based on the transmission task comprises the following steps: adding the transmission task to a synchronization queue according to the file priority; When the transmission task is read from the synchronization queue, a file is obtained based on the unique identifier to synchronize the obtained file to the new system.
7. The method of claim 1, wherein, Further comprising: pre-establishing a synchronization state table; recording the synchronization state of the file in the file transmission record into the synchronization state table.
8. The method of claim 1, wherein, Synchronizing the file in the file transmission record to the new system based on the transmission task, comprising: splitting the file in the file transmission record into multiple shards and setting numbers for the multiple shards; synchronizing the multiple shards to the new system based on the transmission task.
9. The method of claim 8, wherein, Further comprising: if the synchronization state of the file in the file transmission record is to be retransmitted, obtaining a shard that has not been synchronized; synchronizing the shard that has not been synchronized to the new system.
10. The method of claim 1, wherein, Creating a transmission task based on the unique identifier, comprising: obtaining transmission information of the file in the file transmission record; wherein the transmission information comprises an original system path, a new system path, an original system node, and a new system node; creating a transmission task based on the unique identifier and the transmission information.
11. The method of claim 1, wherein, When the file synchronization task is started, the file transmission record generated within a set period is traversed, comprising: starting the file synchronization task every set time length; When the file synchronization task is started, the file transmission record generated within a set period is traversed.
12. A file synchronization apparatus characterized by comprising: Comprising: a file transmission record traversal module, configured to traverse the file transmission record generated within a set period when the file synchronization task is started; wherein the file transmission record is a record of the file transmission from the upstream system to the original system within the set period; wherein the file transmission record comprises a unique identifier of the file; the unique identifier comprises at least one of the following: an upstream system identifier ID, a file generation timestamp, a file check code, and a file name; a synchronization state determination module, configured to analyze the traversed file transmission record to determine the synchronization state of the file in the file transmission record; wherein the synchronization state comprises any one of the following: not synchronized, in synchronization, synchronization completed, synchronization failed, and to be retransmitted; a transmission task creation module, configured to create a transmission task according to the unique identifier if the synchronization state of the file in the file transmission record is not synchronized or to be retransmitted; a file synchronization module, configured to synchronize the file in the file transmission record to the new system based on the transmission task, and update the synchronization state of the file in the file transmission record to in synchronization; a synchronization state update module, configured to continue updating the synchronization state of the file in the file transmission record based on the synchronization result fed back by the new system, and continue traversing the next file transmission record until the file transmission record generated within the set period is traversed.
13. An electronic device, comprising: A computer program product comprising a memory, a processor, and a computer program stored on the memory and executable by the processor, wherein the processor implements the file synchronization method of any one of claims 1-11 when executing the computer program.
14. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the file synchronization method of any one of claims 1-11.
15. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the file synchronization method of any one of claims 1-11.