A real-time file synchronization method and system

Abstract

This application discloses a real-time file synchronization method and system, relating to the fields of data transmission and distributed system consistency technology. The method includes: a server monitoring changes in the source file, using a sliding window to perform rolling hash calculations, dividing the file into multiple data blocks and calculating content hash values; generating block metadata for each data block, and connecting all block metadata through predecessor and successor identifiers to form a doubly linked list structure; comparing the differences in content between the old and new data blocks; and a client performing XOR restoration on incremental block type messages using locally cached old data block content, directly caching full block type messages, and concatenating them into a complete file based on the doubly linked list structure. This application reduces bandwidth consumption through intra-block XOR increments, achieves efficient version management using a linked list structure, optimizes anomaly recovery through snapshot chains, and provides industrial-grade lifecycle management capabilities, significantly improving the real-time performance, integrity, and system availability of data transmission.

Description

Technical Field

[0001] This application relates to the field of computer network data transmission and distributed system consistency technology, specifically to a real-time file synchronization method and system. Background Technology

[0002] With the development of distributed systems, real-time file synchronization technology has been widely applied. Currently, most mainstream solutions use Content-Defined Chunking (CDC) combined with difference algorithms to transmit changed data. However, traditional CDC algorithms have a coarse granularity, and when a small change occurs within a block, the entire block still needs to be transmitted, failing to achieve fine-grained difference extraction within the block, resulting in wasted bandwidth. At the same time, existing block metadata mostly uses flat indexes, and when insertions or deletions occur within the file that change the block boundaries, the file index needs to be rebuilt or extensively adjusted, resulting in high dynamic maintenance overhead.

[0003] Furthermore, in abnormal scenarios such as weak networks or client offline, existing technologies typically employ blind retransmission of the entire file or sequential cumulative retransmission of incremental blocks. This lacks an optimal path assessment for transmission costs and can easily lead to excessively long recovery times due to the cumulative increment far exceeding the full file. Moreover, when transmitting through a single channel, existing systems usually send data sequentially, and large file blocks can easily cause channel congestion, resulting in the blocking of critical messages such as recovery responses and system control, thus reducing the real-time performance and robustness of synchronization.

[0004] Finally, during long-term system operation, server restarts or cross-cycle historical data cleanup can easily lead to state synchronization issues. Existing technologies lack proactive state verification mechanisms after restarts and collaborative management mechanisms for cross-cycle cleanup, which can easily result in synchronization failures or dirty data residue due to unilateral cleanup. Summary of the Invention

[0005] To help solve the above-mentioned technical problems, this application provides a real-time file synchronization method and system, which adopts the following technical solution: Firstly, this application provides a real-time file synchronization method applied to a server, comprising: The source file is monitored for changes, and a rolling hash algorithm is used to dynamically divide the source file into multiple data blocks, and the content hash value of each data block is calculated. For each data block, construct block metadata, which includes the current block identifier, content hash value, predecessor block identifier, and successor block identifier, so that all data blocks of the source file are linked together through the predecessor block identifier and successor block identifier to form a doubly linked list structure. Retrieve the old version data block corresponding to the current data block, compare the content differences between the new and old version data blocks, and if the preset difference rate threshold is met, calculate the XOR difference data between the new and old data blocks and generate an incremental block type message; otherwise, generate a full block type message. Send the incremental block type message or the full block type message to the client so that the client can reconstruct and assemble the file based on the doubly linked list structure.

[0006] The method of dynamically dividing the source file using a rolling hash algorithm specifically includes: using a sliding window to perform rolling hash calculation on the content of the source file; when the hash value of the sliding window meets a preset feature condition or the size of the current data block exceeds a preset block size threshold, marking the block boundary; wherein, the preset feature condition is that the lower several bits of the rolling hash value meet a preset pattern. The current block identifier adopts a composite structure, which includes at least the source file name, file version serial number and block index; the method also includes generating a multi-dimensional version vector for the source file, which includes at least the file version serial number, timestamp and client group identifier. The incremental block type message includes the difference start offset, the difference data length, and the XOR difference data.

[0007] The sending of messages to the client specifically includes: Messages are sent through a single communication channel and scheduled using a priority queue at the application layer. The priority queue divides message priorities from high to low into at least the following levels: system event and recovery response level, file list level, incremental block level, and full block level. The sending thread retrieves messages from the head of the priority queue and sends them.

[0008] The method further includes: File snapshots are generated periodically. Each file snapshot contains the complete content of a specific version of the file and the corresponding block metadata linked list. When a client's patch request is received or a version jump is detected, the available full snapshot closest to the current version identifier is determined based on the snapshot chain index. The amount of full data transferred from that snapshot to the target version is calculated as the full snapshot transfer cost. The total amount of data of all cumulative incremental blocks between the current version and the target version is calculated as the cumulative incremental transfer cost. If the full snapshot transfer cost is less than the cumulative incremental transfer cost, the full snapshot file is sent to the client; otherwise, the cumulative incremental block list is sent. The method further includes: after the server restarts, broadcasting a system event message carrying a restart instruction to the online client so that the client can perform status verification; The method further includes: detecting cross-cycle cleanup events through a scheduled task, sending a data cleanup command carrying the cleanup scope and command identifier to the client, and cleaning up the source files after receiving a cleanup confirmation message from the client.

[0009] Secondly, this application provides a real-time file synchronization method applied to a client, comprising: Receive a message sent by the server, the message containing block metadata and corresponding data content, the block metadata containing at least a block identifier, a predecessor block identifier and a successor block identifier; Parse the message. If it is an incremental block type message, perform an XOR operation between the old data block content of the corresponding block identifier in the local cache and the XOR difference data in the message to restore the new data block content. If it is a full block type message, directly obtain the new version data block content. Based on the predecessor and successor pointer relationships in the block metadata, the contents of each data block are concatenated into a complete file locally using a doubly linked list structure.

[0010] The step of performing an XOR operation between the content of the old data block identified by the corresponding block in the local cache and the XOR difference data in the message specifically includes: The corresponding old data block content is retrieved from the local cache according to the block identifier; the part of the old data block content starting from the difference start offset and with a length equal to the difference data length is XORed with the XOR difference data to obtain the corresponding part in the new data block content; if the old data block content corresponding to the block identifier does not exist in the local cache, a block-level replenishment request is sent to the server. The method further includes: receiving a system event message broadcast by the server carrying a restart instruction, comparing the local file version serial number with the file version serial number of the server, and if there is a version jump, sending a patch request to the server; The method further includes: receiving a data cleanup instruction from the server carrying the cleanup scope and instruction identifier, pausing the file synchronization service, atomically cleaning up file data in the specified storage location and resetting the local file version status, and sending a cleanup confirmation message to the server after the cleanup is completed.

[0011] Thirdly, this application provides a real-time file synchronization system, wherein the system is a server-side system for performing the method as described in any one of the first aspects, comprising: The file monitoring module is used to monitor changes to source files; A dynamic chunking engine is used to execute the dynamic chunking and calculate the content hash value; A block-level linked list builder is used to construct block metadata for each data block and form the doubly linked list structure. Version vector manager, used to generate the multidimensional version vector; Snapshot chain storage is used to store periodically generated file snapshots; The lifecycle manager is used to broadcast system event messages after a reboot, send data cleanup commands when a cross-cycle cleanup event is detected, and clean up source files after receiving cleanup confirmation. A priority scheduling queue is used to schedule message sending according to the stated priority; A single-channel transmission interface is used to send incremental block type messages or full block type messages to the client through a single communication channel.

[0012] The dynamic block engine is specifically used to: perform rolling hash calculation on the content of the source file using a sliding window; and mark the block boundary when the hash value of the sliding window meets the preset feature conditions or the size of the current data block exceeds the preset block size threshold. The current block identifier generated by the block-level linked list builder adopts a composite structure, which includes at least the source file name, file version serial number, and block index; The version vector manager is also used to generate a multi-dimensional version vector for the source file, the multi-dimensional version vector including at least a file version serial number, a timestamp, and a client group identifier; The file snapshots in the snapshot chain storage contain the complete content of a specific version of the file and the corresponding block metadata chain.

[0013] Fourthly, this application provides a real-time file synchronization system, wherein the system is a client system for performing the method as described in the second aspect, and includes: A single-channel transmission interface is used to receive messages sent by the server. The message parser is used to parse received messages and distinguish between incremental block type messages and full block type messages; The XOR recovery engine is used to perform an XOR operation on incremental block type messages, using the old data block content of the corresponding block identifier in the local cache and the XOR difference data in the message to restore the new data block content. The block cache manager is used to cache the content of data blocks and the content of old data blocks with corresponding block identifiers; The consistency merging module is used to concatenate the contents of each data block into a complete file based on the doubly linked list structure. The lifecycle executor is used to compare the version number when a system event message is received and to trigger a rollback request when a version jump occurs, as well as to perform cleanup and send a cleanup confirmation message when a data cleanup instruction is received. Local state storage is used to store local file version serial numbers, cached data block content, and local file version status.

[0014] The XOR recovery engine is specifically used to: retrieve the corresponding old data block content from the local cache according to the block identifier; perform a bitwise XOR operation on the part of the old data block content starting from the difference start offset and having a length equal to the difference data length, with the XOR difference data to obtain the corresponding part in the new data block content; if there is no old data block content corresponding to the block identifier in the local cache, a block-level replenishment request is triggered to be sent to the server. The lifecycle executor is also used to: compare the local file version serial number with the server's file version serial number when a system event message is received; if a version jump exists, trigger a rollback request; and after receiving a data cleanup instruction, suspend the file synchronization service, atomically clean up the specified storage location and reset the local file version status, and trigger the sending of a cleanup confirmation message after the cleanup is completed.

[0015] In summary, this application has the following beneficial technical effects: By combining CDC block partitioning with intra-block XOR increments, XOR data is sent only when the intra-block difference rate is low, reducing bandwidth consumption; a doubly linked list-style block metadata structure supports efficient block-level insertion and deletion without rebuilding the entire file index; a snapshot chain and shortest recovery path calculation mechanism avoids full retransmission and shortens anomaly recovery time; priority queue scheduling ensures the priority transmission of critical control messages and recovery data; and a server-controlled restart recovery and cross-cycle cleanup mechanism ensures long-term data consistency for the system. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the overall architecture of the system in this application.

[0017] Figure 2 This is a schematic diagram of the linked list-style block metadata structure in an embodiment of this application.

[0018] Figure 3 This is a schematic diagram of the priority queue scheduling process in an embodiment of this application.

[0019] Figure 4 This is a schematic diagram of the intelligent recovery process based on snapshot chain in the embodiments of this application.

[0020] Figure 5 This is a sequence diagram of lifecycle management (restart and cleanup) in the embodiments of this application. Detailed Implementation

[0021] The present application will be further described below with reference to the accompanying drawings. The principles of the present application are very clear to those skilled in the art. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.

[0022] The relevant English terms and their corresponding Chinese translations are as follows: Round Robin, Random Selection, Weighted Round Robin, Least Connections, IP Hashing, Consistent Hashing, Content-Aware Consistent Hashing, BlockID, prevBlockID, nextBlockID, Manifest, Delta Block, Full Block, PriorityBlockingQueue, TCP (Transmission Control Protocol), XOR_Data, SYSTEM_EVENT, CMD_CLEANUP, ACK_CLEANUP.

[0023] I. Dynamic block partitioning and linked list-based block metadata construction applied to the server side.

[0024] This section describes the process by which the server dynamically divides the source file into blocks and constructs a linked list of block metadata.

[0025] Figure 1 This is a schematic diagram of the overall architecture of the system in this application, showing the core modules of the server, client, and communication network. The server includes a file monitoring module, a dynamic block partitioning engine, a block-level linked list builder, a version vector manager, a lifecycle manager, a priority scheduling queue, and a single-channel transmission interface; the client includes a single-channel transmission interface, a message parser, an XOR restoration engine, a block cache manager, a consistency merging module, a lifecycle executor, and local state storage.

[0026] The server monitors changes to the source file. When a modification is detected, a sliding window is used to perform a rolling hash calculation on the file's content. A sliding window is set up starting from the beginning of the file, sliding forward byte by byte and continuously calculating the rolling hash value of the data within the window.

[0027] The determination of block boundaries follows a dual condition: First, when the scrolling hash value of the sliding window meets a preset characteristic condition, a block boundary is marked at the current window position. The preset characteristic condition is that the lower bits of the scrolling hash value meet a preset pattern (e.g., the lower N bits are 0). Second, to prevent the generation of excessively large or small data blocks, a preset block size threshold (e.g., 16KB) is set. When the accumulated data amount from the previous block boundary exceeds this preset block size threshold, regardless of whether the current hash value meets the preset characteristic condition, a block boundary is forcibly marked at the current window position. Through these dual conditions, the file is divided into multiple reasonably sized data blocks, and the block boundaries are determined by content characteristics. This ensures that when partial insertion or deletion occurs in the file, only the block boundaries near the affected area may change, while most other block boundaries remain stable, achieving an anti-file shifting effect.

[0028] After the data is partitioned, a content hash value is calculated for each data block. Subsequently, the version vector management module generates corresponding block metadata for each data block. The block metadata includes the following fields: Block ID, content hash value, block version, predecessor block ID (prevBlockID), and successor block ID (nextBlockID). The block ID uses a composite structure: BlockID = FileName + "_" + FileVersion, which is composed of the source filename, file version sequence number, and block index. The block version field is the same as the file version sequence number in the block ID, both used to identify the file version to which the data block belongs. The content hash value is a summary of the data block's content. The predecessor block ID points to the block ID of the block preceding this one in the file, and the successor block ID points to the block ID of the block following this one in the file. The predecessor block ID for the first block is empty, and the successor block ID for the last block is empty.

[0029] Through the mutual references between the predecessor block identifier and the successor block identifier, the block metadata of all data blocks are linked together to form a doubly linked list structure.

[0030] When a file is updated from version V9 to version V10, some data blocks may be reused from version V9. Only the version number of the data blocks that have changed is updated to V10, while the unchanged data blocks retain their original version numbers. Through this doubly linked list structure, when block-level insertions or deletions occur in subsequent versions, the linked list can be updated simply by modifying the predecessor or successor pointers of adjacent nodes, without having to rebuild the entire file's index structure.

[0031] Meanwhile, the version vector management module generates a multi-dimensional version vector for the source file, which includes at least a file version sequence number, a timestamp, and a client group identifier. The file version sequence number increments with each file change, the timestamp records the time the version was generated, and the client group identifier is used to distinguish the synchronization range of different client groups. In addition, the metadata of each data block also records the file version at the time the data block was created, i.e., the block version field, which is used to identify the file version to which the data block belongs, providing a basis for subsequent version tracking and incremental comparison.

[0032] Figure 2 This diagram illustrates the linked list-based block metadata structure in this application, showcasing the block-level linked list relationship during file version iteration. Taking the file update from version V10 to V11 as an example, blocks 1 and 3 reuse the old version (V10) because their content hashes are identical, only updating the version identifier; block 2 generates a new version (V11) due to content changes. Through references to the predecessor block identifier (prevBlockID) and the successor block identifier (nextBlockID), all data blocks are linked together into a doubly linked list, supporting block-level version management and efficient updates. II. Application to XOR increment generation on the server side.

[0033] This section describes the process by which the server compares old and new versions of data blocks and generates incremental messages.

[0034] After the server completes the segmentation and metadata generation of the new version source file, incremental analysis is performed for each new version data block. First, the block identifier of the corresponding old version data block is determined based on the block identifier of the current data block. Since the block identifier contains the file version sequence number, the server replaces the current version sequence number with the previous version sequence number to locate the corresponding indexed data block identifier in the old version. Then, the content of that old version data block is retrieved from the cache.

[0035] After retrieving the old data block, compare the content differences between the old and new data blocks. First, determine if the sizes of the old and new data blocks are the same: if the sizes are different, directly generate a full block type message containing the complete content of the new version data block; if the sizes are the same, further calculate the content difference rate.

[0036] A preset difference rate threshold is set as the threshold value for determining whether to use incremental transmission. When the content difference rate is lower than the preset difference rate threshold (e.g., 20%), it indicates that the old and new data blocks are highly similar, and incremental transmission is more efficient. In this case, the XOR difference data between the old and new data blocks is calculated, that is, a bitwise XOR operation (XOR_Data = Old_Block ^ New_Block) is performed on each byte of the two data blocks to obtain the XOR difference data; at the same time, the difference interval [offset, length] is recorded, and an incremental block type message (BLOCK_DELTA) is generated. The incremental block type message contains the block identifier, the difference start offset (offset), the difference data length (length), and the XOR difference data (XOR_Data). When the content difference rate is not lower than the preset difference rate threshold, it indicates that the data block has changed significantly, and the bandwidth saved by incremental transmission is limited. A full block type message (BLOCK_FULL) containing the complete new version of the data block content is directly generated.

[0037] III. Priority-based scheduling of transmission applied to the server side.

[0038] This section describes the process by which the server schedules message transmissions using a priority queue.

[0039] Figure 3 This is a schematic diagram of the priority queue scheduling process in this application embodiment, illustrating the priority scheduling mechanism that divides messages according to their type after generation. Message types include system events / recovery (P0), file lists (P1), incremental blocks (P2), and full blocks (P3). After each type of message enters its corresponding priority queue, it is uniformly scheduled by the priority merging queue. Network transmission is achieved through a single-channel sending thread to ensure that high-priority messages are processed first.

[0040] The server sends messages via a single communication channel (e.g., a persistent TCP connection). During sending, a priority queue (PriorityBlockingQueue) is used at the application layer for scheduling. The priority queue defines message priorities from high to low as follows: P0: System events (reboot notification), recovery response (data restoration); P1: Manifest. P2: Delta Block; P3: Full Block.

[0041] The sending thread retrieves the highest-priority message from the head of the priority queue and sends it. This priority scheduling mechanism ensures that critical system control messages, recovery data, and file lists reach the client first, avoiding blockage by large data transfers. Previously generated incremental block type messages and full block type messages are sent with priorities P2 and P3, respectively.

[0042] IV. XOR restoration and file concatenation applied to the client side.

[0043] This section provides a real-time file synchronization method, specifically applied to the client, describing the process by which the client receives messages and restores file content.

[0044] The client receives messages sent by the server and distinguishes between incremental block type messages and full block type messages based on the message type.

[0045] For full block type messages, the client directly caches the complete new version data block content contained in the message.

[0046] For incremental block type messages, the client performs XOR restoration using the old data block content with the corresponding block identifier in the local cache. Specifically: the block identifier, difference start offset, difference data length, and XOR difference data (XOR_Data) are extracted from the message. The corresponding old data block content is then searched for in the local cache based on the block identifier. If the corresponding old data block content is not found in the local cache, it indicates that the client lacks the basic data required for restoration, and a block-level replenishment request is sent to the server.

[0047] If the corresponding old data block content exists in the local cache, a data segment with a length equal to the difference data length, starting from the offset from the beginning of the difference, is located in the old data block content. This segment is then XORed with the XOR difference data (New_Block = Old_Block ^ XOR_Data) to obtain the corresponding part in the new data block content. Since the parts of the old data block not covered by the difference interval remain unchanged, the restored part is combined with the rest of the old data block to obtain the complete new data block content. Finally, the restored new data block content is cached.

[0048] After all data blocks (whether from incremental restoration or full reception) have been cached, the client, based on the predecessor and successor relationships of the metadata of each block in the doubly linked list structure, concatenates the contents of each data block into a complete file in sequence, and performs atomic replacement and overall hash verification to ensure file consistency.

[0049] V. Intelligent snapshot chain replenishment through collaboration between the server and the client.

[0050] This section covers the collaborative steps of the aforementioned server-side and client-side methods, describing the recovery process when data inconsistency is detected.

[0051] Figure 4 This is a schematic diagram of the intelligent recovery process based on a snapshot chain in this application embodiment, illustrating the calculation process of the recovery strategy when data inconsistency is detected. When data inconsistency is detected, the recovery strategy is calculated to determine conditions such as the number of missing blocks, version difference, and snapshot availability. The strategy then selects to send missing blocks, version differences, snapshots and differences, or the full file. Finally, the file hash is verified to ensure successful recovery, thus achieving abnormal recovery.

[0052] On the server side, file snapshots are generated periodically, for example, a complete file snapshot and block list index are saved every N versions (e.g., 10). A file snapshot contains the complete content of the file for a specific version and the corresponding block metadata linked list. All snapshots form a snapshot chain index.

[0053] On the client side, when a version jump or missing data block is detected, the client initiates a recovery request, carrying the local current version number and the target version number. The server calculates the shortest recovery path based on the snapshot chain index, specifically including the following scenarios: If the number of missing data blocks is small, the missing data blocks are sent directly. If the version difference is small, send the difference data of the intermediate version; If the version difference is large, find the nearest available snapshot to the current version and send a combined data packet of the snapshot and subsequent differences; If no snapshot is available, send the full file.

[0054] More specifically, the server compares the cost of restoring from the current version to the target version using a full snapshot transfer cost and the cumulative incremental transfer cost. First, it determines the nearest available full snapshot to the current version identifier and calculates the amount of data to transfer from that snapshot to the target version, which is the full snapshot transfer cost. Second, it calculates the total amount of data in all cumulative incremental blocks between the current version identifier and the target version identifier, which is the cumulative incremental transfer cost. Finally, it compares the two costs. If the full snapshot transfer cost is less than the cumulative incremental transfer cost, the full snapshot path is selected, and the full snapshot file is sent to the client; otherwise, the cumulative incremental path is selected, and a list of incremental blocks between the current version and the target version is sent to the client.

[0055] After receiving the replacement data, the client restores and reassembles it according to the above method. Upon completion, the overall hash of the file is verified to ensure data consistency.

[0056] VI. Lifecycle management for collaboration between the server and the client.

[0057] This section covers the collaborative steps of the aforementioned server-side and client-side methods, describing the process of server restart recovery and cross-cycle data cleanup.

[0058] Figure 5 This is a lifecycle management sequence diagram in this application embodiment, illustrating the interaction flow between the server and client in restart and cross-day cleanup scenarios. In the restart scenario, the server broadcasts a restart event, and the client requests a patch after comparing versions. In the cleanup scenario, the server periodically triggers a cleanup command, the client pauses synchronization and cleans up data, and ACK confirmation ensures that the server and client data cleanup synchronization, guaranteeing the consistency of the system's long-term operation.

[0059] (a) Restart and recovery of server-side centralized control.

[0060] After the server restarts, it loads the latest file version serial number from persistent storage. Then, it broadcasts a system event message (SYSTEM_EVENT, type RESTART) carrying a restart instruction to all online clients. This message is sent via the highest priority (P0) priority queue.

[0061] Correspondingly, after receiving the system event message, the client compares the local version serial number with the server's version serial number. If they match, no additional action is required; if there is a version jump (i.e., the local version is lower than the server version), the patching process is triggered, a patching request is sent to the server, and the client restores itself according to the patching process described above to catch up with the latest version.

[0062] (ii) Cross-cycle cleanup under server-side control.

[0063] The server detects cross-cycle cleanup events via a scheduled task. When a cross-cycle cleanup event is detected, a data cleanup command (CMD_CLEANUP) carrying the cleanup scope and a unique command identifier is generated and sent to each client through a high-priority channel.

[0064] Correspondingly, upon receiving the data cleanup command, the client first pauses the file synchronization service, then atomically cleans up the file data in the specified storage location and resets the local version state. After cleanup is complete, it sends a cleanup confirmation message (ACK_CLEANUP) to the server.

[0065] After receiving confirmation messages from all relevant clients, the server performs the source file cleanup operation. This server-side centralized confirmation mechanism ensures that data cleanup between the server and clients remains synchronized, preventing data inconsistencies or residual old data caused by unilateral cleanup, and ensuring long-term data consistency and cleanliness for the system.

[0066] The technical scope of this application is not limited to the contents of the above specification. Those skilled in the art can make various modifications and variations to the above embodiments without departing from the technical concept of this application, and all such modifications and variations should fall within the scope of this application.

Claims

1. A real-time file synchronization method, applied to a server, characterized in that, include: The source file is monitored for changes, and a rolling hash algorithm is used to dynamically divide the source file into multiple data blocks, and the content hash value of each data block is calculated. For each data block, construct block metadata, which includes the current block identifier, content hash value, predecessor block identifier, and successor block identifier, so that all data blocks of the source file are linked together through the predecessor block identifier and successor block identifier to form a doubly linked list structure. Retrieve the old version data block corresponding to the current data block, compare the content differences between the new and old version data blocks, and if the preset difference rate threshold is met, calculate the XOR difference data between the new and old data blocks and generate an incremental block type message; otherwise, generate a full block type message. Send the incremental block type message or the full block type message to the client so that the client can reconstruct and assemble the file based on the doubly linked list structure.

2. The real-time file synchronization method according to claim 1, characterized in that, The method of dynamically dividing the source file using a rolling hash algorithm specifically includes: using a sliding window to perform rolling hash calculation on the content of the source file; when the hash value of the sliding window meets a preset feature condition or the size of the current data block exceeds a preset block size threshold, marking the block boundary; wherein, the preset feature condition is that the lower several bits of the rolling hash value meet a preset pattern. The current block identifier adopts a composite structure, which includes at least the source file name, file version serial number and block index; the method also includes generating a multi-dimensional version vector for the source file, which includes at least the file version serial number, timestamp and client group identifier. The incremental block type message includes the difference start offset, the difference data length, and the XOR difference data.

3. The real-time file synchronization method according to claim 1, characterized in that, The sending of messages to the client specifically includes: Messages are sent through a single communication channel and scheduled using a priority queue at the application layer. The priority queue divides message priorities from high to low into at least the following levels: system event and recovery response level, file list level, incremental block level, and full block level. The sending thread retrieves messages from the head of the priority queue and sends them.

4. The real-time file synchronization method according to claim 1, characterized in that, The method further includes: File snapshots are generated periodically. Each file snapshot contains the complete content of a specific version of the file and the corresponding block metadata linked list. When a client's patch request is received or a version jump is detected, the available full snapshot closest to the current version identifier is determined based on the snapshot chain index. The amount of full data transferred from that snapshot to the target version is calculated as the full snapshot transfer cost. The total amount of data of all cumulative incremental blocks between the current version and the target version is calculated as the cumulative incremental transfer cost. If the full snapshot transfer cost is less than the cumulative incremental transfer cost, the full snapshot file is sent to the client; otherwise, the cumulative incremental block list is sent. The method further includes: after the server restarts, broadcasting a system event message carrying a restart instruction to the online client so that the client can perform status verification; The method further includes: detecting cross-cycle cleanup events through a scheduled task, sending a data cleanup command carrying the cleanup scope and command identifier to the client, and cleaning up the source files after receiving a cleanup confirmation message from the client.

5. A real-time file synchronization method, applied to a client, characterized in that, include: Receive a message sent by the server, the message containing block metadata and corresponding data content, the block metadata containing at least a block identifier, a predecessor block identifier and a successor block identifier; Parse the message. If it is an incremental block type message, perform an XOR operation between the old data block content of the corresponding block identifier in the local cache and the XOR difference data in the message to restore the new data block content. If it is a full block type message, directly obtain the new version data block content. Based on the predecessor and successor pointer relationships in the block metadata, the contents of each data block are concatenated into a complete file locally using a doubly linked list structure.

6. The real-time file synchronization method according to claim 5, characterized in that, The step of performing an XOR operation between the content of the old data block identified by the corresponding block in the local cache and the XOR difference data in the message specifically includes: The corresponding old data block content is retrieved from the local cache according to the block identifier; the part of the old data block content starting from the difference start offset and with a length equal to the difference data length is XORed with the XOR difference data to obtain the corresponding part in the new data block content; if the old data block content corresponding to the block identifier does not exist in the local cache, a block-level replenishment request is sent to the server. The method further includes: receiving a system event message broadcast by the server carrying a restart instruction, comparing the local file version serial number with the file version serial number of the server, and if there is a version jump, sending a patch request to the server; The method further includes: receiving a data cleanup instruction from the server carrying the cleanup scope and instruction identifier, pausing the file synchronization service, atomically cleaning up file data in the specified storage location and resetting the local file version status, and sending a cleanup confirmation message to the server after the cleanup is completed.

7. A real-time file synchronization system, characterized in that, The system is a server system for performing the method as described in any one of claims 1 to 4, comprising: The file monitoring module is used to monitor changes to source files; A dynamic chunking engine is used to execute the dynamic chunking and calculate the content hash value; A block-level linked list builder is used to construct block metadata for each data block and form the doubly linked list structure. Version vector manager, used to generate the multidimensional version vector; Snapshot chain storage is used to store periodically generated file snapshots; The lifecycle manager is used to broadcast system event messages after a reboot, send data cleanup commands when a cross-cycle cleanup event is detected, and clean up source files after receiving cleanup confirmation. A priority scheduling queue is used to schedule message sending according to the stated priority; A single-channel transmission interface is used to send incremental block type messages or full block type messages to the client through a single communication channel.

8. The real-time file synchronization system according to claim 7, characterized in that, The dynamic block engine is specifically used to: perform rolling hash calculation on the content of the source file using a sliding window; and mark the block boundary when the hash value of the sliding window meets the preset feature conditions or the size of the current data block exceeds the preset block size threshold. The current block identifier generated by the block-level linked list builder adopts a composite structure, which includes at least the source file name, file version serial number, and block index; The version vector manager is also used to generate a multi-dimensional version vector for the source file, the multi-dimensional version vector including at least a file version serial number, a timestamp, and a client group identifier; The file snapshots in the snapshot chain storage contain the complete content of a specific version of the file and the corresponding block metadata chain.

9. A real-time file synchronization system, characterized in that, For performing the method as described in claim 5 or 6, the system is a client system, comprising: A single-channel transmission interface is used to receive messages sent by the server. The message parser is used to parse received messages and distinguish between incremental block type messages and full block type messages; The XOR recovery engine is used to perform an XOR operation on incremental block type messages, using the old data block content of the corresponding block identifier in the local cache and the XOR difference data in the message to restore the new data block content. The block cache manager is used to cache the content of data blocks and the content of old data blocks with corresponding block identifiers; The consistency merging module is used to concatenate the contents of each data block into a complete file based on the doubly linked list structure. The lifecycle executor is used to compare the version number when a system event message is received and to trigger a rollback request when a version jump occurs, as well as to perform cleanup and send a cleanup confirmation message when a data cleanup instruction is received. Local state storage is used to store local file version serial numbers, cached data block content, and local file version status.

10. The real-time file synchronization system according to claim 9, characterized in that, The XOR recovery engine is specifically used to: retrieve the corresponding old data block content from the local cache according to the block identifier; perform a bitwise XOR operation on the part of the old data block content starting from the difference start offset and having a length equal to the difference data length, with the XOR difference data to obtain the corresponding part in the new data block content; if there is no old data block content corresponding to the block identifier in the local cache, a block-level replenishment request is triggered to be sent to the server. The lifecycle executor is also used to: compare the local file version serial number with the server's file version serial number when a system event message is received; if a version jump exists, trigger a rollback request; and after receiving a data cleanup instruction, suspend the file synchronization service, atomically clean up the specified storage location and reset the local file version status, and trigger the sending of a cleanup confirmation message after the cleanup is completed.

Images