Cloud storage data synchronization method oriented to multi-terminal collaboration

Abstract

The invention discloses a cloud storage data synchronization method oriented to multi-terminal collaboration, which comprises the following steps of: deploying a synchronization agent at each terminal, and generating a terminal identifier and a clock calibration result; monitoring a file system event at each terminal to form an operation event record; updating a corresponding version vector of the operation event record, and combining with the operation event record to form a local operation log entry; performing blocking on the target file to obtain changed block information; three-way merging is adopted based on the change block information, and merged block-level operation data is output; carrying out ANS entropy coding on the combined block-level operation data, and constructing an incremental updating packet; submitting the incremental update package to a multi-copy storage cluster by adopting an EPaxos consensus protocol, and outputting a target file version confirmed by a consensus; and generating an updating instruction set oriented to the subscription terminal, and obtaining a final file state consistent with the cloud. According to the invention, the cloud storage data synchronization method is realized.

Description

Technical Field

[0001] This invention relates to the field of distributed cloud storage technology, and in particular to a cloud storage data synchronization method for multi-terminal collaboration. Background Technology

[0002] In existing cloud storage systems, data synchronization between multiple terminals typically relies on file-level or directory-level comparison mechanisms. When a file is modified on one terminal, the system uses timestamps or hash values ​​to determine the file differences and re-uploads the entire file or its main parts to the cloud, which then distributes it to other terminals. This approach is relatively simple to implement, but it is prone to conflicts in scenarios where multiple terminals collaborate simultaneously, especially when multiple people modify the same document or binary file, often resulting in version overwriting, content loss, or inconsistencies. To resolve conflicts, some systems have introduced simple version number mechanisms, but due to the lack of granular operation records, they cannot accurately reflect the modification order and dependencies of each terminal. This often leads to conflict resolution remaining at the level of manual comparison and selection, resulting in insufficient efficiency and accuracy.

[0003] On the other hand, most existing synchronization mechanisms transmit data at the overall file level. Even if a file undergoes only minor changes, a large amount of data needs to be uploaded, resulting in a waste of bandwidth and storage. Some improvement solutions attempt to reduce the amount of data by transmitting in blocks, but there is a lack of unified standards in the block partitioning, verification, and merging process, leading to inconsistent partitioning results on different terminals, making it difficult for the cloud to correctly merge and modify the data.

[0004] Therefore, how to provide a cloud storage data synchronization method for multi-device collaboration is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose a cloud storage data synchronization method for multi-device collaboration. This invention generates terminal identifiers and clock calibration results by deploying a synchronization agent on the terminal, monitors file system events and generates operation event records, generates log entries by combining version vectors, divides the target file into blocks to obtain change block information, and uses a three-way merging process to generate block-level operation data. Furthermore, it utilizes ANS entropy coding to construct incremental update packets, submits them to a multi-replica storage cluster based on the EPaxos protocol, outputs a consensus-confirmed target file version, generates an update instruction set, and sends it to the terminal, achieving a consistent file state across multiple devices.

[0006] A cloud storage data synchronization method for multi-terminal collaboration according to an embodiment of the present invention includes the following steps:

[0007] Deploy synchronization agents on each terminal and complete registration. Use a time synchronization protocol to perform local clock calibration, generate terminal identifiers and register them to the cloud coordination service via encrypted connection, and output clock calibration results.

[0008] Based on the terminal identifier and clock calibration results, the system monitors file system events on each terminal to generate change descriptions, record file paths, operation types, timestamps, operator identifiers and check values, and forms operation event records.

[0009] Update the corresponding version vector of the operation event record, generate a globally unique operation identifier, and merge it with the operation event record to form a local operation log entry;

[0010] The target file is segmented using local operation log entries as input, block signatures are calculated, block indexes are generated and changed blocks are marked, and block-level checksums are appended to obtain changed block information.

[0011] Based on the change block information, the operation merging is performed on document files using a conflict-free copy data type or operation transformation algorithm, on hierarchical structure files using a tree editing algorithm for three-way merging, and on binary files using a three-way merging based on block signatures, and the merged block-level operation data is output.

[0012] Differential encoding is performed on the merged block-level operation data, and ANS entropy encoding is used to compress the incremental data to construct an incremental update package. The incremental update package contains operation log entries, version vectors, merge strategy lists, block indexes, compressed incremental data and block-level check values, and a digital signature algorithm is used to complete the package-level signature.

[0013] The incremental update package is submitted to the cloud coordination service for signature verification and sequential application. The cloud file status is updated according to the operation order and the merge strategy is applied. The package is submitted to the multi-replica storage cluster using the EPaxos consensus protocol, and the target file version confirmed by consensus is output and persisted on each storage node.

[0014] Using the target file version as the basis for distribution, an update instruction set is generated for the subscribing terminals. After receiving the instruction, each terminal verifies the block-level checksum and reconstructs the target file, performs atomic replacement and local version update, records audit entries and completes confirmation receipt, and obtains the final file status consistent with that in the cloud.

[0015] Optionally, the process of outputting the clock calibration result specifically includes:

[0016] Deploy synchronization agents on each terminal and complete local initialization. Collect account identifiers, device identifiers and initial times. Combine them and obtain the terminal identifier through hashing and interception. Generate a one-time random number and a temporary public-private key pair. Write the terminal identifier, temporary public key, random number and sending time into the registration request message. Set the sequence number and retransmission policy and prepare to send to the cloud coordination service.

[0017] The terminal sends a registration request message to the cloud and receives a confirmation message containing a random number from the cloud, confirmation time, and public key version. The terminal and the cloud use their temporary public and private key pairs and random numbers to perform key negotiation and derivation, generate a session key and session identifier, and after verification, register the registration information and subscription relationship on the terminal and local machine, load the session key into the encrypted channel, and establish a secure connection.

[0018] Clock calibration is performed during the established secure connection. The terminal records the request sending and response receiving times, and the cloud records the request receiving and response sending times. The terminal calculates the time offset by taking half of the two transmission intervals. Then, the time delay is estimated by subtracting the cloud processing time from the round-trip time. The local clock is corrected, and a clock calibration result containing the time offset and round-trip time is generated.

[0019] Optionally, the formation of the operation event record specifically includes:

[0020] Based on the terminal identifier and clock calibration results, each participating terminal establishes a listener for file system events. When an operation event, including creation, modification, deletion, renaming, or moving, occurs, the current event is immediately captured and an initial event entry containing the terminal identifier, calibration timestamp, and file path is generated.

[0021] The initial event entry is invoked, the byte sequence involved in the operation is read, the content summary before and after the operation is calculated, and the corresponding length information is recorded. The start and end offsets of the overwrite record, the new position and size of the append operation record, the deletion range of the clear operation record, and the source and target paths of the path adjustment record are merged with the initial event entry to form a standardized change description entry, which is then appended to the change description log in sequence.

[0022] The system receives change description entries, performs event parsing on the byte sequence and marks boundary positions, applies minimum and maximum block size constraints to obtain an ordered list of boundaries, calculates check values ​​and generates check vectors, and merges them with change description entries to form a complete operation event record.

[0023] Optionally, the process of forming the local operation log entries specifically includes:

[0024] Load the operation event log, read the change description, boundary list, block length list and block-level check value, and load them together with the terminal identifier and calibration time into the same processing context; locate the version vector and local serial number that the current terminal has saved locally for the target file.

[0025] In the processing context, lock and process the component in the version vector corresponding to the current terminal identifier, increment the current component by one while keeping the other components unchanged, write back the version vector field of the current operation event record, and at the same time increment the local serial number and write back the local serial number field of the record, marking it as the record to be generated with the operation identifier.

[0026] Perform fixed-length hashing and truncation to generate a globally unique operation identifier. Merge the current operation identifier and the record to be generated into a local operation log entry, append it to the local operation log queue of the current terminal for the current file, and complete the persistence.

[0027] Optionally, obtaining the change block information specifically includes:

[0028] Read local operation log entries, open the target file according to the file path in the entry and obtain the current byte sequence, set the sliding window size, matching conditions and minimum and maximum allowed lengths, start byte stream scanning and advance by byte step size, mark the current position as the boundary when the fingerprint result meets the matching conditions, force the boundary to fall when the number of consecutive bytes exceeds the maximum allowed length, postpone the boundary fall when the minimum allowed length is not reached, and output the boundary list sorted in ascending order of offset;

[0029] The byte sequence is divided into different content segments according to the boundary list. The starting offset and length of each segment are recorded in sequence. The corresponding byte content is read and a fixed-length strong digest is calculated as the block signature. The starting offset, length and block signature are appended in the order of appearance to generate a file-oriented block index. The operation identifier related to this operation is stored in the index for tracking.

[0030] The system reads the previous version of the same-named block index from the local index storage, compares each item according to the starting offset, registers content segments with different signatures, changed lengths, or no corresponding items in the old index as changed segments, summarizes the sequence number, starting offset, length and new signature of the changed segments to form changed block information, and writes the newly generated block index back to the local index storage.

[0031] Optionally, the process of outputting the merged block-level operation data specifically includes:

[0032] Receive change block information and block index, open the target file directly according to the file path in the local operation log entry, call the reading program to load three contents, namely the same-name block index of the previous version, the standardized change description entry, and the cloud confirmed version. Match the three contents one by one according to the change block number, establish an alignment relationship for each change block, and send the result to the merge processing queue.

[0033] The merge processing queue is traversed sequentially. Three-way merge is performed according to file type. For document files, the three fragments are scanned line by line and aligned. The comparison program is called to identify the consistent parts and write them into the merged fragment. Conflict markers are inserted for areas with differences. For hierarchical structure files, the node tree structure is extracted. The parent-child relationship of each node is compared and the addition, deletion and rearrangement operations are performed. Nodes that cannot be determined are directly recorded as conflict nodes. For binary files, the signature value is checked block by block. Fragments with consistent signatures are directly written into the merged fragment. Fragments with inconsistent signatures are superimposed with the current difference and the reference based on the baseline to generate a merged fragment and register it as a conflict block.

[0034] The merged fragments are written to the result buffer one by one, generating block-level operation entries containing the starting offset order, content length, merged content, and conflict identifier. The merged block-level operation data is sorted according to the starting offset and output. At the same time, a conflict list and block number mapping relationship are generated and written to the temporary storage area.

[0035] Optionally, the construction of the incremental update package specifically includes:

[0036] The system receives the merged block-level operation data, compares it sequentially with the file content at the same position in the previous version according to the starting offset, compares and extracts the byte intervals with inconsistent content, concatenates them into a symbol sequence to be encoded according to the order of appearance, counts the number of times each symbol appears in the current sequence and generates a total count, and calculates the cumulative starting position of the prefix of each symbol according to the lexicographical order of the symbols. In order to maintain the reversible mapping with the block-level operation data, the starting offset and length of each inconsistent interval in the original block are recorded, and an index relationship between block-level entries and symbol sequence intervals is established.

[0037] The symbol sequence is compressed using rANS normalized arithmetic coding based on ANS entropy coding. A radix with a power of two as the base and a lower bound for renormalization are selected. An initial coding state is selected between the lower bound and the radix multiple lower bound. Symbols are processed one by one from back to front, and the coding state is updated. When the updated coding state exceeds the renormalization interval, the least significant radix bits are repeatedly output according to the selected radix, and the state is scaled according to the radix until it returns to the interval. To improve throughput, a multi-interleaving method is adopted and the coding is performed in parallel on multiple independent state lines in a round-robin order. The bit stream output according to the radix and the termination state of each state line are collected synchronously.

[0038] The system receives the bit stream and termination status, and encapsulates them together with the operation identifier and the index relationship from the block-level entry to the symbol sequence interval to form an incremental update packet containing the operation identifier, index relationship, termination status and bit stream. This incremental update packet is written to the local update queue as direct input for submission to the cloud coordination service.

[0039] Optionally, the process of outputting the target file version confirmed by consensus specifically includes:

[0040] Upon receiving an incremental update packet, the terminal uses the session key to calculate a digest of the update packet and generate a signature accompanying the packet. It then submits the terminal identifier, operation identifier, signature, and update packet to the cloud coordination service via an existing encrypted connection. The cloud coordination service distributes the update packet to each replica node in the multi-replica storage cluster. The cloud verifies the signature based on the archived temporary public key, registers the receipt time, assigns an instance number and enqueue number, and writes it into the consensus processing queue.

[0041] Based on EPaxos execution order determination and commit, the cloud sends pre-acceptance requests to different replicas starting from the allocated instance number. Each replica loads the known operation set and the corresponding version vector locally. Combining the initial order label and dependency set of this update, it calculates the updated dependency set and order label. The dependency set is determined using the partial order relationship of the version vector. The cloud summarizes the updated dependency set and order label of each replica. When the fast confirmation return content from the fast quorum set of EPaxos is consistent, a commit decision is directly generated and broadcast. If not, the set with the strongest consistency is selected, an acceptance request is sent to the replica, and a commit decision is generated and broadcast after confirmation.

[0042] In a multi-replica storage cluster, each replica node applies incremental update packages sequentially according to a defined order tag and dependency set. It decodes and compresses the bitstream, restores incremental data, and writes it to the specified offset range of the target file. It updates the block index and verification record, and persists the metadata including instance number, operation identifier, order tag, dependency set, commit time, and generated file version identifier to form a consensus-confirmed target file version.

[0043] Optionally, the process of obtaining the final file status consistent with that in the cloud specifically includes:

[0044] The consensus mechanism confirms the target file version, reads the terminal registration information and subscription relationship, queries the most recently confirmed version identifier for each subscribed terminal, and generates an update instruction based on the sequence tag and dependency set. The update instruction includes the target file identifier, target version identifier, the most recently confirmed version identifier of the terminal, the location of the incremental update package to be pulled, the sequence and dependency constraints, the integrity verification parameters and the retry strategy, and writes the generated update instruction into the distribution queue.

[0045] The update instructions are sent to different subscription terminals according to the distribution queue. The terminals are instructed to pull the corresponding incremental update package set according to the instructions and execute the sequential application according to the sequential tags and dependency sets. After the writing is completed, the local block index and verification record are updated, audit entries are generated, and package-level digests and completion times are calculated. The confirmation receipt is signed using the session key and sent back.

[0046] Receive and verify the confirmation receipts from each terminal, check the signature and digest and match them with the issued update instructions one by one, register the consistency status of the completed terminals with the corresponding target version, and output the final file status consistent with the cloud.

[0047] The beneficial effects of this invention are:

[0048] 1. This invention enables collaborative operation of multiple terminals under the same time base by deploying a synchronization agent on each terminal and generating terminal identifiers and clock calibration results. Compared with existing solutions that rely on local timestamps, this invention effectively avoids version confusion and synchronization delays caused by time drift between different terminals.

[0049] 2. This invention generates fine-grained operation event records when monitoring file system events, and combines them with version vectors to generate globally unique operation identifiers. This enables the accurate capture of operation order and dependencies in the event of concurrent modifications from multiple devices, ensuring the integrity and reliability of conflict detection and resolution.

[0050] 3. This invention limits data changes to the block level and incremental range through block division, three-way merging and differential coding, and achieves efficient compression by combining ANS entropy coding, which greatly reduces redundant transmission and improves bandwidth utilization and storage efficiency.

[0051] 4. By generating update instructions and combining them with a terminal receipt mechanism, this invention ensures the consistency of the final state between the cloud and each terminal, providing efficient, stable and secure technical support for real-time collaboration across multiple terminals. Attached Figure Description

[0052] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0053] Figure 1 This is a flowchart of a cloud storage data synchronization method for multi-terminal collaboration proposed in this invention;

[0054] Figure 2 This is a schematic diagram of the algorithm structure of a cloud storage data synchronization method for multi-terminal collaboration proposed in this invention. Detailed Implementation

[0055] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0056] refer to Figure 1-2 A cloud storage data synchronization method for multi-device collaboration includes the following steps:

[0057] Deploy synchronization agents on each terminal and complete registration. Use a time synchronization protocol to perform local clock calibration, generate terminal identifiers and register them to the cloud coordination service via encrypted connection, and output clock calibration results.

[0058] Based on the terminal identifier and clock calibration results, the system monitors file system events on each terminal to generate change descriptions, record file paths, operation types, timestamps, operator identifiers and check values, and forms operation event records.

[0059] Update the corresponding version vector of the operation event record, generate a globally unique operation identifier, and merge it with the operation event record to form a local operation log entry;

[0060] The target file is segmented using local operation log entries as input, block signatures are calculated, block indexes are generated and changed blocks are marked, and block-level checksums are appended to obtain changed block information.

[0061] Based on the change block information, the operation merging is performed on document files using a conflict-free copy data type or operation transformation algorithm, on hierarchical structure files using a tree editing algorithm for three-way merging, and on binary files using a three-way merging based on block signatures, and the merged block-level operation data is output.

[0062] Differential encoding is performed on the merged block-level operation data, and ANS entropy encoding is used to compress the incremental data to construct an incremental update package. The incremental update package contains operation log entries, version vectors, merge strategy lists, block indexes, compressed incremental data and block-level check values, and a digital signature algorithm is used to complete the package-level signature.

[0063] The incremental update package is submitted to the cloud coordination service for signature verification and sequential application. The cloud file status is updated according to the operation order and the merge strategy is applied. The package is submitted to the multi-replica storage cluster using the EPaxos consensus protocol, and the target file version confirmed by consensus is output and persisted on each storage node.

[0064] Using the target file version as the basis for distribution, an update instruction set is generated for the subscribing terminals. After receiving the instruction, each terminal verifies the block-level checksum and reconstructs the target file, performs atomic replacement and local version update, records audit entries and completes confirmation receipt, and obtains the final file status consistent with that in the cloud.

[0065] This invention achieves a unified benchmark and secure access across multiple terminals through terminal clock calibration and unique identifier generation. It uses block-level checksums to accurately locate file changes, ensuring data integrity. It employs multiple merging algorithms for different file types to improve the accuracy and flexibility of conflict handling. Entropy encoding compression effectively reduces transmission and storage overhead. Combined with digital signatures, it ensures the security of update packages. Furthermore, relying on EPaxos consensus and multi-replica persistence, it achieves highly fault-tolerant and strongly consistent data synchronization, ensuring efficient unification of file status between terminals and the cloud.

[0066] In this embodiment, the process of outputting the clock calibration result specifically includes:

[0067] Deploy synchronization agents on each terminal and complete local initialization. Collect account identifiers, device identifiers and initial times. Combine them and obtain the terminal identifier through hashing and interception. Generate a one-time random number and a temporary public-private key pair. Write the terminal identifier, temporary public key, random number and sending time into the registration request message. Set the sequence number and retransmission policy and prepare to send to the cloud coordination service.

[0068] The terminal sends a registration request message to the cloud and receives a confirmation message containing a random number from the cloud, confirmation time, and public key version. The terminal and the cloud use their temporary public and private key pairs and random numbers to perform key negotiation and derivation, generate a session key and session identifier, and after verification, register the registration information and subscription relationship on the terminal and local machine, load the session key into the encrypted channel, and establish a secure connection.

[0069] Clock calibration is performed during the established secure connection. The terminal records the request sending and response receiving times, and the cloud records the request receiving and response sending times. The terminal calculates the time offset by taking half of the two transmission intervals. Then, the time delay is estimated by subtracting the cloud processing time from the round-trip time. The local clock is corrected, and a clock calibration result containing the time offset and round-trip time is generated.

[0070] This invention achieves trusted registration and secure access of terminal identities through the unique generation of terminal identifiers and the use of temporary key pairs, ensuring the reliability of communication processes in multi-terminal environments. Combined with subscription relationship registration, the cloud can accurately identify the update needs of target files. Clock calibration in secure connections ensures time consistency across terminals in distributed scenarios, providing a unified time benchmark for subsequent operation log generation, version management, and consensus submission, thereby enhancing the overall collaborative efficiency and stability of the system.

[0071] In this embodiment, the formation of the operation event record specifically includes:

[0072] Based on the terminal identifier and clock calibration results, each participating terminal establishes a listener for file system events. When an operation event, including creation, modification, deletion, renaming, or moving, occurs, the current event is immediately captured and an initial event entry containing the terminal identifier, calibration timestamp, and file path is generated.

[0073] The initial event entry is invoked, the byte sequence involved in the operation is read, the content summary before and after the operation is calculated, and the corresponding length information is recorded. The start and end offsets of the overwrite record, the new position and size of the append operation record, the deletion range of the clear operation record, and the source and target paths of the path adjustment record are merged with the initial event entry to form a standardized change description entry, which is then appended to the change description log in sequence.

[0074] The system receives change description entries, performs event parsing on the byte sequence and marks boundary positions, applies minimum and maximum block size constraints to obtain an ordered list of boundaries, calculates check values ​​and generates check vectors, and merges them with change description entries to form a complete operation event record.

[0075] This invention, by monitoring file system events in real time on the terminal side, can instantly capture and generate change descriptions with timestamps and path information when various operations occur, thereby ensuring the integrity and timeliness of event records. Combining fine-grained information such as content summaries, lengths, and offsets before and after the operation, standardized change description entries are formed, improving the traceability of the modification process. Through the generation of boundary lists and check vectors, precise parsing and consistency verification of byte sequences are achieved, effectively supporting subsequent version management and conflict detection, and enhancing the accuracy and reliability of synchronization.

[0076] In this embodiment, the process of forming the local operation log entry specifically includes:

[0077] Load the operation event log, read the change description, boundary list, block length list and block-level check value, and load them together with the terminal identifier and calibration time into the same processing context; locate the version vector and local serial number that the current terminal has saved locally for the target file.

[0078] In the processing context, lock and process the component in the version vector corresponding to the current terminal identifier, increment the current component by one while keeping the other components unchanged, write back the version vector field of the current operation event record, and at the same time increment the local serial number and write back the local serial number field of the record, marking it as the record to be generated with the operation identifier.

[0079] Perform fixed-length hashing and truncation to generate a globally unique operation identifier. Merge the current operation identifier and the record to be generated into a local operation log entry, append it to the local operation log queue of the current terminal for the current file, and complete the persistence.

[0080] This invention achieves precise characterization of operation order and dependencies in a multi-terminal concurrent environment by introducing joint management of version vectors and local sequence numbers into operation event records. The generation of globally unique operation identifiers ensures the uniqueness and traceability of log entries, effectively avoiding conflicts and overwriting when different terminals submit operations. Furthermore, the combination of a persistent local log queue not only enhances the system's fault tolerance and recovery capabilities but also provides a reliable source of operations for subsequent block comparison and consensus submission, thereby improving the overall synchronization stability.

[0081] In this embodiment, obtaining the change block information specifically includes:

[0082] Read local operation log entries, open the target file according to the file path in the entry and obtain the current byte sequence, set the sliding window size, matching conditions and minimum and maximum allowed lengths, start byte stream scanning and advance by byte step size, mark the current position as the boundary when the fingerprint result meets the matching conditions, force the boundary to fall when the number of consecutive bytes exceeds the maximum allowed length, postpone the boundary fall when the minimum allowed length is not reached, and output the boundary list sorted in ascending order of offset;

[0083] The byte sequence is divided into different content segments according to the boundary list. The starting offset and length of each segment are recorded in sequence. The corresponding byte content is read and a fixed-length strong digest is calculated as the block signature. The starting offset, length and block signature are appended in the order of appearance to generate a file-oriented block index. The operation identifier related to this operation is stored in the index for tracking.

[0084] The system reads the previous version of the same-named block index from the local index storage, compares each item according to the starting offset, registers content segments with different signatures, changed lengths, or no corresponding items in the old index as changed segments, summarizes the sequence number, starting offset, length and new signature of the changed segments to form changed block information, and writes the newly generated block index back to the local index storage.

[0085] This invention achieves dynamic boundary division of file content through a sliding window and fingerprint matching mechanism, which can improve the sensitivity of change detection while ensuring the stability of block division. Combined with block signature generation using strong digests and index comparison, it can accurately locate changed content segments in the file and generate traceable change block information, effectively avoiding redundant transmission of the entire file and significantly reducing storage and bandwidth overhead.

[0086] In this embodiment, the process of outputting the merged block-level operation data specifically includes:

[0087] Receive change block information and block index, open the target file directly according to the file path in the local operation log entry, call the reading program to load three contents, namely the same-name block index of the previous version, the standardized change description entry, and the cloud confirmed version. Match the three contents one by one according to the change block number, establish an alignment relationship for each change block, and send the result to the merge processing queue.

[0088] The merge processing queue is traversed sequentially. Three-way merge is performed according to file type. For document files, the three fragments are scanned line by line and aligned. The comparison program is called to identify the consistent parts and write them into the merged fragment. Conflict markers are inserted for areas with differences. For hierarchical structure files, the node tree structure is extracted. The parent-child relationship of each node is compared and the addition, deletion and rearrangement operations are performed. Nodes that cannot be determined are directly recorded as conflict nodes. For binary files, the signature value is checked block by block. Fragments with consistent signatures are directly written into the merged fragment. Fragments with inconsistent signatures are superimposed with the current difference and the reference based on the baseline to generate a merged fragment and register it as a conflict block.

[0089] The merged fragments are written to the result buffer one by one, generating block-level operation entries containing the starting offset order, content length, merged content, and conflict identifier. The merged block-level operation data is sorted according to the starting offset and output. At the same time, a conflict list and block number mapping relationship are generated and written to the temporary storage area.

[0090] This invention addresses the challenges of concurrent modification across multiple terminals by introducing a three-way merging mechanism to differentiate between document-type files, hierarchical structure files, and binary files. It enables rapid merging of consistent parts while accurately locating and marking conflict areas. Furthermore, by combining block indexing and alignment strategies for change description entries, it effectively ensures the precision and traceability of the merging process. The generated merged data and conflict list provide reliable input for subsequent differential coding and consistency verification, significantly improving the accuracy and robustness of data synchronization.

[0091] In this embodiment, the construction of the incremental update package specifically includes:

[0092] The system receives the merged block-level operation data, compares it sequentially with the file content at the same position in the previous version according to the starting offset, compares and extracts the byte intervals with inconsistent content, concatenates them into a symbol sequence to be encoded according to the order of appearance, counts the number of times each symbol appears in the current sequence and generates a total count, and calculates the cumulative starting position of the prefix of each symbol according to the lexicographical order of the symbols. In order to maintain the reversible mapping with the block-level operation data, the starting offset and length of each inconsistent interval in the original block are recorded, and an index relationship between block-level entries and symbol sequence intervals is established.

[0093] The symbol sequence is compressed using rANS normalized arithmetic coding based on ANS entropy coding. A base-two power is selected, along with a lower bound for renormalization. An initial coding state is chosen between the lower bound and the base-multiple lower bound. Symbols are processed sequentially from back to front, and the coding state is updated accordingly.

[0094]

[0095] Where, x ′ The updated encoding state value is represented by f, where x represents the current encoding state value. s The frequency of the current symbol is represented by s, the starting position in the prefix cumulative count is represented by M, and the sum of the frequencies of all symbols is represented by C. s Indicates the starting position of the current symbol in the prefix cumulative count. The symbol 's' represents the floor function, and 'mod' represents the remainder function. The remainder determines the offset position of the symbol 's' within the interval. When the updated encoded state exceeds the re-normalization interval, the least significant radix bits are repeatedly output according to the selected radix, and the state is scaled according to the radix until it returns to the interval. To improve throughput, a multi-interleaving method is adopted, and the encoding is performed in parallel on multiple independent state lines in a round-robin order. The bit stream output according to the radix and the termination state of each state line are collected synchronously.

[0096] The system receives the bit stream and termination status, and encapsulates them together with the operation identifier and the index relationship from the block-level entry to the symbol sequence interval to form an incremental update packet containing the operation identifier, index relationship, termination status and bit stream. This incremental update packet is written to the local update queue as direct input for submission to the cloud coordination service.

[0097] This invention introduces symbolic statistics and interval indexing mechanisms into block-level operational data, achieving precise capture and reversible mapping of byte-level differences. This ensures the integrity and traceability of updated data. Furthermore, it employs rANS-based normalized arithmetic coding for compression, significantly improving throughput and compression ratio with the support of multi-way interleaved parallel coding, effectively reducing redundant storage and transmission overhead of incremental data. Simultaneously, the combined encapsulation of symbolic interval indexing and termination states ensures efficient, secure, and consistent decoding and application of incremental update packets across different terminals and the cloud.

[0098] In this embodiment, the process of outputting the target file version confirmed by consensus specifically includes:

[0099] Upon receiving an incremental update packet, the terminal uses the session key to calculate a digest of the update packet and generate a signature accompanying the packet. It then submits the terminal identifier, operation identifier, signature, and update packet to the cloud coordination service via an existing encrypted connection. The cloud coordination service distributes the update packet to each replica node in the multi-replica storage cluster. The cloud verifies the signature based on the archived temporary public key, registers the receipt time, assigns an instance number and enqueue number, and writes it into the consensus processing queue.

[0100] Based on EPaxos execution order determination and commit, the cloud sends pre-acceptance requests to different replicas, starting with the allocated instance number. Each replica loads the known operation set and corresponding version vector locally. Combining the initial order label and dependency set of this update, the updated dependency set and order label are calculated. The partial order relationship of the version vector is used to determine the dependency set.

[0101]

[0102] Among them, deps r The set of dependencies computed by replica r is represented by `deps`, the initial set of dependencies is represented by `o`, a candidate operation entry is represented by `o`, and `f` represents a file. The set of known operations for the replica r is shown. `conflict(o, OPID)` indicates whether operation `o` conflicts with the currently pending operation. `OPID` represents the globally unique operation identifier of the currently committed operation. o V represents the version vector associated with operation o. OPID This represents the version vector corresponding to the current commit operation. ∪ and ∧ represent the union and intersection operations of sets, respectively. The cloud aggregates the updated dependency sets and order tags of each replica. When the fast confirmation return content from the fast quorum set from EPaxos is consistent, a commit decision is directly generated and broadcast. If not, the set with the strongest consistency is selected, an acceptance request is sent to the replica, and a commit decision is generated and broadcast after confirmation.

[0103] In a multi-replica storage cluster, each replica node applies incremental update packages sequentially according to a defined order tag and dependency set. It decodes and compresses the bitstream, restores incremental data, and writes it to the specified offset range of the target file. It updates the block index and verification record, and persists the metadata including instance number, operation identifier, order tag, dependency set, commit time, and generated file version identifier to form a consensus-confirmed target file version.

[0104] This invention introduces the EPaxos consensus mechanism, enabling multi-replica storage clusters to achieve efficient and consistent update commits in a distributed environment, ensuring accurate resolution of operation order and dependencies under complex concurrency conditions. Furthermore, by combining session key signing with encrypted transmission of incremental update packets, the security and integrity of data transmission are effectively improved.

[0105] In this embodiment, the process of obtaining the final file status consistent with that in the cloud specifically includes:

[0106] The consensus mechanism confirms the target file version, reads the terminal registration information and subscription relationship, queries the most recently confirmed version identifier for each subscribed terminal, and generates an update instruction based on the sequence tag and dependency set. The update instruction includes the target file identifier, target version identifier, the most recently confirmed version identifier of the terminal, the location of the incremental update package to be pulled, the sequence and dependency constraints, the integrity verification parameters and the retry strategy, and writes the generated update instruction into the distribution queue.

[0107] The update instructions are sent to different subscription terminals according to the distribution queue. The terminals are instructed to pull the corresponding incremental update package set according to the instructions and execute the sequential application according to the sequential tags and dependency sets. After the writing is completed, the local block index and verification record are updated, audit entries are generated, and package-level digests and completion times are calculated. The confirmation receipt is signed using the session key and sent back.

[0108] Receive and verify the confirmation receipts from each terminal, check the signature and digest and match them with the issued update instructions one by one, register the consistency status of the completed terminals with the corresponding target version, and output the final file status consistent with the cloud.

[0109] This invention enables efficient and reliable cloud storage data synchronization in multi-terminal collaborative scenarios. It ensures the accurate transmission of operation order and dependencies between terminals, significantly reduces data transmission and storage overhead through block-level incremental updates and entropy coding, and avoids single point of failure and performance bottlenecks by using consensus protocols to complete multi-replica consistency confirmation, ensuring that the data versions in the cloud and on the terminal are always consistent.

[0110] Example 1:

[0111] To verify the feasibility of this invention in practice, it was applied to a multi-terminal collaborative office scenario. In this scenario, there are multiple departments within the enterprise, and each department member needs to access and edit the same type of files stored in the cloud. Traditional cloud storage synchronization methods often encounter problems such as frequent file conflicts, inaccurate merging, redundant data transmission, and significant latency when multiple users modify files simultaneously, thus seriously affecting work efficiency and file consistency.

[0112] In practical applications, this invention ensures that each device has a unique identifier and a unified time base by deploying a synchronization agent on each terminal and performing clock calibration. The system can capture and generate a change description as soon as an operation event occurs, and use checksums to accurately locate the change location. After the terminal completes the recording, it will form a log entry with an operation identifier, and combine block segmentation and signature mechanisms to obtain reliable change block information. This process avoids redundant scanning and unnecessary duplicate calculations.

[0113] To address the differences between various file types, this invention employs a differentiated merging strategy: text documents utilize a conflict-free copy data type algorithm to reduce manual intervention; hierarchical structure files maintain the integrity of their hierarchical relationships through a tree editing algorithm; and binary files achieve rapid merging through a three-way merging based on block signatures. The merged result is further processed using differential processing and ANS entropy encoding compression, significantly reducing data transmission volume and ensuring both synchronization efficiency and security / integrity.

[0114] To further demonstrate the beneficial effects of the present invention, the present invention compares the performance of conventional methods with that of the present invention in terms of data volume, transmission latency, collision rate, and consistency:

[0115] Table 1 Performance comparison of different methods in multi-terminal collaborative scenarios

[0116]

[0117] As shown in Table 1, the performance of this invention in multi-terminal collaborative scenarios is significantly better than that of traditional synchronization methods: In terms of data volume, the average synchronization data volume of this invention is only 2.3MB, which is about 77% less than the 10MB of the traditional method; in terms of latency, the average transmission latency is reduced from 4.5 seconds to 1.2 seconds, an improvement of about 73%; in terms of consistency, the file consistency rate is increased from 92% to 100%, and the number of conflict merging failures is reduced from 2 / 10 operations to 0, achieving complete control over conflicts; in terms of resource utilization, the network utilization rate of this invention reaches 87%, which is 32% higher than that of the traditional method; at the same time, the multi-replica fault tolerance mechanism significantly enhances the availability of the system and avoids the risk of failure caused by single point of failure in the traditional method.

[0118] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A cloud storage data synchronization method for multi-terminal collaboration, characterized in that, Includes the following steps: Deploy synchronization agents on each terminal and complete registration, generate terminal identifiers, register them to the cloud coordination service via encrypted connection, and output clock calibration results; Based on the terminal identifier and clock calibration results, change descriptions are generated by monitoring file system events on each terminal, and the byte sequences are divided into content to form operation event records. Update the corresponding version vector of the operation event record, generate a globally unique operation identifier, and merge it with the operation event record to form a local operation log entry; The target file is segmented using local operation log entries as input, block signatures are calculated, block indexes are generated, and changed blocks are marked to obtain changed block information. Based on the change block information, a three-way merge is performed on document files, hierarchical structure files and binary files, and the merged block-level operation data is output. Differential encoding is performed on the merged block-level operation data, and incremental data is compressed using ANS entropy encoding to construct incremental update packets; The incremental update package is submitted to the cloud coordination service for signature verification and sequential application. It is then submitted to the multi-replica storage cluster using the EPaxos consensus protocol, and the target file version confirmed by consensus is output. The target file version is used to generate an update instruction set for the subscribing terminals. Each terminal records the audit entries and completes the confirmation receipt, resulting in a final file status consistent with that in the cloud.

2. The cloud storage data synchronization method for multi-terminal collaboration according to claim 1, characterized in that, The specific process for outputting the clock calibration results includes: Deploy synchronization agents on each terminal and complete local initialization. Collect account identifiers, device identifiers and initial times. Combine them, hash and intercept the terminal identifier, generate a one-time random number and a temporary public-private key pair, and write them and the sending time into the registration request message. The terminal sends a registration request message to the cloud and receives a confirmation message. The terminal and the cloud use a temporary public-private key pair and a random number to perform key negotiation and derivation, generate a session key and a session identifier, and after verification, register the registration information and subscription relationship on the terminal and the local machine, and establish a secure connection. Clock calibration is performed within the established secure connection. The terminal records the request sending and response receiving times, the cloud records the request receiving and response sending times, the local clock is corrected, and the clock calibration result is generated.

3. The cloud storage data synchronization method for multi-terminal collaboration according to claim 1, characterized in that, The formation of the operation event record specifically includes: Based on the terminal identifier and clock calibration results, each participating terminal establishes a listener for file system events. When an operation event occurs, the current event is immediately captured and an initial event entry containing the terminal identifier, calibration timestamp, and file path is generated. Invoke the initial event entry, read the byte sequence involved in the operation, form a standardized change description entry, and append it to the change description log in sequence; The system receives change description entries, performs event parsing on the byte sequence and marks boundary positions to obtain an ordered list of boundaries, calculates check values ​​and generates check vectors, and merges them with change description entries to form a complete operation event record.

4. The cloud storage data synchronization method for multi-terminal collaboration according to claim 1, characterized in that, The process of forming the local operation log entries specifically includes: Load the operation event log, along with the terminal identifier and calibration time, into the same processing context, and locate the version vector and local serial number that the current terminal has saved locally for the target file; In the processing context, lock and process the component in the version vector corresponding to the current terminal identifier, and mark the record as the operation identifier to be generated; Perform fixed-length hashing and truncation to generate a globally unique operation identifier, and merge the current operation identifier with the record of the operation identifier to be generated to form a local operation log entry.

5. A cloud storage data synchronization method for multi-terminal collaboration according to claim 1, characterized in that, The acquisition of the change block information specifically includes: Read local operation log entries, open the target file according to the file path in the entry and obtain the current byte sequence, start byte stream scanning and advance by byte step, and output the boundary list sorted in ascending order of offset; The byte sequence is divided into different content segments according to the boundary list. The starting offset and length of each segment are recorded in sequence. The corresponding byte content is read and a fixed-length strong digest is calculated as the block signature. The starting offset, length and block signature are appended in the order of appearance to generate a file-oriented block index. The system reads the previous version's block index with the same name from the local index storage, compares each item with the starting offset, and registers content segments with different signatures, changed lengths, or no corresponding items in the old index as changed segments. The system summarizes the sequence number, starting offset, length, and new signature of the changed segments to form the changed block information.

6. A cloud storage data synchronization method for multi-terminal collaboration according to claim 1, characterized in that, The process of outputting the merged block-level operation data specifically includes: Receive change block information and block index, open the target file directly according to the file path in the local operation log entry, call the reading program to load three contents, match the three contents one by one according to the change block number, establish an alignment relationship for each change block, and send the result to the merge processing queue. Iterate through the merge processing queue sequentially, perform a three-way merge based on the file type, generate merged fragments and register them as conflict blocks; The merged fragments are written to the result buffer one by one to generate block-level operation entries, which are then sorted by the starting offset and the merged block-level operation data is output.

7. A cloud storage data synchronization method for multi-terminal collaboration according to claim 1, characterized in that, The construction of the incremental update package specifically includes: Receive the merged block-level operation data, compare it with the file content at the same position in the previous version according to the starting offset, compare and extract the inconsistent byte intervals one by one, concatenate them into a symbol sequence to be encoded according to the order of appearance, count the number of times each symbol appears in the current sequence and generate a total count, calculate the cumulative starting position of the prefix of each symbol according to the lexicographical order of the symbols, record the starting offset and length of each inconsistent interval in the original block, and establish the index relationship from block-level entries to symbol sequence intervals; The symbol sequence is compressed using rANS normalized arithmetic coding based on ANS entropy coding. A radix with a power of two as the base and a lower bound for renormalization are selected. An initial coding state is selected between the lower bound and the radix multiple lower bound. Symbols are processed one by one from back to front, and the coding state is updated. When the updated coding state exceeds the renormalization interval, the least significant radix bits are repeatedly output according to the selected radix, and the state is scaled according to the radix until it returns to the interval. A multi-interleaving method is used and the coding is performed in parallel on multiple independent state lines in a round-robin order. The bit stream output according to the radix and the termination state of each state line are collected synchronously. The received bit stream and termination status are encapsulated together with the operation identifier and the index relationship from block-level entry to symbol sequence interval to form an incremental update packet.

8. A cloud storage data synchronization method for multi-terminal collaboration according to claim 1, characterized in that, The process of outputting the target file version confirmed by consensus specifically includes: Upon receiving an incremental update packet, the terminal uses the session key to calculate a digest of the update packet and generate a signature for the packet. It then submits the terminal identifier, operation identifier, and update packet to the cloud coordination service via an existing encrypted connection. The cloud coordination service distributes the update packet to each replica node in the multi-replica storage cluster. The cloud verifies the signature based on the archived temporary public key, registers the receipt time, assigns an instance number and enqueue number, and writes it into the consensus processing queue. Based on EPaxos execution order determination and commit, the cloud sends pre-acceptance requests to different replicas starting from the allocated instance number. Each replica loads the known operation set and the corresponding version vector locally. Combining the initial order label and dependency set of this update, it calculates the updated dependency set and order label. The cloud summarizes the updated dependency set and order label of each replica. When the fast confirmation return content from the fast quorum of EPaxos is consistent, a commit decision is directly generated and broadcast. If not, the set of return results with the strongest consistency is selected, an acceptance request is sent to the replica, and a commit decision is generated and broadcast after confirmation. In a multi-replica storage cluster, each replica node applies incremental update packages sequentially according to a defined order of tags and dependency sets, decodes and compresses the bitstream, restores incremental data and writes it to the specified offset range of the target file, updates the block index and checksum, records and persists metadata, and forms a consensus-confirmed target file version.

9. A cloud storage data synchronization method for multi-terminal collaboration according to claim 1, characterized in that, The process of obtaining the final file status consistent with that in the cloud specifically includes: Load consensus to confirm the target file version, read terminal registration information and subscription relationship, query the most recently confirmed version identifier for each subscribed terminal, generate update instructions based on sequence tags and dependency sets, and write the generated update instructions into the distribution queue; The update instructions are sent to different subscription terminals according to the distribution queue. The terminals are instructed to pull the corresponding incremental update package set according to the instructions and execute the sequential application according to the sequential tags and dependency sets. After the writing is completed, the local block index and verification record are updated, audit entries are generated, and package-level digests and completion times are calculated. The confirmation receipt is signed using the session key and sent back. Receive and verify the confirmation receipts from each terminal, check the signature and digest and match them with the issued update instructions one by one, register the consistency status of the completed terminals with the corresponding target version, and output the final file status consistent with the cloud.

Images