A file synchronization system for failover

Abstract

The application discloses a file synchronization system for fault switching, relates to the technical field of distributed data processing, and aims to solve the technical problems that the existing file synchronization system lacks dynamic adaptation ability in the aspect of synchronization granularity in the fault switching scene, conflict processing is lagged, and thus the synchronization efficiency is low and data is prone to inconsistency. The system comprises a fault detection module, a dynamic granularity synchronization module, a conflict prediction and preprocessing module, a file synchronization execution module and an incremental fault recovery module. The dynamic granularity synchronization module outputs a synchronization configuration based on multi-dimensional parameters, which serves as the basis for the operation of each module. The conflict prediction and preprocessing module predicts conflicts in advance and processes them in stages. The incremental fault recovery module accurately locates fault blocks and performs incremental recovery. Each module is cooperatively linked. The application improves the file synchronization efficiency, reduces the conflict occurrence rate, accelerates the fault recovery speed, and significantly enhances the stability and availability of the system.

Description

Technical Field

[0001] This invention relates to the field of distributed data processing technology, and in particular to a file synchronization system for failover. Background Technology

[0002] In scenarios such as distributed storage, cloud-based office work, and multi-terminal collaboration, file synchronization is a core support for ensuring data consistency and business continuity. Its operational efficiency and reliability directly determine the system service quality and user experience. However, existing file synchronization systems suffer from the following prominent technical defects when facing failover scenarios, which severely restrict system performance.

[0003] First, existing systems generally use fixed synchronization granularity and file block size to perform chunking and transmission operations, making it impossible to adjust configurations according to changes in key parameters in real-world scenarios. For example, using fine-grained synchronization for large-sized video files with low modification frequency will generate a large amount of redundant data transmission, wasting network and storage resources; using coarse-grained synchronization for small-sized text files with high modification frequency will easily lead to missed modifications or synchronization delays. Especially in failover scenarios, when anomalies such as a sudden drop in network bandwidth or node storage shortages occur, the fixed granularity cannot adapt to the dynamically changing system state, further exacerbating the problems of low synchronization efficiency and resource consumption, and even causing synchronization interruptions.

[0004] Secondly, existing systems largely rely on a "post-event response" mechanism to handle file synchronization conflicts. This means that conflicts are only resolved after they actually occur (such as multiple users simultaneously modifying the same file area or cross-terminal synchronization timing misalignment), through methods like version comparison, manual selection, or simple overwriting. This approach not only increases the resource overhead of conflict handling but also makes it more prone to data inconsistencies during failover. In a faulty state, the system's inherent stability is weak, and delayed conflict handling can lead to the spread of abnormal data, prolonging the fault recovery period and even causing partial data loss, severely impacting the system's availability in fault scenarios.

[0005] Therefore, this application aims to solve the above-mentioned problems by adopting a new technical solution. Summary of the Invention

[0006] In view of the above situation, the purpose of this invention is to provide a file synchronization system for failover, so as to solve the technical problems of existing file synchronization systems lacking dynamic adaptability of synchronization granularity and lagging conflict handling in failover scenarios, resulting in low synchronization efficiency and data inconsistency, and to ensure the high efficiency and data reliability of file synchronization during failover.

[0007] The technical solution is: a file synchronization system for failover, including a fault detection module, a dynamic granular synchronization module, a conflict prediction and preprocessing module, a file synchronization execution module, and an incremental fault recovery module; The fault detection module collects fault information during the file synchronization process in real time and distributes it to the other modules; The dynamic granularity synchronization module outputs synchronization granularity parameters and block size configuration based on multi-dimensional parameters, which serve as the core operating basis for the conflict prediction preprocessing module, file synchronization execution module and incremental fault recovery module. The conflict prediction and preprocessing module realizes early conflict prediction and preprocessing based on the synchronization granularity parameter and block size configuration. The file synchronization execution module performs file segmentation, transmission, and verification according to the configuration. The incremental fault recovery module performs incremental recovery operations by combining the fault information with the configuration.

[0008] Optionally, the fault detection module identifies faults through a combination of multiple methods: Heartbeat detection is used to identify network outages or node crashes, CRC check is used to identify file block corruption, and access log analysis is used to identify precursors to multi-user concurrent conflicts. The fault priority is divided into levels based on file importance and synchronization progress.

[0009] Optionally, the multi-dimensional parameters of the dynamic granularity synchronization module include file attribute parameters, system status parameters, and fault information parameters; The file attribute parameters include file type, file size, and modification frequency; The system status parameters include network bandwidth, free storage space, and CPU load. The fault information parameters include fault type, fault impact range, and fault priority.

[0010] Optionally, the dynamic granularity synchronization module calculates the optimal synchronization granularity parameters and block size configuration through a weighted multi-factor decision algorithm. The weighted multi-factor decision algorithm calculates a comprehensive score according to a preset weight after standardizing each parameter, and then matches the synchronization granularity interval and block size based on the comprehensive score.

[0011] Optionally, the matching rule between the synchronization granularity interval and the block size is: When the overall score S∈[0, 0.3], it corresponds to a fine-grained 4KB / block; when S∈[0.3, 0.7], it corresponds to a medium-grained 32KB / block; and when S∈[0.7, 1], it corresponds to a coarse-grained 128KB / block. The dynamic granularity synchronization module re-collects parameters and updates the configuration every 10 seconds. If a sudden fault is detected, it will trigger an immediate update.

[0012] Optionally, the input data of the conflict prediction preprocessing module includes dynamic parameters, historical synchronization data, and real-time system status; The dynamic parameters include synchronization granularity parameters and block size configuration; The historical synchronization data includes historical conflict records and user modification habits; The real-time system status includes the number of concurrent accesses by multiple users and the file block locking status.

[0013] Optionally, the conflict prediction preprocessing module uses an LSTM-attention mechanism conflict prediction model to generate conflict prediction results. The conflict prediction results include conflict probability, conflict type and high-risk block number, and perform hierarchical preprocessing operations based on the conflict probability.

[0014] Optionally, the rules for the hierarchical preprocessing operation are as follows: Only record synchronization logs when the probability of conflict is less than 30%; When the probability of conflict is 30% or less and less than 70%, an operation alert will be pushed to high-risk users; When the probability of conflict is ≥70%, high-risk file blocks are automatically locked and conflict resolution plans are generated. The conflict types include content conflicts, timing conflicts, and permission conflicts, and the conflict resolution plan includes version backup rules and modification area marking rules.

[0015] Optionally, the file synchronization execution module divides the file into blocks according to the block size configuration, and assigns a unique block ID to each file block. The block ID consists of the file ID, the block sequence number, and the timestamp. The file synchronization execution module adjusts the transmission strategy according to the synchronization granularity parameter. Fine-grained file blocks adopt a small packet high-frequency transmission mode, while coarse-grained file blocks adopt a large packet merging transmission mode. The module also performs transmission integrity verification on each file block using the SHA-256 algorithm. The verification results and synchronization progress are fed back to the fault detection module in real time.

[0016] Optionally, the incremental recovery operation of the incremental fault recovery module includes: If the fault is a block corruption, pull the latest version of the corresponding block from the backup node; If the fault is a synchronization interruption, recover the blocks that have not been synchronized based on the synchronization progress; After recovery, the pre-processing module for conflict prediction verifies the data and updates the synchronization log after ensuring there are no conflicts.

[0017] Through the above technical solutions, the beneficial effects of this invention are as follows: By constructing a collaborative architecture comprising a fault detection module, a dynamic granularity synchronization module, a conflict prediction and preprocessing module, a file synchronization execution module, and an incremental fault recovery module, this invention achieves a comprehensive improvement in file synchronization performance and reliability in fault switching scenarios. Specifically, the dynamic granularity synchronization module dynamically outputs synchronization granularity and block size configurations based on multi-dimensional parameters, adapting to different file attributes, system states, and fault conditions, avoiding resource waste or synchronization delays caused by fixed granularity; the conflict prediction and preprocessing module, relying on this configuration and combining a machine learning model, identifies conflict risks in advance and performs tiered processing, transforming post-event conflict resolution into pre-event prevention, effectively reducing data inconsistency problems caused by conflicts; the incremental fault recovery module accurately locates faulty blocks by combining fault information and synchronization configuration, performing recovery operations only on damaged or unsynchronized blocks, avoiding the redundant overhead of full recovery; simultaneously, each module operates in conjunction with the synchronization configuration as the core, ensuring consistent response at each stage during fault switching, further enhancing the stability and continuity of system operation, ultimately achieving a comprehensive technical effect of improved file synchronization efficiency, reduced conflict incidence, faster fault recovery speed, and enhanced overall system availability. Attached Figure Description

[0018] Figure 1 This is a system module structure diagram of the file synchronization system for fault switching according to the present invention.

[0019] Figure 2 This is a flowchart of the weighted multi-factor decision-making algorithm for the dynamic granularity synchronization module in this invention.

[0020] Figure 3 This is a flowchart of the differentiated recovery operation of the incremental fault recovery module in this invention. Detailed Implementation

[0021] The foregoing and other technical contents, features and effects of the present invention are described in conjunction with the appendix below. Figure 1 To be continued Figure 3 The detailed description of the embodiments will make this clear. All structural details mentioned in the following embodiments are based on the accompanying drawings.

[0022] Exemplary embodiments of the present invention will now be described with reference to the accompanying drawings.

[0023] To achieve the above objectives, the present invention provides a file synchronization system for failover, such as... Figure 1 As shown, the system includes a fault detection module, a dynamic granularity synchronization module, a conflict prediction and preprocessing module, a file synchronization execution module, and an incremental fault recovery module. Each module operates based on the synchronization granularity parameters and block size configuration output by the dynamic granularity synchronization module, as detailed below: In one specific embodiment, the fault detection module collects fault information during the file synchronization process in real time and distributes it to the other modules.

[0024] First, the fault detection module identifies faults through a combination of multiple methods, namely, by using three differentiated technical means to cover the main fault types in file synchronization scenarios, and each technical means is designed for specific fault scenarios to ensure identification accuracy and efficiency.

[0025] Heartbeat detection refers to a technology in which nodes in the system (such as user terminals, cloud synchronization nodes, and backup nodes) periodically send preset small data packets, or "heartbeat packets," to determine whether communication between nodes is normal. Network interruptions or node failures are identified by monitoring the frequency and integrity of heartbeat packet reception. For example, if a user terminal sends a heartbeat packet to the cloud synchronization node and does not receive a response within a preset time (e.g., 2 seconds) three times consecutively, or if the received heartbeat packet contains missing data, it is determined to be a "network interruption failure." If the cloud node fails to send heartbeat packets to the backup node five times consecutively, it is determined to be a "node failure."

[0026] CRC, or Cyclic Redundancy Check, is an error detection technique that determines data integrity by performing polynomial calculations on transmitted or stored data to generate a fixed-length checksum, and then comparing the checksums at the sending and receiving ends to see if they match. CRC checks can identify corrupted file blocks. For example, when transmitting a 100MB PowerPoint file, the sending end calculates a CRC checksum for each 32KB block and sends it along with the data. The receiving end recalculates the CRC checksum for the same file block. If the received checksum for a particular file block does not match the sending checksum, it is considered a corrupted file block.

[0027] Access log analysis refers to the technique of statistically analyzing and extracting features from real-time records of user file operations (including access time, operation type such as read / modify / delete, and file block numbers involved) to discover potential abnormal behaviors. This submodule identifies precursory faults of multi-user concurrent conflicts through access log analysis. For example, if user A initiates a modification operation on the 2nd to 4th file blocks of a TXT file at 9:00, and user B initiates a modification operation on the 3rd to 5th file blocks of the same file at 9:00:05, log analysis reveals that the file blocks operated by the two users overlap and the time interval is extremely short, which is determined to be a precursory fault of multi-user concurrent conflicts.

[0028] After identifying the fault type, this module further integrates the associated information, including the list of files affected by the fault, the specific file block number, and the synchronization links involved. It then prioritizes the faults based on file importance and synchronization progress, dividing them into four levels from P0 to P3, with higher levels requiring priority handling. File importance is preset by the user or the system, such as being marked as "core business file," "general office file," or "temporary cache file." Synchronization progress refers to the percentage of the file that has been synchronized, such as 50% or 80%. For example, if a core file marked as "monthly financial report" (high importance) experiences file block corruption when the synchronization progress reaches 90%, it will be classified as level P3 and must be prioritized to prevent near-complete synchronization from failing. Conversely, a temporary screenshot file uploaded by the user (low importance) experiences network interruption when the synchronization progress is only 10%, and can be retried when resources are available.

[0029] After completing the integration and prioritization of fault information, this module will distribute complete fault information, including fault type, fault impact range, and fault priority, in real time to the dynamic granularity synchronization module, conflict prediction and preprocessing module, and incremental fault recovery module according to the system's preset communication protocol. For example, when this submodule distributes the information of "high-priority file block corruption fault (involving file blocks 120-125 of the monthly financial report)" to the dynamic granularity synchronization module, the dynamic granularity synchronization module can adjust the subsequent synchronization granularity of the file accordingly to adapt to the fault scenario; after being distributed to the incremental fault recovery module, the incremental fault recovery module can accurately locate the range of file blocks that need to be recovered, avoiding the waste of resources in full file recovery.

[0030] In one specific embodiment, the dynamic granularity synchronization module outputs synchronization granularity parameters and block size configuration based on multi-dimensional parameters. This serves as the core operating basis for the conflict prediction preprocessing module, the file synchronization execution module, and the incremental fault recovery module. This ensures that each module can work together to adapt to the dynamically changing operating environment in fault switching scenarios, avoiding problems such as low synchronization efficiency and resource waste caused by fixed configurations.

[0031] The multi-dimensional parameters relied upon by this module are divided into three categories, each corresponding to a key state of system operation and providing a comprehensive basis for decision-making. The first category is file attribute parameters, including file type, file size, and modification frequency. Different types of files have significantly different synchronization requirements. For example, text files are usually small in size and modified frequently, while video files are generally large in size and modified infrequently, requiring completely different synchronization strategies. The second category is system status parameters, including network bandwidth, storage space, and CPU load. Fluctuations in network bandwidth directly affect data transmission efficiency. For example, when network bandwidth drops sharply from 100Mbps to 20Mbps, the chunk size needs to be adjusted to reduce transmission pressure; when storage space is insufficient, metadata redundancy caused by excessive chunking needs to be avoided; and when CPU load is too high, the complexity of chunk calculation needs to be reduced. The third category is fault information parameters, including fault type, fault impact range, and fault priority. Different fault types correspond to different adaptation requirements. For example, when the fault type is network interruption, the resumability of the segment should be prioritized; when the fault impact range is a single file block corruption, the segment size should be accurately matched to locate the faulty block; when the fault priority is the highest level, the configuration should be output quickly to support emergency recovery.

[0032] To accurately calculate the optimal configuration, this module employs a weighted multi-factor decision-making algorithm. This algorithm standardizes multi-dimensional parameters, calculates a comprehensive score according to preset weights, and then matches the configuration based on the score. Figure 2As shown, the specific steps are as follows: The first step is parameter standardization. This process aims to eliminate differences in parameters with different units and ranges. Parameters with different units, such as network bandwidth (in Mbps), file size (in MB), and CPU load (in %), are uniformly converted into normalized values ​​in the range of 0-1 to ensure that each parameter can participate in the decision-making fairly. For example, the range of network bandwidth is set to 0-100Mbps, where 100Mbps corresponds to a normalized value of 1, and 20Mbps corresponds to a normalized value of 0.2; the range of file size is set to 0-2GB, where 2MB corresponds to a normalized value of 0.001, and 1GB corresponds to a normalized value of 0.5. The second step is weight calculation. The module assigns weights to the three types of parameters according to a preset ratio, with file attribute parameters accounting for 30%, system status parameters accounting for 40%, and fault information parameters accounting for 30%. The comprehensive score is calculated using the formula: "Comprehensive S = Normalized value of file attributes × 30% + Normalized value of system status × 40% + Normalized value of fault information × 30%". Taking a PPT file from a project as an example, its file attribute normalization value is 0.4 (file size 50MB, moderate modification frequency), system status normalization value is 0.5 (network bandwidth 50Mbps, storage free 30GB, CPU load 50%), and fault information normalization value is 0.3 (fault type is low-priority network fluctuation). Substituting these values ​​into the formula, we can get the comprehensive score S = 0.4 × 30% + 0.5 × 40% + 0.3 × 30% = 0.12 + 0.2 + 0.09 = 0.41. The third step is score matching. The module correlates the overall score with the synchronization granularity and block size according to fixed rules. When the overall score S is in the range [0, 0.3], it corresponds to a fine-grained 4KB / block. This configuration is suitable for scenarios with small file sizes, high modification frequency, and limited system resources. For example, a 2MB TXT file in a low-priority fault environment with a network bandwidth of 20Mbps, if the overall score is 0.25, can be precisely synchronized by dividing it into 4KB / blocks. When S is in the range [0.3, 0.7], it corresponds to a medium-grained 32KB / block, suitable for medium-sized files. In scenarios with moderate file size, moderate modification frequency, and stable system status, such as the 50MB PPT file with a comprehensive score of 0.41, dividing it into blocks of 32KB / block can balance synchronization accuracy and transmission efficiency; when S is in the range of [0.7, 1], the corresponding coarse-grained block size is 128KB / block, which is suitable for scenarios with large file size, low modification frequency, and sufficient system resources. For example, if a 1GB video file has a comprehensive score of 0.85 under a fault-free environment with a network bandwidth of 100Mbps, dividing it into blocks of 128KB / block can reduce the number of blocks and reduce CPU and storage overhead.

[0033] Furthermore, this module employs a dual mechanism combining periodic and real-time updates to ensure configuration timeliness. Under normal operating conditions, the module automatically re-collects all multi-dimensional parameters every 10 seconds, re-executes the weighted multi-factor decision-making algorithm, and updates the synchronization granularity and block size configuration to ensure adaptation to continuous changes in system status. For example, if network bandwidth is monitored every 10 seconds and increases from 50Mbps to 80Mbps, the original medium-granularity 32KB / block can be promptly adjusted to a coarse-granularity 128KB / block to improve transmission efficiency. If a sudden fault is detected, such as network interruption, node failure, or file block corruption, the module will immediately trigger a real-time update without waiting for periodic collection. It quickly recalculates and outputs a configuration adapted to the fault scenario. For instance, if a sudden 70% drop in network bandwidth occurs during file synchronization, the module will immediately re-collect parameters and calculate the score, adjusting the original coarse-granularity 128KB / block to a fine-granularity 4KB / block to avoid transmission timeouts due to excessively large single blocks, supporting synchronization continuity during failover.

[0034] In one specific embodiment, the conflict prediction and preprocessing module realizes early conflict prediction and preprocessing based on the synchronization granularity parameter and block size configuration, so as to avoid data inconsistency or synchronization interruption caused by the actual occurrence of conflict.

[0035] The input data for this module includes three key types of information, each providing crucial support for conflict prediction. The first type is dynamic parameters, specifically synchronization granularity parameters and block size configuration. These two parameters directly determine the model's accuracy in judging "block-level conflicts." For example, when the current synchronization granularity is medium (32KB / block), the model will analyze whether different users' operations overlap in 32KB units. If the block size configuration is adjusted to fine-grained (4KB / block), the model will further refine the analysis dimensions, more accurately capturing the risk of concurrent operations on small blocks of data. The second type is historical synchronization data, including historical conflict records and user modification habits. Historical conflict records provide patterns in conflict occurrence; for example, records show that the probability of conflict when multiple users modify the same project document is significantly higher between 9:00 AM and 11:00 AM on Mondays than at other times. User modification habits reflect individual operational preferences; for example, records show that user A consistently modifies meeting minutes at 3:00 PM every day, while user B tends to modify design drawings in the evening. This data helps the model predict the time and scope of user operations. The third type is real-time system status, including the number of concurrent accesses by multiple users and file block locking status. For example, if five users are accessing a product specification document simultaneously, and three of them have initiated modification operations, while blocks 10-15 of the document are unlocked, this type of real-time data allows the model to grasp the dynamic risks of the current synchronization scenario and avoid prediction bias caused by static data.

[0036] To generate accurate conflict prediction results, this module employs an LSTM-attention mechanism conflict prediction model. LSTM, or Long Short-Term Memory network, is a deep learning model capable of capturing long-term dependencies in time-series data, effectively uncovering temporal correlation patterns in historical data. The attention mechanism allows the model to actively focus on information crucial to conflict prediction during analysis, reducing interference from irrelevant data. The combined model balances learning temporal patterns with focusing on key information, improving prediction accuracy.

[0037] During model execution, the model first processes historical synchronization data through an LSTM layer to learn the timing characteristics of conflicts. For example, it identifies a high-incidence pattern of conflicts such as "multiple users concurrently modifying text files during weekday morning rush hours" from historical conflict records. Next, an attention mechanism is used to enhance the focus on dynamic parameters and real-time system status. For instance, when the synchronization granularity is fine-grained at 4KB / block, the model focuses on analyzing the overlap of access to adjacent small blocks by different users. When the number of concurrent accesses exceeds three and the file blocks are not locked, the model increases its attention to the conflict risk in that scenario. Finally, the model outputs three types of conflict prediction results: conflict probability (quantifying the likelihood of a conflict), conflict type (distinguishing between content conflicts, timing conflicts, permission conflicts, etc.), and high-risk block number (locating the specific file blocks where a conflict may occur). For example, for a core business report, the model outputs a conflict probability of 68%, a conflict type of content conflict, and high-risk block numbers of blocks 8-12.

[0038] Based on the conflict probability output by the model, this module performs tiered preprocessing to ensure that conflicts of different risk levels are handled appropriately. When the conflict probability is less than 30%, the module only records the synchronization log and does not interfere with the current synchronization operation. For example, if the conflict probability of a temporary notification file is 22%, the module will write the prediction result, the corresponding file information, and the real-time status to the synchronization log for later traceability, without affecting normal user synchronization. When the conflict probability is between 30% and 70%, the module pushes an operation warning to high-risk users. For example, for the core business report with a conflict probability of 68%, the module will push a warning message to the user who is initiating the modification, reminding them that other users are simultaneously operating on the high-risk block of the report, and suggesting that they confirm the necessity of the operation or wait for other users to complete before continuing, reducing the probability of conflict. When the conflict probability is greater than or equal to 70%, the module automatically locks the high-risk file block and generates a conflict resolution plan. For example, if the probability of conflict in a certain annual financial report is 85%, the module will immediately lock the high-risk blocks 5-9, allowing only the user who initiated the modification earliest to continue. At the same time, it will generate a conflict resolution plan, which includes retaining the latest modified version, backing up the old version before modification, and marking the modification areas of different users. This ensures that even if a sudden conflict occurs later, data consistency can be quickly restored. Especially in failover scenarios, this preprocessing can avoid synchronous crashes caused by the superposition of conflicts and failures, ensuring the overall availability of the system.

[0039] In one specific embodiment, the block size configuration and synchronization granularity parameters output by the file synchronization execution module and the dynamic granularity synchronization module sequentially complete file segmentation, differentiated data transmission and transmission integrity verification, and feed back the verification results and synchronization progress to the fault detection module in real time.

[0040] The module's workflow begins with file segmentation and block ID allocation. After receiving the block size configuration from the dynamic granularity synchronization module, the module standardizes the file to be synchronized into blocks according to this configuration, ensuring that the size of each block is completely consistent with the configuration. To avoid confusion between block data from different files and to facilitate subsequent location and traceability, the module assigns a unique block ID to each file block. This block ID is generated by combining the file ID, block sequence number, and timestamp. The file ID uniquely identifies the file to be synchronized, the block sequence number marks the block's position within the entire file, and the timestamp records the specific time the block was generated. The combination of these three elements ensures accurate differentiation even if different files have the same block sequence number, thanks to the file ID and timestamp. For example, a file named "Project Plan" with the file ID "DOC-20240601-001" is divided into 20 blocks at a configuration of 32KB / block. The 5th block was generated on June 1, 2024 at 14:30:25. The unique block ID of this block is "DOC-20240601-001_5_20240601143025". This ID can be used to quickly locate the file, location and generation time of the block.

[0041] After segmentation, the module adjusts its data transmission strategy based on the synchronization granularity parameters, employing differentiated transmission modes to adapt to file blocks of different granularities, thus balancing transmission efficiency and resource consumption. When the synchronization granularity is fine-grained, the corresponding file block size is small. The module uses a small-packet, high-frequency transmission mode, transmitting a single small file block at short intervals and high frequency. This mode reduces the amount of data transmitted per transmission, lowers the probability of transmission timeouts or failures, and facilitates timely detection of transmission anomalies. For example, for a text file with a synchronization granularity of 4KB / block, each file block is only 4KB. The module will transmit each block 5 times per second, sending only one 4KB data packet at a time, ensuring flexibility in the transmission process. When the synchronization granularity is coarse-grained, the corresponding file block size is larger. The module uses a large-packet merging transmission mode, merging multiple adjacent coarse-grained file blocks into a larger data packet for transmission, reducing the total number of data packets, decreasing the frequency of network connection establishment and disconnection, and improving overall transmission efficiency. For example, for a video file with a synchronization granularity of 128KB / block, the module will merge five 128KB file blocks into a single 640KB data packet for transmission. Compared to transmitting block by block, this reduces the number of data packets sent by 80%, significantly reducing network overhead.

[0042] During data transmission, this module uses the SHA-256 algorithm to verify the integrity of each file block, ensuring that the file block received by the receiving end is completely consistent with that sent by the sending end, thus preventing data corruption due to network fluctuations, data loss, or other reasons. The SHA-256 algorithm, or Secure Hash Algorithm 256-bit, is a cryptographic hash function that can convert input data of any size into a fixed-length hash value of 256 bits (32 bytes). The probability of different input data generating the same hash value is extremely low, providing high uniqueness and security.

[0043] The specific verification process is as follows: Before transmitting each file block, the sending end calculates the hash value of the block using the SHA-256 algorithm and sends the hash value along with the block data to the receiving end. After receiving the data, the receiving end re-executes the SHA-256 algorithm to calculate the hash value of the block data and compares the result with the hash value sent by the sending end. If the two are completely identical, the block transmission is considered complete; otherwise, the transmission is considered abnormal, and a retransmission mechanism needs to be triggered. For example, when the sending end transmits a 128KB video file block, the calculated SHA-256 hash value is "a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2c3d4e5f6a7b8c9d0e1f2a3b". The receiving end recalculates the hash value and it is exactly the same, thus confirming that the block transmission is complete.

[0044] Furthermore, this module provides real-time feedback to the fault detection module on the verification results (complete or abnormal) of each file block and the overall synchronization progress (the proportion of blocks that have been synchronized out of the total number of blocks). For example, if a file consists of 100 blocks, and 30 blocks have been transmitted and verified completely, the module will report "Blocks 1-30 verified completely, synchronization progress 30%" to the fault detection module; if block 31 is verified abnormally, the module will report "Block 31 verified abnormally, synchronization progress 30%". Based on this feedback information, the fault detection module can promptly identify faults such as corrupted file blocks or transmission interruptions, providing accurate real-time data for subsequent dynamic granular adjustments and fault recovery, ensuring timely response and abnormal handling of file synchronization in fault switching scenarios.

[0045] In one specific embodiment, the incremental fault recovery module combines the fault information output by the fault detection module with the synchronization configuration output by the dynamic granularity synchronization module to accurately locate the file blocks affected by the fault and perform incremental recovery operations, avoiding the waste of resources and time caused by full recovery, while ensuring the consistency of data after recovery through conflict checking.

[0046] First, this module obtains two types of key information from other modules in the system: one is fault information distributed by the fault detection module, including the fault type and the scope of its impact; the other is the synchronization configuration output by the dynamic granularity synchronization module, focusing on the block size configuration. This configuration is the core basis for accurately locating the faulty block, ensuring that the module does not need to scan the entire file, but only needs to lock the fault range by block size. For example, if a product design drawing file has a block corruption fault, the fault detection module reports that the affected file blocks are numbered 22-25, and the dynamic granularity synchronization module outputs that the block size configuration for this file is 32KB / block. The incremental fault recovery module can then directly lock the target to be recovered, namely the 32KB file blocks 22-25 in the design drawing, without needing to process other unaffected blocks.

[0047] After locating the fault block, this module performs differentiated incremental recovery operations based on the fault type. For example... Figure 3 As shown, for block corruption failures, the module retrieves the latest version of the corresponding file block from the backup node. The backup node is a dedicated node in the system that stores backup data of the latest versions of each file block, providing a traceable and recoverable data source when the original data is corrupted, preventing unrecoverable data loss due to the original node's data. For example, if blocks 22-25 of the design drawing are corrupted, the module will send a request to the backup node to retrieve only the latest versions of these four blocks, not the entire design drawing file. After retrieval, it directly replaces the corrupted blocks, completing partial repair. For synchronization interruption failures, the module recovers only the file blocks that have not yet been synchronized, based on the synchronization progress reported by the file synchronization execution module. For example, a 1GB training video file is divided into 782 blocks at a configuration of 128KB / block. After the synchronization execution module completes the synchronization of the first 300 blocks, a network interruption triggers a fault. The fault detection module determines that it is a synchronization interruption fault and reports that 300 blocks have been synchronized and blocks 301-782 have not been synchronized. The incremental fault recovery module then initiates synchronization recovery only for blocks 301-782, without having to retransmit the first 300 blocks that have been completed, which greatly reduces the resource overhead of repeated transmission.

[0048] After the recovery operation is complete, the module invokes the conflict resolution plan generated by the conflict prediction and preprocessing module to perform consistency verification on the recovered data. The conflict resolution plan, generated in advance by the conflict prediction and preprocessing module, includes version backup rules, modification area marking rules, and other content. It can be used to verify whether there are content conflicts, timing conflicts, or other issues between the recovered data and existing normal data. For example, after recovering blocks 15-18 of a project report, the module uses the version comparison rules in the plan to match the version timestamp of the recovered block with the timestamps of other blocks in the report. Simultaneously, it uses modification area marking rules to confirm whether the content of the recovered block is logically consistent with adjacent blocks. Only after ensuring there are no conflicts is the synchronization log updated. The synchronization log records detailed information such as the fault type, recovered file block number, recovery completion time, and verification results. This provides a basis for subsequent fault tracing and also provides feedback to the fault detection module that the fault has been resolved, ensuring the overall system synchronization process returns to normal.

[0049] Through the above processing, the incremental fault recovery module can significantly shorten the recovery time and reduce the consumption of network and storage resources in fault switching scenarios, while ensuring the consistency of data after recovery, providing important support for the stable operation of the system.

[0050] The above is a further detailed description of the present invention in conjunction with specific embodiments. It should not be considered that the specific implementation of the present invention is limited to this. For those skilled in the art to which the present invention pertains and related fields, any extensions, operation methods, and data substitutions made based on the technical solution concept of the present invention should fall within the protection scope of the present invention.

Claims

1. A file synchronization system for failover, characterized in that, It includes a fault detection module, a dynamic granularity synchronization module, a conflict prediction and preprocessing module, a file synchronization execution module, and an incremental fault recovery module; The fault detection module collects fault information during the file synchronization process in real time and distributes it to the other modules; The dynamic granularity synchronization module outputs synchronization granularity parameters and block size configuration based on multi-dimensional parameters, which serve as the core operating basis for the conflict prediction preprocessing module, file synchronization execution module and incremental fault recovery module. The conflict prediction and preprocessing module realizes early conflict prediction and preprocessing based on the synchronization granularity parameter and block size configuration. The file synchronization execution module performs file segmentation, transmission, and verification according to the configuration. The incremental fault recovery module performs incremental recovery operations by combining the fault information with the configuration.

2. The file synchronization system for failover as described in claim 1, characterized in that, The fault detection module identifies faults through a combination of multiple methods: Heartbeat detection is used to identify network outages or node crashes, CRC check is used to identify file block corruption, and access log analysis is used to identify precursors to multi-user concurrent conflicts. The fault priority is divided into levels based on file importance and synchronization progress.

3. The file synchronization system for failover as described in claim 1, characterized in that, The multi-dimensional parameters of the dynamic granularity synchronization module include file attribute parameters, system status parameters, and fault information parameters. The file attribute parameters include file type, file size, and modification frequency; The system status parameters include network bandwidth, free storage space, and CPU load. The fault information parameters include fault type, fault impact range, and fault priority.

4. The file synchronization system for failover as described in claim 3, characterized in that, The dynamic granularity synchronization module calculates the optimal synchronization granularity parameters and block size configuration through a weighted multi-factor decision algorithm. The weighted multi-factor decision algorithm standardizes each parameter, calculates a comprehensive score according to a preset weight, and then matches the synchronization granularity range and block size based on the comprehensive score.

5. The file synchronization system for failover as described in claim 4, characterized in that, The matching rule between the synchronization granularity range and the block size is as follows: When the overall score S∈[0, 0.3], it corresponds to a fine-grained 4KB / block; when S∈[0.3, 0.7], it corresponds to a medium-grained 32KB / block; and when S∈[0.7, 1], it corresponds to a coarse-grained 128KB / block. The dynamic granularity synchronization module re-collects parameters and updates the configuration every 10 seconds. If a sudden fault is detected, it will trigger an immediate update.

6. The file synchronization system for failover as described in claim 1, characterized in that, The input data for the conflict prediction preprocessing module includes dynamic parameters, historical synchronization data, and real-time system status. The dynamic parameters include synchronization granularity parameters and block size configuration; The historical synchronization data includes historical conflict records and user modification habits; The real-time system status includes the number of concurrent accesses by multiple users and the file block locking status.

7. The file synchronization system for failover as described in claim 6, characterized in that, The conflict prediction preprocessing module uses an LSTM-attention mechanism conflict prediction model to generate conflict prediction results. The conflict prediction results include conflict probability, conflict type and high-risk block number, and perform hierarchical preprocessing operations based on the conflict probability.

8. The file synchronization system for failover according to claim 7, characterized in that, The rules for the hierarchical preprocessing operation are as follows: Only record synchronization logs when the probability of conflict is less than 30%; When the probability of conflict is 30% or less and less than 70%, an operation alert will be pushed to high-risk users; When the probability of conflict is ≥70%, high-risk file blocks are automatically locked and conflict resolution plans are generated. The conflict types include content conflicts, timing conflicts, and permission conflicts, and the conflict resolution plan includes version backup rules and modification area marking rules.

9. The file synchronization system for failover as described in claim 1, characterized in that, The file synchronization execution module divides the file into blocks according to the block size configuration, and assigns a unique block ID to each file block. The block ID consists of the file ID, the block sequence number, and the timestamp. The file synchronization execution module adjusts the transmission strategy according to the synchronization granularity parameter. Fine-grained file blocks adopt a small packet high-frequency transmission mode, while coarse-grained file blocks adopt a large packet merging transmission mode. The module also performs transmission integrity verification on each file block using the SHA-256 algorithm. The verification results and synchronization progress are fed back to the fault detection module in real time.

10. The file synchronization system for failover as described in claim 2, characterized in that, The incremental recovery operation of the incremental fault recovery module includes: If the fault is a block corruption, pull the latest version of the corresponding block from the backup node; If the fault is a synchronization interruption, recover the blocks that have not been synchronized based on the synchronization progress; After recovery, the pre-processing module for conflict prediction verifies the data and updates the synchronization log after ensuring there are no conflicts.

Images