Intelligent slice-based distributed large file high-speed reliable transmission method

Abstract

The application relates to the technical field of computers and discloses a distributed super-large file high-speed reliable transmission method based on intelligent fragmentation, which comprises the following steps: file preprocessing and fragmentation strategy generation, second transmission is realized through cryptography hashing, and the fragmentation size and the transmission mode are dynamically selected; fragmentation transmission and state synchronization, the transmission state is recorded by relying on distributed caching, and breakpoint continuation is supported; multi-stage checking and exception processing, the checking is implemented at the fragmentation stage, the file stage and the storage stage, and local retransmission is triggered when failure occurs; file recombination and storage writing, the file is recombined in sequence and is adapted to various storage engines to perform efficient writing; state synchronization and resource cleaning, the state is updated and temporary data is cleaned up. The application effectively improves the efficiency, reliability and cross-platform compatibility of super-large file transmission and is suitable for big data and cloud computing environments.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method for high-speed and reliable transmission of distributed ultra-large files based on intelligent fragmentation. Background Technology

[0002] With the advancement of enterprise digital transformation and the construction of data platforms, the scale of data is growing rapidly, especially the demand for high-speed transmission of ultra-large files. Data platforms need to support the flow of multi-source, heterogeneous, and massive amounts of data, but traditional transmission solutions have limitations in terms of reliability, efficiency, storage compatibility, and operation and maintenance monitoring.

[0003] Traditional solutions suffer from several drawbacks in terms of transmission reliability. Due to the complexity of the intranet environment and bandwidth fluctuations, they lack effective fault tolerance and recovery mechanisms, leading to wasted resources during retransmissions and failing to meet service continuity requirements. Regarding transmission efficiency, fixed sharding strategies and single-threaded modes cannot adapt to network changes, resulting in low bandwidth utilization and high latency. In terms of storage compatibility, existing solutions are mostly limited to specific storage engines, lacking a unified abstraction layer and seamless switching capabilities. For operation and maintenance monitoring, traditional solutions lack end-to-end visualization and management, increasing operational complexity and fault recovery costs. While some technologies have attempted to address these issues, they cannot meet the demands of data platforms for handling ultra-large file transfers in high-concurrency, high-reliability, and multi-tenant environments. Therefore, a more intelligent, adaptive, and fault-tolerant transmission solution is needed. Summary of the Invention

[0004] This invention aims to address the shortcomings of traditional ultra-large file transfer solutions in terms of reliability, efficiency, and compatibility. Through technologies such as intelligent fragmentation, multi-level verification, and multi-storage adaptation, it achieves efficient, reliable, and cross-platform distributed file transfer, suitable for big data and cloud computing environments.

[0005] Therefore, this invention provides a method for high-speed and reliable distributed transmission of ultra-large files based on intelligent fragmentation, comprising the following steps: File preprocessing and fragmentation strategy generation steps: The client uploads file metadata to the server. The server generates a unique file identifier and checks whether the file already exists based on the file content hash value. If it exists, the transmission process is terminated to achieve file deduplication. The fragmentation management module dynamically calculates and selects the optimal fragmentation strategy based on the file size, file type and real-time network bandwidth, and determines the fragmentation level, fragment size and transmission mode. Fragmented transmission and status synchronization steps: The client requests to upload a token according to the fragment sequence number, and the server returns the token after verifying the fragment status; the client concurrently transmits fragmented data, the server receives the fragments and performs fragment-level real-time hash verification, and updates the fragmented transmission status in the distributed cache after successful verification; the transmission control module monitors the transmission process in real time, and supports breakpoint resumption after transmission interruption based on the fragment identifier, transmission progress and hash value status triplet recorded in the distributed cache; Multi-level verification and exception handling steps: The verification engine module performs real-time verification at the fragment level, file-level tree-structured aggregation verification, and storage-level consistency verification; when a verification failure is detected, a self-healing mechanism is activated to locate the problematic fragment and perform partial retransmission to ensure data integrity; File reassembly and storage write steps: After all fragments are transmitted and verified, the server reassembles the files in memory according to the fragment index table order; the storage adaptation module selects the corresponding storage engine adapter according to the system configuration, performs efficient data write operation, and performs final consistency verification after the write is completed; State synchronization and resource cleanup steps: Update the file status to complete, retain fragment metadata for a certain period of time in preparation for continued transmission needs; clean up temporary cache data and intermediate state information, and release system resources.

[0006] Technical effects: This invention significantly improves the transmission speed and bandwidth utilization of ultra-large files through intelligent dynamic fragmentation, multi-level verification mechanism and multi-storage engine adaptation, ensures data integrity and transmission reliability, supports breakpoint resume and cross-platform hot switching, greatly reduces operation and maintenance costs, and is suitable for high-concurrency big data scenarios. Attached Figure Description

[0007] Figure 1 This is a system functional architecture diagram of the present invention; Figure 2 This is a flowchart of the processing of the present invention; Figure 3 This is a flowchart of the method of the present invention. Detailed Implementation

[0008] To make the technical solution, advantages, and implementation details of the present invention clearer, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Those skilled in the art will understand that the embodiments described are merely examples and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, and improvements made without departing from the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0009] This invention addresses several key issues in distributed, large-file transmission, particularly in data platforms and big data applications where traditional solutions struggle to balance transmission efficiency, reliability, flexibility, and maintainability. Specifically, traditional solutions are inadequate in transmission reliability; after transmission interruptions due to network fluctuations or node failures, they lack efficient and accurate breakpoint resumption mechanisms, resulting in high retransmission costs. Regarding transmission efficiency, fixed sharding strategies and static concurrency models cannot adapt to dynamic network environments, leading to low bandwidth utilization and high transmission latency. In terms of storage compatibility, existing solutions are often bound to specific storage engines, making seamless switching and unified management in multi-cloud and hybrid environments difficult. Regarding operation and maintenance monitoring, the transmission process lacks end-to-end visualization capabilities, resulting in delayed anomaly detection and inefficient fault location and recovery. Finally, in terms of resource utilization, the lack of intelligent scheduling and sharding strategies based on real-time load leads to low system resource utilization and an inability to effectively handle high-concurrency transmission scenarios. To address these technical bottlenecks, this invention proposes a comprehensive solution.

[0010] The system of this invention includes the following core modules: The fragmentation management module is responsible for intelligent file fragmentation and reassembly control. It employs a multi-level dynamic fragmentation mechanism, adaptively adjusting the fragmentation strategy based on real-time network bandwidth, node load, and file characteristics. It integrates adaptive maximum transmission unit (MTB) detection technology to optimize fragment size and reduce protocol overhead. It supports automatic switching between 1MB, 10MB, and 100MB fragmentation modes. A built-in instant transfer function uses cryptographic hashing of file content to duplicate files, avoiding redundant transmission. During the fragmentation and reassembly phase, a fragmentation index table and a dual-buffering mechanism (memory + distributed KV storage) ensure rapid recovery and complete file reassembly after transmission interruptions.

[0011] Transmission Control Module: Responsible for scheduling and status management of fragmented transmission. It implements high-frequency transmission status recording (fragment identifier, progress, hash value) and breakpoint resumption functionality based on distributed caching; it employs a multi-threaded concurrent scheduling algorithm to dynamically allocate transmission tasks based on real-time node load; and it avoids resource conflicts and duplicate transmissions through state locks and incremental snapshot technology, ensuring high efficiency and reliability in the transmission process.

[0012] Verification Engine Module: Responsible for ensuring the integrity of the entire data transmission chain. It constructs a multi-level verification system at the shard level, file level, and storage level: at the shard level, hash values ​​are calculated in real time, and shards that fail to transmit are retransmitted a limited number of times; at the file level, shard hashes are aggregated to generate a global hash value, and efficient tree-structured aggregation verification is performed in memory; at the storage level, deep value comparison is performed when data is written to disk. The module also has self-healing capabilities; when verification fails, it uses intelligent logs to locate problematic shard groups and triggers precise retransmissions.

[0013] Storage Adaptation Module: Responsible for unified storage interface abstraction and multi-engine support. Defines standard storage interfaces and encapsulates core read, write, and verification methods; provides adapters for various storage engines such as local file systems, HDFS (Distributed File System), and MinIO (Object Storage System), supports zero-copy writes and hot-swapping functions, ensuring that the system can flexibly adapt to various storage environments without affecting data consistency.

[0014] Status monitoring module: Responsible for the observability of the entire system link. It collects indicators such as transmission progress, speed, success rate, and node health in real time; pushes monitoring data to the visualization front end in real time via TCP protocol; performs real-time detection and alarm for transmission anomalies and node failures, supports multiple notification methods such as email and SMS, and manual intervention interface, greatly improving the maintainability of the system.

[0015] Figure 3 This is a flowchart of the method of the present invention. The following is a detailed description. The present invention provides a method for high-speed and reliable distributed transmission of ultra-large files based on intelligent fragmentation, the method comprising the following steps: Step 1: File preprocessing and fragmentation strategy generation. The client uploads file metadata (filename, size, type, etc.) to the server. The server generates a unique file ID and checks if the file already exists using the instant transfer function. The fragmentation management module dynamically calculates the optimal fragmentation strategy based on file size, type, and real-time network conditions, determining the fragmentation level, size, and transmission mode.

[0016] Step 2: Fragmented Transmission and Status Synchronization. The client requests and uploads a token according to the fragment sequence number. The server verifies the fragment status and returns the token. The client concurrently transmits fragmented data. The server receives the fragments and performs fragment-level verification. Upon successful verification, the server updates the transmission status in the distributed cache. The transmission control module monitors the transmission process in real time and supports resuming interrupted transmissions.

[0017] Step 3: Multi-level verification and exception handling. The verification engine module performs real-time verification at the shard level, aggregate verification at the file level, and consistency verification at the storage layer. When a verification failure is detected, a self-healing mechanism is activated to locate the problematic shard and perform partial retransmission to ensure data integrity.

[0018] Step 4: File reassembly and storage writing. After all fragments are transmitted, the server reassembles the files according to the fragment index table. The storage adaptation module selects the corresponding storage engine adapter according to the configuration to perform efficient write operations. After the write is completed, an eventual consistency check is performed.

[0019] Step 5: Status synchronization and resource cleanup. Update the file status to "Completed" and retain fragment metadata for a certain period of time in case of continued transmission needs. Clear temporary cache data and release system resources.

[0020] like Figure 1As shown, Figure 1 This is a functional architecture diagram of the system of the present invention. The system consists of five core modules that work together to achieve distributed, high-speed, and reliable transmission of ultra-large files of TB level and above.

[0021] The fragmentation management module, as the system's entry point, bears the crucial responsibility of file analysis and fragmentation strategy formulation. In its implementation, this module first performs a multi-dimensional scan of the source files, acquiring metadata information including file size, type, and hash characteristics. Subsequently, it integrates real-time network bandwidth fluctuations, the current load status of each node, and historical transmission performance data, dynamically generating the optimal fragmentation scheme through a built-in intelligent decision-making model. This scheme not only determines the basic fragment size and number but also selects different fragmentation modes based on file type characteristics, such as sequential fragmentation, skip fragmentation, or overlapping fragmentation, to adapt to different transmission scenario requirements. Fragment metadata (including offset, size, priority, and status identifiers) is persistently stored in a structured form in a distributed KV (key-value store) database, while a hot-cache copy is maintained in high-performance memory, enabling millisecond-level status query and update responses. Furthermore, this module integrates advanced instant transfer functionality. By calculating the cryptographic hash value of the file content and intelligently comparing it with records in the metadata database, it can accurately identify duplicate files, avoiding unnecessary consumption of network bandwidth and storage resources.

[0022] The transmission control module, acting as the system's scheduling hub, enables fine-grained management and full lifecycle control of fragmented transmission tasks. Built on a distributed caching architecture, this module maintains a state triplet (fragment identifier, transmission progress, hash value) for each fragment and ensures the atomicity and consistency of state updates through a distributed lock mechanism. For task scheduling, the module employs an adaptive load balancing algorithm, monitoring key metrics such as CPU utilization, memory usage, and network I / O of each transmission node in real time, dynamically allocating fragmented tasks to the most suitable node for execution. During transmission, the module continuously records fragment state changes and generates incremental snapshots. These snapshots are not only used for real-time monitoring but also provide a foundation for rapid recovery after transmission interruptions. When network jitter or node failure causes transmission interruptions, the module can intelligently scan these state snapshots to accurately pinpoint the offset of the last successful transmission, achieving precise breakpoint resumption without manual intervention, greatly improving the resilience and reliability of the transmission task.

[0023] The verification engine module constructs a multi-layered, three-dimensional verification guarantee system throughout the entire data transmission chain. After a fragment is transmitted, the module immediately calculates the hash value of that fragment and compares it with the expected value to ensure the integrity of the individual fragment data. For fragments that fail verification, the module initiates a limited number of automatic retransmissions to avoid task failures caused by temporary network problems. When all fragments have been transmitted, the module aggregates the hash values ​​of all fragments using an efficient tree-structure algorithm to generate a global file checksum and performs rapid verification in memory. This design significantly reduces reliance on disk I / O and improves verification efficiency. Before the data is finally written to disk, the module also performs storage-level consistency verification by comparing the hash digests of the data before and after writing to ensure the consistency of data persistence. Notably, this module has strong self-healing capabilities. When an anomaly is detected during the verification process, it can accurately locate the problematic fragment group by analyzing intelligent logs and trigger a partial retransmission, avoiding the resource waste of full retransmission in traditional solutions and significantly improving the system's fault tolerance and processing efficiency.

[0024] The storage adaptation module achieves unified support and seamless integration for heterogeneous storage environments through an abstract interface design. This module defines a standard storage interface specification, encapsulating core functions including read / write operations, data verification, and space management. Based on this specification, the module provides adapter implementations for various storage engines, supporting local file systems, HDFS distributed file systems, MinIO object storage, and other backend storage solutions. For data writing, the module employs zero-copy technology, minimizing the overhead of data replication between user space and kernel space, thus improving write efficiency. Regarding storage engine switching, users can achieve hot-swapping of drivers through simple configuration changes; the system automatically performs necessary state synchronization and data migration, ensuring that the integrity and consistency of transmitted data are not affected. This design greatly enhances the system's flexibility and scalability, enabling it to adapt to various complex deployment environments.

[0025] The status monitoring module provides comprehensive observability support for the system, enabling real-time visual monitoring of the entire transmission link. This module uses data collection technology to gather multiple key indicators in real time, including transmission progress, transmission rate, success rate, and node health status. The collected data is pushed to the front-end visualization interface in real time via efficient TCP protocol, providing operations and maintenance personnel with an intuitive display of the system's operational status. The module has a built-in intelligent alarm mechanism that can detect and issue warnings in real time for events such as transmission anomalies, node failures, and performance degradation, supporting alarm information to be sent via email, messages, and other channels. Simultaneously, the module provides rich manual intervention interfaces, allowing operations and maintenance personnel to manually schedule tasks, pause tasks, and release resources when necessary, greatly improving the system's maintainability and fault recovery capabilities. All monitoring data is aggregated, analyzed, and persistently stored, providing data support for subsequent system optimization and capacity planning.

[0026] like Figure 2 As shown, Figure 2 This is a flowchart of the processing of the present invention. The method includes the following steps in its specific implementation: Task initialization and fragmentation strategy generation: After the client initiates a file transfer request, the system first performs a quick scan of the source file to obtain basic metadata such as file size and type. The fragmentation management module combines real-time network bandwidth, node load, and file characteristics to generate the optimal fragmentation scheme through a dynamic decision-making algorithm, determining the fragmentation granularity, transmission priority, and concurrency strategy, and generating a unique identifier and metadata record for each fragment.

[0027] Sharding Task Scheduling and Node Allocation: Based on the sharding strategy, the transmission control module selects the optimal node from the available node cluster according to real-time load indicators and establishes multiple parallel transmission channels. Each sharding task is encapsulated as an independent transmission unit, containing sharding data, verification information, and transmission control parameters, and is distributed to the selected node through a load balancing algorithm.

[0028] Fragmented Transmission and State Synchronization: After receiving the fragmented task, each transmission node establishes an independent transmission thread to execute data upload. During transmission, nodes report fragmented transmission progress, rate, and other status information to a distributed cache in real time. The transmission control module ensures state consistency among multiple nodes through a state lock mechanism. When transmission is interrupted, the system can accurately recover based on the last recorded state snapshot.

[0029] Multi-level verification and exception handling: After fragment transmission is complete, the verification engine immediately calculates the fragment hash value for real-time verification. Fragments that fail verification are added to the retransmission queue and rescheduled by the transmission control module. After all fragments have been transmitted, file-level tree-structured aggregation verification is performed to verify overall data integrity. Before data is written to disk, storage-level consistency verification is performed to ensure write consistency.

[0030] File Reassembly and Storage Write: After successful verification, the system reassembles the files in memory according to the ordered structure of the shard index table. The storage adaptation module selects the corresponding storage engine adapter based on the configuration, performs efficient data write operations, supports zero-copy writes and dynamic replica adjustments, and performs eventual consistency verification upon completion.

[0031] State synchronization and resource release: After the file is written, the system updates the task status to complete and archives and saves key metadata information. Temporary cache data and intermediate state information are cleared, and occupied network connections and memory resources are released, completing the entire transmission lifecycle management.

[0032] Through the coordinated operation of the above modules and steps, this invention achieves efficient and reliable transmission of ultra-large files in a distributed environment, with good scalability, maintainability and compatibility, and is suitable for high-concurrency and high-volume data integration and migration scenarios.

[0033] The parts of this invention not described in detail are well-known in the field.

[0034] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is impossible to exhaustively list all embodiments here. All obvious variations or modifications derived from the technical solutions of the present invention are still within the protection scope of the present invention.

Claims

1. A method for high-speed and reliable distributed transmission of ultra-large files based on intelligent fragmentation, characterized in that, Includes the following steps: The file preprocessing and fragmentation strategy generation steps are as follows: The client uploads file metadata to the server. The server generates a unique file identifier and checks whether the file already exists based on the file content hash value. If it exists, the transmission process is terminated to achieve file deduplication. The fragmentation management module integrates adaptive maximum transmission unit detection technology. Based on the file size, file type, real-time network bandwidth status and detection results, it dynamically calculates and selects the optimal fragmentation strategy to determine the fragmentation level, fragmentation size and transmission mode. The fragment transmission and status synchronization steps are as follows: the client requests to upload a token according to the fragment sequence number, and the server returns the token after verifying the fragment status. The client concurrently transmits fragmented data, the server receives the fragments and performs fragment-level real-time hash verification, and updates the fragment transmission status in the distributed cache after successful verification; The transmission control module monitors the transmission process in real time, based on the triplet consisting of fragment identifier, transmission progress and hash value status recorded in the distributed cache, and uses distributed state lock and incremental snapshot technology to synchronize the status, supporting breakpoint resumption after transmission interruption. The multi-level verification and exception handling steps include fragment-level, file-level, and storage-level verification. Among them, file-level verification is to aggregate fragment hash values ​​in memory using a tree-structured algorithm to generate a global checksum, thereby reducing the dependence on disk input and output. If any step fails to verify, the problematic fragment group is accurately located by analyzing the logs and added to the retransmission queue. The transmission control module then reschedules the retransmission, thereby triggering the local retransmission mechanism. After all fragments have been transmitted and verified, the server reassembles the file in memory according to the fragment index table order. The storage adaptation module selects the corresponding storage engine adapter according to the system configuration and uses zero-copy technology to write data to avoid redundant copying of data between user space and kernel space memory, performs efficient data writing operations, and performs eventual consistency verification after writing is completed. The status synchronization and resource cleanup steps update the file status to "completed," retain fragment metadata for a certain period of time in preparation for continued transmission needs, clean up temporary cache data and intermediate status information, and release system resources.

2. The method according to claim 1, characterized in that, In the file reorganization and storage writing steps, the storage adaptation module provides support for multiple storage types, including local file systems, distributed file systems, and object storage systems, and supports dynamic switching between different storage systems through configuration changes, automatically maintaining data consistency and service continuity during the switching process.

3. The method according to claim 1, characterized in that, In the fragmentation strategy generation step, the file deduplication function specifically calculates the cryptographic hash value of the file content and compares it with existing records in the metadata database. If the hash values ​​match, the file is determined to be a duplicate to avoid redundant transmission.

4. The method according to claim 1, characterized in that, In the fragmented transmission step, the transmission control module adopts an adaptive load balancing algorithm to dynamically allocate fragmented tasks to the optimal node for execution based on the CPU utilization, memory usage, and key network input / output indicators of each transmission node.

5. The method according to claim 1, characterized in that, Following the state synchronization and resource cleanup steps, a state monitoring step is also included: real-time collection of transmission progress, transmission rate, success rate and node health status indicators through data tracking technology, and real-time push of monitoring data to the visualization front end through the transmission control protocol. Real-time detection and alarm for transmission anomalies and node failures, supporting multiple notification methods and manual intervention interfaces; In the status monitoring step, the system establishes a hierarchical alarm mechanism, which automatically triggers different notification channels and operation and maintenance intervention processes according to the severity of the anomaly, so as to realize intelligent fault response and handling.

6. The method according to claim 1, characterized in that, In the fragmentation strategy generation step, the fragmentation management module dynamically generates the optimal fragmentation scheme, including fragment size, number, and fragmentation mode, by comprehensively considering real-time network bandwidth fluctuations, current node load status, and historical transmission performance data, using an intelligent decision model.

Images