File transmission method, system and equipment and storage medium

Abstract

The invention relates to a file transmission method, system and device and a storage medium. The method comprises the following steps: establishing a connection with a client, sending test data packets of preset times to the client, and counting the bandwidth when the test data packets are transmitted each time; calculating the block length based on the bandwidth statistical data of the transmission test data packet; partitioning a file requested to be transmitted by the client according to the partitioning length to obtain a plurality of partitioning blocks, and constructing metadata of each partitioning block; transmitting a file requested to be transmitted by the client to the client in the form of a plurality of blocks and a plurality of metadata; and after receiving all the blocks, the client merges all the blocks based on the metadata of each block to obtain a transmitted file.

Description

Technical Field

[0001] The present invention relates to a file transmission method, system, equipment and storage medium, and belongs to the technical field of data transmission. Background Art

[0002] With the rapid development of network technology, the demand for file data transmission and storage has increased dramatically. However, traditional file data transmission systems have exposed a series of technical bottlenecks and shortcomings in meeting these demands: Traditional file data transmission systems mostly use a fixed block strategy, meaning that files are always segmented into fixed-size data blocks for transmission, regardless of network bandwidth fluctuations. The main drawback of this strategy is that it cannot dynamically adjust block size based on real-time changes in network bandwidth, resulting in wasted bandwidth resources or increased transmission errors. When network bandwidth is high, fixed-size blocks may not fully utilize the bandwidth, resulting in idle bandwidth resources. However, when network bandwidth is low, overly large blocks may cause transmission timeouts or errors, compromising transmission stability and reliability.

[0003] During file transfers, the efficient storage and parsing of breakpoint information is crucial for transfer recovery speed and overall performance. However, traditional systems typically use text formats like JSON to store breakpoint information. This format not only takes up a lot of space but also suffers from inefficient parsing, especially when processing large amounts of data. This inefficient storage and parsing of breakpoint information increases the time cost of transfer recovery and degrades the user experience.

[0004] In traditional file data transmission systems, metadata (such as file name, size, and block information) is often very large. This metadata not only requires additional storage space but also imposes an additional burden during the transmission process. Furthermore, the large metadata volume can lead to decreased transmission speeds and overall performance degradation. This drawback is particularly pronounced in scenarios that require frequent reading and writing, as well as precise retrieval of breakpoint information, such as debugging systems or version control tools.

[0005] With the continuous advancement and widespread adoption of network technology, network environments are becoming increasingly complex and dynamic. However, traditional file data transmission systems often lack adaptability to this dynamic network environment. They are unable to adjust transmission strategies in real time based on changing network conditions, thus failing to fully utilize network resources and ensure efficient and reliable transmission. This lack of adaptability is particularly prominent when transferring large files or in networks with high latency and packet loss rates.

[0006] In summary, when facing the needs of modern network environments, traditional file data transmission systems have exposed problems such as the limitations of fixed block strategies, low efficiency in breakpoint information storage and parsing, large metadata volume, and insufficient adaptability to dynamic network environments. Summary of the Invention

[0007] In order to solve the above problems in the prior art, the present invention proposes a file transmission method, system, device and storage medium.

[0008] The technical solutions of the present invention are as follows: In one aspect, the present invention provides a file transmission method, comprising the following steps: Establish a connection with the client, send a preset number of test data packets to the client and calculate the bandwidth of each test data packet transmission; Calculate the chunk length based on bandwidth statistics for transmitting test packets; The file requested by the client is divided into multiple blocks according to the block length, and metadata for each block is constructed; Transmitting the file requested by the client to be transmitted to the client in the form of multiple blocks and multiple metadata; After receiving all the blocks, the client merges all the blocks based on the metadata of each block to obtain the transferred file.

[0009] Preferably, the specific steps of calculating the block length based on bandwidth statistics of the transmission test data packet are: A bandwidth record queue with a preset length is constructed, and a bandwidth value each time a test data packet is transmitted to a client is input into the bandwidth record queue; Preset maximum block length value and minimum block length value; Calculate the average bandwidth value of all bandwidth records in the bandwidth record queue and compare it with the minimum block length. If the minimum block length is greater than the average bandwidth value, set the block length to the minimum block length. If the average bandwidth value is greater than the minimum block length, compare the average bandwidth value with the maximum block length and select the smaller value as the block length.

[0010] Preferably, the bandwidth record queue adopts a first-in-first-out principle. When the bandwidth record queue is full and a new bandwidth value is input into the bandwidth record queue, the bandwidth value at the end of the bandwidth record queue is discarded and the new bandwidth value is input into the head of the bandwidth record queue.

[0011] Preferably, when the bandwidth record queue is empty, the block length is set to the minimum block length value.

[0012] Preferably, during the process of transmitting the file to the corresponding client, a preset time interval is set, and the transmission status of all blocks is recorded once every preset time interval and written into the metadata corresponding to each block; When a file is interrupted during transmission to the corresponding client, if the client initiates a resume request, the untransmitted blocks are determined based on the transmission status in the block metadata and the untransmitted blocks are resumed.

[0013] Preferably, the block metadata also includes a block check code.

[0014] Preferably, the block check code is generated based on a CRC32 algorithm.

[0015] Preferably, the location information of all blocks is stored in a B+ tree, and the location information of the current block is queried in the B+ tree based on the metadata corresponding to the current block.

[0016] On the other hand, the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method of the present invention when executing the program.

[0017] In another aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which implements the method of the present invention when the program is executed by a processor.

[0018] The present invention has the following beneficial effects: 1. The present invention can dynamically adjust data block size based on real-time changes in network conditions, thereby fully utilizing bandwidth resources and avoiding bandwidth waste and increased transmission error rates. This improves data transmission efficiency and reliability, enabling the system to maintain stable transmission performance in different network environments.

[0019] 2. The present invention can quickly locate and restore the interrupted transmission part based on the block metadata, reducing the recovery delay and ensuring the continuity and stability of the transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0021] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0022] It should be understood that the step numbers used herein are only for convenience of description and are not intended to limit the order in which the steps are to be executed.

[0023] It should be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0024] The terms “include” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0025] The term "and / or" refers to and includes any and all possible combinations of one or more of the associated listed items.

[0026] See also Figure 1 , a file transmission method, comprising the following steps: Establish a connection with the client, send a preset number of test data packets to the client and calculate the bandwidth of each test data packet transmission; Calculate the chunk length based on bandwidth statistics for transmitting test packets; The file requested by the client is divided into multiple blocks according to the block length, and metadata for each block is constructed; Transmitting the file requested by the client to be transmitted to the client in the form of multiple blocks and multiple metadata; After receiving all the blocks, the client merges all the blocks based on the metadata of each block to obtain the transferred file.

[0027] In a specific embodiment, the block metadata is constructed based on TLV encoding.

[0028] In some embodiments, the specific steps of calculating the block length based on bandwidth statistics of the transmission test data packet are: A bandwidth record queue with a preset length is constructed, and a bandwidth value each time a test data packet is transmitted to a client is input into the bandwidth record queue; Preset maximum block length value and minimum block length value; Calculate the average bandwidth value of all bandwidth records in the bandwidth record queue and compare it with the minimum block length. If the minimum block length is greater than the average bandwidth value, set the block length to the minimum block length. If the average bandwidth value is greater than the minimum block length, compare the average bandwidth value with the maximum block length and select the smaller value as the block length.

[0029] In some embodiments, the bandwidth record queue adopts a first-in-first-out principle. When the bandwidth record queue is full and a new bandwidth value is input into the bandwidth record queue, the bandwidth value at the end of the bandwidth record queue is discarded and the new bandwidth value is input into the head of the bandwidth record queue.

[0030] In some embodiments, when the bandwidth record queue is empty, the chunk length is set to a minimum chunk length value.

[0031] In some embodiments, during the process of transmitting the file to the corresponding client, a preset time interval is set, and the transmission status of all blocks is recorded once every preset time interval and written into the metadata corresponding to each block; When a file is interrupted during transmission to the corresponding client, if the client initiates a retransmission request, the untransmitted blocks are determined based on the transmission status in the block metadata and the untransmitted blocks are retransmitted; In a specific embodiment, the transmission status of the blocks includes waiting for transmission, transmitting, transmission completed, and transmission failed. Blocks with the transmission status of waiting for transmission, transmitting, and transmission failed are regarded as untransmitted blocks.

[0032] In a specific embodiment, the transmission status is monitored through a heartbeat packet mechanism.

[0033] In some embodiments, the block metadata further includes a block check code. When the block is input to the corresponding client, the block check code is used to verify whether the block is complete.

[0034] In some embodiments, the block check code is generated based on a CRC32 algorithm.

[0035] In some embodiments, the location information of all blocks is stored in a B+ tree, and the location information of the current block is queried in the B+ tree based on metadata corresponding to the current block.

[0036] In a specific embodiment, the specific steps of constructing the B+ tree are: When storing files in blocks, record the physical storage starting point (i.e. the physical address of the first block). For each subsequent block generated, record its offset from the physical storage starting point. Use the offset of each block from the physical storage starting point as the index value of the leaf node, construct the same number of leaf nodes as the blocks, and input each leaf node from left to right in the order in which the blocks were generated; Generate a B+ tree based on all leaf nodes.

[0037] In a specific embodiment, the block metadata also includes a block unique identifier, and the unique identifier of each block is used as the index key of its corresponding leaf node in the B+ tree. When the client requests to resume transmission, the corresponding position of the untransmitted block is queried in the B+ tree through the unique identifier of the untransmitted block.

[0038] In a specific embodiment, the file is divided into 10 blocks, and the unique identifier of each block is set to 1 to 10 according to the block order. When the blocks are transmitted to the corresponding client, the client merges the blocks in ascending order according to the block unique identifiers to obtain the complete file, and cleans up the garbage blocks generated by the interrupted transmission.

[0039] In some embodiments, an electronic device is proposed, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in any embodiment of the present invention is implemented.

[0040] In some embodiments, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method according to any embodiment of the present invention is implemented.

[0041] In the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can represent: a, b, c, a and b, a and c, b and c or a and b and c, where a, b, c can be single or multiple.

[0042] Those skilled in the art will appreciate that the various units and algorithm steps described in the embodiments disclosed herein can be implemented using a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0043] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0044] In the several embodiments provided in this application, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of this application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory; hereinafter referred to as: ROM), random access memory (Random Access Memory; hereinafter referred to as: RAM), magnetic disk or optical disk, and other media that can store program code.

[0045] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention's description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A file transmission method, characterized in that: The following steps are involved: Establish a connection with the client, send a preset number of test data packets to the client and calculate the bandwidth of each test data packet transmission; Calculate the chunk length based on bandwidth statistics for transmitting test packets; The file requested by the client is divided into multiple blocks according to the block length, and metadata for each block is constructed; Transmitting the file requested by the client to be transmitted to the client in the form of multiple blocks and multiple metadata; After receiving all the blocks, the client merges all the blocks based on the metadata of each block to obtain the transferred file.

2. A file transmission method according to claim 1, characterized in that: The specific steps to calculate the block length based on the bandwidth statistics of the transmitted test data packets are as follows: A bandwidth record queue with a preset length is constructed, and a bandwidth value each time a test data packet is transmitted to a client is input into the bandwidth record queue; Preset maximum block length value and minimum block length value; Calculate the average bandwidth value of all bandwidth records in the bandwidth record queue and compare it with the minimum block length. If the minimum block length is greater than the average bandwidth value, set the block length to the minimum block length. If the average bandwidth value is greater than the minimum block length, compare the average bandwidth value with the maximum block length and select the smaller value as the block length.

3. A file transmission method according to claim 2, characterized in that: The bandwidth record queue adopts a first-in-first-out principle. When the bandwidth record queue is full and a new bandwidth value is input into the bandwidth record queue, the bandwidth value at the end of the bandwidth record queue is discarded and the new bandwidth value is input into the head of the bandwidth record queue.

4. A file transmission method according to claim 2, characterized in that: When the bandwidth record queue is empty, the block length is set to the minimum block length value.

5. A file transmission method according to claim 1, characterized in that: During the file transfer process to the corresponding client, the transmission status of all blocks is recorded once every preset time interval and written into the metadata corresponding to each block; When a file is interrupted during transmission to the corresponding client, if the client initiates a resume request, the untransmitted blocks are determined based on the transmission status in the block metadata and the untransmitted blocks are resumed.

6. A file transmission method according to claim 1, characterized in that: The block metadata also includes a block check code.

7. A file transmission method according to claim 6, characterized in that: The block check code is generated based on the CRC32 algorithm.

8. A file transmission method according to claim 1, characterized in that: The location information of all blocks is stored in a B+ tree, and the location information of the current block is queried in the B+ tree based on the metadata corresponding to the current block.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images