File transfer method and device based on rdma network

Abstract

The application discloses a file transmission method and device based on an RDMA network, and relates to the technical field of data processing.The method comprises the following steps: a sending end dynamically determines the encoding parameters corresponding to an encoding block based on to-be-transmitted file information, current link state information and transmission resource state information fed back by a receiving end, and performs RaptorQ encoding on the to-be-transmitted file information to obtain source symbols and initial repair symbols; the application determines additional repair symbols based on preset feedback channels and feedback information sent by the receiving end; and the source symbols, the initial repair symbols and the additional repair symbols are sent to the receiving end through preset RDMA data channels.The application dynamically determines the encoding parameters according to the to-be-transmitted file information, the link state information and the decoding freedom gap fed back by the receiving end, the receiving end calculates the decoding freedom gap of each encoding block, and the sending end generates the additional repair symbols according to the decoding freedom gap as required, so that the dependence on network layer packet loss retransmission is reduced.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a file transfer method and apparatus based on an RDMA network. Background Technology

[0002] With the development of artificial intelligence, scientific computing, cloud-edge collaboration, and computing power networks, the demand for large-scale file transfers between computing power centers is constantly increasing. For example, after a computing power center completes data preprocessing, it needs to synchronize the training data to another computing power center; or after a computing power center completes model training, it needs to synchronize the model checkpoint file to a remote disaster recovery center.

[0003] Traditional file transfer methods typically rely on TCP, multi-connection TCP, object storage synchronization protocols, or application-layer fragmentation retransmission mechanisms. While these methods perform well in local area networks (LANs) or short-distance networks, they still present significant challenges in long-distance, high-bandwidth networks spanning multiple computing centers. In these long-distance networks, link round-trip times (RTTs) are high, bandwidth-delay products are large, and network jitter and occasional packet loss can significantly impact transmission stability. If the file transfer system relies entirely on network-layer or transport-layer packet loss retransmission mechanisms, a long RTT (Round-Trip Time) is required to recover after packet loss, easily leading to decreased throughput. Summary of the Invention

[0004] The main objective of this application is to provide a file transfer method and apparatus based on RDMA networks, aiming to solve the technical problem of balancing throughput, reliability and redundancy overhead in large-scale file transfers across computing centers.

[0005] To achieve the above objectives, this application proposes a file transfer method based on an RDMA network, applied at the sending end, the method comprising: The encoding parameters corresponding to the encoding block are dynamically determined based on the information of the file to be transmitted, the current link status information, and the transmission resource status information fed back by the receiver. The current link status information includes at least one of bandwidth, round-trip time, recent link packet loss rate, normalized link jitter, and current effective throughput. The transmission resource status information includes at least one of the following: the sender's RDMA transmission request processing capability, registered memory capacity, receiver's buffer occupancy status, and receiver's decoding capability. The encoding parameters include at least one of the following: encoding block size, symbol size, number of source symbols, initial redundancy rate, and decoding safety margin. Based on the encoding parameters, the information of the file to be transmitted is RaptorQ encoded to obtain the source symbol and the initial repair symbol. The receiver receives feedback information sent by the receiving end based on a preset feedback channel, and determines additional repair symbols based on the feedback information. The feedback information includes the decoding degree of freedom gap of each coding block. The decoding degree of freedom gap is used to characterize the number of effective symbols that the coding block needs to supplement in order to meet the decoding conditions. The source symbol, the initial repair symbol, and the additional repair symbol are sent to the receiving end through a preset RDMA data channel, wherein the preset RDMA data channel is a channel for data transmission built based on an RDMA network.

[0006] In one implementation, the step of dynamically determining the encoding parameters corresponding to the encoding block based on the file information to be transmitted, the current link state information, and the transmission resource state information fed back by the receiving end includes: Calculate the bandwidth-delay product based on the link bandwidth and link round-trip time in the current link status information; The coding block size is dynamically determined based on the bandwidth-delay product, the coding block adjustment coefficient, the recent link packet loss rate in the current link state information, and the state adjustment factor. The symbol size is dynamically determined based on the bandwidth-delay product and the number of target symbols in transit. The initial redundancy rate is determined based on the recent link packet loss rate, normalized link jitter, and minimum initial redundancy rate in the current link status information. The coding parameters corresponding to the coding block are determined based on the coding block size, the symbol size, and the initial redundancy rate.

[0007] In one implementation, the step of performing RaptorQ encoding on the file information to be transmitted based on the encoding parameters to obtain source symbols and initial repair symbols includes: Based on the block size and the symbol size, perform RaptorQ encoding on the file information to be transmitted to determine the source symbol; The number of initial repair symbols is determined based on the initial redundancy rate and the number of source symbols, and initial repair symbols are generated based on the number of initial repair symbols.

[0008] In one implementation, the step of receiving feedback information sent by the receiving end based on a preset feedback channel and determining additional repair symbols based on the feedback information includes: The recent link risk is determined based on the preset basic protection redundancy rate, recent link packet loss rate, normalized link jitter, and recent decoding failure rate in the link status information. The amount of protection redundancy is determined based on the recent link risks and the number of source symbols. Additional repair symbols are determined based on the decoding degree-of-freedom gap and the protection redundancy amount received from the feedback information sent by the receiving end through the preset feedback channel.

[0009] In one implementation, sending the source symbol, the initial repair symbol, and the additional repair symbol to the receiving end via a preset RDMA data channel includes: When multiple coding blocks have simultaneous degree-of-freedom gaps, the sending priority of the additional repair symbols is determined based on the decoding degree-of-freedom gap, the coding block waiting time, the position of the coding block in the file reassembly order, and the receiving end buffer pressure state. Based on the transmission priority and the preset RDMA data channel, the source symbol, the initial repair symbol, and the additional repair symbol are sent to the receiving end.

[0010] To achieve the above objectives, this application proposes a file transfer method based on an RDMA network, applied at the receiving end, the method comprising: Receive encoded information from the transmitter through a preset RDMA data channel; The number of valid encoded symbols received for each encoded block is counted, and the decoding degree of freedom gap for each encoded block is determined based on the decoding threshold. Feedback information is generated based on the decoding degree of freedom gap of each coding block, and the feedback information is sent to the transmitting end through a preset RDMA feedback channel, wherein the preset RDMA feedback channel is a channel separate from the preset RDMA data channel; The additional repair symbols sent by the transmitting end are received through a preset RDMA data channel, and RaptorQ decoding is performed to obtain the decoded coded block; The decoded encoded block is written to the target file according to the file offset position.

[0011] In one implementation, the step of counting the number of valid encoded symbols received for each encoded block and determining the decoding degree-of-freedom gap for each encoded block based on a decoding threshold includes: Perform integrity verification on the symbols in the encoded information sent by the sending end, and obtain the verification result; If the verification result is successful, then update the number of valid coded symbols in the corresponding coded block; The decoding threshold is calculated based on the number of source symbols sent by the sending end and the decoding security margin. The decoding degree of freedom gap for each coded block is determined based on the number of effective coded symbols and the decoding threshold.

[0012] In one implementation, calculating the decoding threshold based on the number of source symbols transmitted by the sending end and the decoding security margin includes: The decoding safety margin is determined based on the number of source symbols sent by the sending end, the recent link packet loss rate in the link status information, the recent code block decoding failure rate, and the preset minimum safety margin. The decoding threshold is determined based on the number of source symbols and the decoding security margin.

[0013] Furthermore, to achieve the above objectives, this application also proposes a file transfer device based on an RDMA network, applied at the sending end, wherein the RDMA network-based file transfer device includes: The encoding parameter decision module is used to dynamically determine the encoding parameters corresponding to the encoding block based on the file information to be transmitted, the current link status information, and the transmission resource status information fed back by the receiver. The current link status information includes at least one of bandwidth, round-trip time, recent link packet loss rate, normalized link jitter, and current effective throughput. The transmission resource status information includes at least one of the following: the sender's RDMA transmission request processing capability, registered memory capacity, receiver's buffer occupancy status, and receiver's decoding capability. The encoding parameters include at least one of the following: encoding block size, symbol size, number of source symbols, initial redundancy rate, and decoding safety margin. The RaptorQ encoding and redundancy generation module is used to perform RaptorQ encoding on the file information to be transmitted based on the encoding parameters to obtain source symbols and initial repair symbols. The feedback processing and repair scheduling module is used to receive feedback information sent by the receiving end based on a preset feedback channel, and to determine additional repair symbols based on the feedback information. The feedback information includes the decoding degree of freedom gap of each coding block. The decoding degree of freedom gap is used to characterize the number of effective symbols that the coding block needs to supplement in order to meet the decoding conditions. The RDMA transmission scheduling module is used to send the source symbol, the initial repair symbol, and the additional repair symbol to the receiving end through a preset RDMA data channel. The preset RDMA data channel is a channel for data transmission built based on the RDMA network.

[0014] Furthermore, to achieve the above objectives, this application also proposes a file transfer device based on an RDMA network, applied at the receiving end, wherein the RDMA network-based file transfer device includes: The RDMA receiver and symbol management module is used to receive encoded information from the transmitter through a preset RDMA data channel; The coding block state management module is used to count the number of valid coded symbols received for each coding block and determine the decoding degree of freedom gap for each coding block based on the decoding threshold. The decoding degree of freedom calculation and feedback module is used to generate feedback information based on the decoding degree of freedom gap of each coding block, and send the feedback information to the transmitting end through a preset RDMA feedback channel. The preset RDMA feedback channel is a channel separate from the preset RDMA data channel. The RaptorQ decoding module is used to receive additional repair symbols sent by the transmitter through a preset RDMA data channel, and perform RaptorQ decoding to obtain the decoded encoded block; The file reconstruction and integrity verification module is used to write the decoded encoded block into the target file according to the file offset position.

[0015] One or more technical solutions proposed in this application have at least the following technical effects: In this application, the sending end dynamically determines the encoding parameters corresponding to the encoding block based on the information of the file to be transmitted, the current link status information, and the transmission resource status information fed back by the receiving end. The encoding parameters include the encoding block size, symbol size, and initial redundancy rate. The sending end performs RaptorQ encoding on the information of the file to be transmitted based on the encoding parameters to obtain source symbols and initial repair symbols. The sending end receives feedback information sent by the receiving end through a preset feedback channel and determines additional repair symbols based on the feedback information. The feedback information includes the decoding degree of freedom gap of each encoding block. The decoding degree of freedom gap is used to characterize the number of effective symbols that the encoding block needs to supplement to meet the decoding conditions. The sending end sends the source symbols, initial repair symbols, and additional repair symbols to the receiving end through a preset RDMA data channel. The preset RDMA data channel is a channel for data transmission built based on an RDMA network. The receiving end receives encoded information from the sending end through a preset RDMA data channel. The receiving end counts the number of valid symbols received for each encoded block and determines the decoding degree-of-freedom gap for each encoded block based on a decoding threshold. The receiving end generates feedback information based on the decoding degree-of-freedom gap for each encoded block and sends this feedback information to the sending end through a preset RDMA feedback channel. The receiving end receives additional repair symbols sent by the sending end through the preset RDMA data channel and performs RaptorQ decoding to obtain the decoded encoded block. The decoded encoded block is then written to the target file according to the file offset position. This application establishes a closed-loop transmission control process at the file transfer application layer between the sending and receiving ends. The sending end dynamically determines the RaptorQ encoding parameters based on the file information to be transmitted, link status information, and the decoding degree-of-freedom gap fed back by the receiving end. The receiving end calculates the decoding degree-of-freedom gap for each encoded block based on the actual reception situation, and the sending end generates additional repair symbols as needed based on the decoding degree-of-freedom gap, thereby reducing reliance on network layer packet loss retransmission. This avoids the problem of balancing throughput, reliability, and redundancy overhead in large-scale file transfers across computing centers. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating an embodiment of the file transfer method based on an RDMA network provided in this application. Figure 2 This is a schematic diagram of the overall architecture of the file transfer method based on RDMA network provided in Embodiment 1 of this application; Figure 3 This is a schematic diagram of the link-aware RaptorQ coding parameter dynamic determination mechanism provided in Embodiment 1 of the file transfer method based on RDMA network in this application. Figure 4 This is a schematic diagram of the adaptive repair symbol generation mechanism based on decoding degree of freedom feedback provided in Embodiment 1 of the file transfer method based on RDMA network of this application; Figure 5 This is a flowchart illustrating Embodiment 2 of the file transfer method based on an RDMA network in this application; Figure 6 This is a schematic diagram of the complete RDMA file transfer process provided in Embodiment 2 of the file transfer method based on RDMA network of this application; Figure 7 This is a schematic diagram of the module structure provided in Embodiment 1 of the file transfer device based on an RDMA network of this application; Figure 8 This is a schematic diagram of the module structure provided in Embodiment 2 of the file transfer device based on RDMA network of this application; Figure 9 This is a schematic diagram of the device structure of the hardware operating environment involved in the file transfer method based on RDMA network in the embodiments of this application.

[0019] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0020] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0021] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0022] The main solution of this application embodiment is as follows: the sending end dynamically determines the encoding parameters corresponding to the encoding block based on the file information to be transmitted, the current link status information, and the transmission resource status information fed back by the receiving end. The encoding parameters include the encoding block size, symbol size, and initial redundancy rate. The sending end performs RaptorQ encoding on the file information to be transmitted based on the encoding parameters to obtain source symbols and initial repair symbols. The sending end receives feedback information sent by the receiving end based on a preset feedback channel and determines additional repair symbols based on the feedback information. The feedback information includes the decoding degree of freedom gap of each encoding block. The decoding degree of freedom gap is used to characterize the number of effective symbols that the encoding block needs to supplement to meet the decoding conditions. The sending end sends the source symbols, initial repair symbols, and additional repair symbols to the receiving end through a preset RDMA data channel. The preset RDMA data channel is a channel for data transmission built based on an RDMA network. The receiver receives encoded information from the transmitter through a preset RDMA data channel. The receiver counts the number of valid symbols received for each encoded block and determines the decoding degree of freedom gap for each encoded block based on the decoding threshold. The receiver generates feedback information based on the decoding degree of freedom gap for each encoded block and sends the feedback information to the transmitter through a preset RDMA feedback channel. The receiver receives additional repair symbols sent by the transmitter through the preset RDMA data channel and performs RaptorQ decoding to obtain the decoded encoded block. The decoded encoded block is then written to the target file according to the file offset position.

[0023] It is difficult to balance throughput, reliability and redundancy overhead in large-scale file transfers across computing centers.

[0024] This application provides a solution that establishes a closed-loop transmission control process at the application layer for file transfer between the sender and receiver. The sender dynamically determines the RaptorQ encoding parameters based on the file information to be transmitted, link status information, and the decoding degree-of-freedom gaps fed back by the receiver. The receiver calculates the decoding degree-of-freedom gaps for each encoded block based on the actual reception situation. The sender then generates additional repair symbols as needed based on the decoding degree-of-freedom gaps, thereby reducing reliance on network layer packet loss retransmission. This avoids the problem of balancing throughput, reliability, and redundancy overhead in large-scale file transfers across computing centers.

[0025] It should be noted that the execution subject of this embodiment can be a file transfer device based on an RDMA network. The file transfer device based on an RDMA network can be a device with an intelligent agent installed, such as a mobile terminal, personal computer, server or other electronic device used by the user. It can also be other devices that can achieve the same or similar functions. This embodiment does not limit this. In this embodiment and the following embodiments, the file transfer method based on an RDMA network is described using a file transfer device based on an RDMA network.

[0026] In this embodiment, the file transfer device based on the RDMA network can also be an electronic device that can control and detect devices equipped with intelligent agents. This embodiment does not limit this.

[0027] Based on this, embodiments of this application provide a file transfer method based on an RDMA network, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the file transfer method based on an RDMA network according to this application.

[0028] In this embodiment, the system architecture of this application includes a transmitter, a receiver, and a data channel and a feedback channel between them. The transmitter is deployed at the source computing power center, and the receiver is deployed at the target computing power center. The transmitter and receiver are connected via a high-speed cross-computing power center network supporting RDMA. For further illustration of the architecture of this application, refer to... Figure 2 The overall architecture diagram shown indicates that the sending end mainly includes the following functional units: a file task management unit 101, an encoding parameter decision unit 102, a RaptorQ encoding and redundancy generation unit 103, an RDMA transmission scheduling unit 104, and a feedback processing and repair scheduling unit 105. 1) The file task management unit 101 receives file transfer tasks across computing centers and generates file-level transmission metadata. The file task management unit first reads the basic information of the file to be transferred, and then passes the file size, file type, and target path to the encoding parameter decision unit 102. 2) The encoding parameter decision unit 102 is the control core of the sending end, used to determine the RaptorQ encoding parameters for each batch of encoding blocks. This unit receives file task information, link status information, and feedback transmission resource status information, thereby dynamically determining the encoding block size. Sign size Number of source symbols Initial redundancy rate Decoding safety margin And subsequent repair symbol generation strategies. Therefore, the determination of the coding block size and symbol size is not a fixed configuration, but dynamically changes with the link state and the receiver state. For example, when the link is stable, the packet loss rate is low, and the receiver buffer is sufficient, the coding parameter decision unit can appropriately increase the coding block size to reduce metadata overhead and improve continuous transmission efficiency. When the link packet loss rate increases, the receiver buffer occupancy is high, or the decoding failure rate increases, the coding parameter decision unit 102 can appropriately decrease the coding block size or symbol size to improve recovery granularity and decoding success rate. 3) The RaptorQ coding and redundancy generation unit 103 is used to perform RaptorQ coding on the file data to be transmitted according to the coding block size, symbol size, and number of source symbols output by the coding parameter decision unit 102. It mainly performs the following processing: reads file segments according to BlockSize, and divides each coding block into The unit generates source symbols and repair symbols according to the RaptorQ encoding rules, generates initial repair symbols according to the initial redundancy rate, and adds symbol header information to each symbol. When the transmitter subsequently receives decoding degree-of-freedom gaps from the receiver, this unit can continue to generate additional repair symbols for the specified coding block without retransmitting a specific lost source symbol. 4) The RDMA transmission scheduling unit 104 is used to send the source symbols and repair symbols output by the RaptorQ encoding and redundancy generation unit 103 to the receiver. This unit is responsible for managing the transmitter's RDMA registration memory, writing encoded symbols into the transmission buffer, maintaining the queue of symbols to be sent, submitting RDMA transmission requests according to scheduling priority, recording symbol transmission status, and maintaining the transmission progress of the coding block. 5) The feedback processing and repair scheduling unit 105 is used to receive and parse the decoding degree-of-freedom information fed back by the receiver, and decide whether additional repair symbols need to be generated accordingly. First, it is determined whether the corresponding coded block still has a decoding degree of freedom gap. If there is no decoding degree of freedom gap, no additional repair symbols are generated for that coded block. If there is a degree of freedom gap, the number of additional repair symbols is calculated. When multiple coded blocks need to be repaired simultaneously, the feedback processing and repair scheduling unit 105 determines the repair priority according to the method described below. Then, the feedback processing and repair scheduling unit 105 passes the repair decision to the RaptorQ coding and redundancy generation unit 103, which generates additional repair symbols, which are then sent by the RDMA transmission scheduling unit 104.

[0029] The receiving end mainly includes the following functional units: RDMA receiving and symbol management unit 201, coded block state management unit 202, decoding degree of freedom calculation and feedback unit 203, RaptorQ decoding unit 204, and file reconstruction and integrity verification unit 205. 1) The RDMA receiving and symbol management unit 201 manages the receiving end's RDMA registration memory and receive buffer, and receives coded symbols sent by the transmitting end through the RDMA data channel 301. It mainly performs the following processes: managing the receiving end's RDMA receive buffer, receiving source symbols and repair symbols, parsing symbol header information, verifying symbol integrity, and assigning valid symbols to their corresponding... and 1) Discard invalid symbols that fail verification or are duplicated. 2) The encoding block state management unit 202 maintains the reception and decoding states of each encoding block. Whenever a new valid encoding symbol is received, the encoding block state management unit 202 updates the state information and notifies the decoding degree of freedom calculation and feedback unit 203 to re-determine whether the encoding block has met the decoding conditions. 3) The decoding degree of freedom calculation and feedback unit 203 is the core of the feedback control at the receiving end. Based on the information maintained by the encoding block state management unit 202, this unit calculates the decoding threshold and decoding degree of freedom gap for each encoding block. When the encoding block meets the decoding conditions, the decoding degree of freedom calculation and feedback unit 203 updates the state of the encoding block to READY and notifies the RaptorQ decoding unit 204 to perform decoding. If the encoding block has not yet met the decoding conditions, and if the periodic or event-based feedback trigger condition is reached, the unit generates a feedback message and sends it to the sending end through the feedback channel. This feedback message does not require specifying the specific number of the lost symbol, but directly expresses how many valid encoding symbols the receiving end is still missing from completing the decoding. 4) The RaptorQ decoding unit 204 performs RaptorQ decoding on encoded blocks that meet the decoding conditions. If decoding is successful, it outputs the recovered original encoded block data and updates the status of the encoded block to SUCCESS. If decoding fails, it updates the status of the encoded block to FAILED or NEED_REPAIR and notifies the decoding degree-of-freedom calculation and feedback unit 203 to generate repair feedback, enabling the sender to continue sending additional repair symbols. 5) The file reconstruction and integrity verification unit 205 writes the successfully decoded encoded blocks into the target file according to the file offset position. This unit... , The unit retrieves the file offset information and writes the recovered original data to the target storage path. After all encoded blocks are successfully decoded, the unit calculates the file-level hash value of the received file and compares it with the FileHash provided by the sender. If the hash values ​​match, the file transfer is confirmed to be complete. If the hash values ​​do not match, the abnormal encoded block is located, and a re-verification or additional repair request is triggered. Furthermore, this invention employs a separate architecture for the RDMA data channel 301 and the RDMA feedback channel 302 between computing centers. The data channel is used to transmit source symbols, initial repair symbols, and additional repair symbols. The RDMA data channel 301 preferably uses RDMA Write or RDMASend to reduce CPU involvement and kernel protocol stack overhead, thereby improving the throughput of large file transfers across computing centers. The feedback channel is used by the receiver to provide feedback on the encoded block-level decoding status to the sender. The RDMA feedback channel 302 can be implemented using RDMA Send or other reliable control channels. By separating large-scale data transmission from small-scale status feedback, this invention retains the high throughput advantage of RDMA data transmission while enabling closed-loop control of RaptorQ decoding status at the application layer.

[0030] In this embodiment, the file transfer method based on the RDMA network is applied at the sending end, including steps S10~S40: Step S10: Dynamically determine the encoding parameters corresponding to the encoding block based on the information of the file to be transmitted, the current link status information, and the transmission resource status information fed back by the receiving end.

[0031] Understandably, the sending end obtains the file to be transferred and generates a file transfer task. Based on the file to be transferred and the file transfer task, it determines the information of the file to be transferred, including the file size, file path, target computing center address, and file hash value. That is, the sending end generates the task based on the file size, file path, target computing center address, and file hash value. .

[0032] It is understandable that the current link status information refers to the current cross-computing center link status information, including link bandwidth B, link round-trip time (RTT), and recent packet loss rate. Normalized link jitter and current effective throughput The transmission resource status information is information fed back by the receiving end, including available RDMA registered memory capacity, sender symbol processing capability, receiver buffer occupancy rate, receiver decoding capability, and receiver disk write capability. The current link status information includes at least one of bandwidth, round-trip time, recent link packet loss rate, normalized link jitter, and current effective throughput. The transmission resource status information includes at least one of sender RDMA send request processing capability, registered memory capacity, receiver buffer occupancy status, and receiver decoding capability. The encoding parameters include at least one of code block size, symbol size, number of source symbols, initial redundancy rate, and decoding safety margin.

[0033] It should be understood that the encoding parameters corresponding to the encoding block are dynamically determined based on the information of the file to be transmitted, the current link status information, and the transmission resource status information fed back by the receiving end. These encoding parameters include at least one of the following: encoding block size, symbol size, number of source symbols, initial redundancy rate, and decoding safety margin. A set of encoding parameters at the RaptorQ file encoding block level is dynamically determined, including at least the encoding block size, symbol size, number of source symbols, initial redundancy rate, decoding safety margin, and subsequent repair symbol generation strategy. These parameters are jointly determined by file task information, link status information, and receiving end resource status. This application dynamically determines the RaptorQ encoding block size and symbol size based on factors such as file size, bandwidth, RTT, packet loss rate, jitter, RDMA transmission capability, registered memory, receiving end buffer, and decoding capability. This addresses the comprehensive balance problem between throughput, redundancy overhead, decoding wait time, and receiving end resource pressure in large file transmission across computing centers.

[0034] In one implementation, the RaptorQ coding block size and symbol size are dynamically determined based on the file information to be transmitted, the current link status information, and the feedback transmission resource status. If the coding block size and symbol size are unreasonable, even if the subsequent redundancy strategy is adaptive, it will be difficult to achieve optimal throughput and reliability.

[0035] Further, step S10 also includes: calculating the bandwidth-delay product based on the link bandwidth and link round-trip delay in the current link state information; dynamically determining the coding block size based on the bandwidth-delay product, coding block adjustment coefficient, recent link packet loss rate in the current link state information, and state adjustment factor; dynamically determining the symbol size based on the bandwidth-delay product and the number of target symbols in transit; determining the initial redundancy rate based on the recent link packet loss rate, normalized link jitter, and minimum initial redundancy rate in the current link state information; and determining the coding parameters corresponding to the coding block based on the coding block size, the symbol size, and the initial redundancy rate.

[0036] It should be noted that determining the code block size needs to improve link throughput, reduce decoding latency, and control receiver memory and decoding pressure. When link bandwidth is high, RTT is large, and receiver resources are sufficient, the code block size should be appropriately increased to reduce the number of blocks and metadata overhead, thereby improving continuous transmission efficiency. When the link packet loss rate increases, the receiver decoding pressure is high, or the buffer usage is high, the code block size should be appropriately decreased to reduce the recovery cost of a single code block and decoding latency.

[0037] (1-1) in, The coding block adjustment coefficient and the state adjustment factor are used. It can be determined based on the link packet loss rate and the receiver load: (1-2) in, As a weight for packet loss rate, As the pressure weight of the receiver buffer, The receiver's buffer is occupied.

[0038] The sending end first calculates the bandwidth-delay product. Then, the code block size is determined using formulas (1-1) and (1-2). When the link is stable and the receiving end has sufficient resources, The loss rate is relatively high. When the packet loss rate increases or the receiver experiences increased pressure, Smaller.

[0039] Among them, sign size It is necessary to balance RDMA transmission efficiency and RaptorQ recovery granularity. When the symbol size is small, the recovery granularity is fine, but the number of transmission requests is large, resulting in high metadata processing overhead. When the symbol size is large, the transmission efficiency is high, but the impact of losing a single symbol is significant, and the repair granularity is coarser. This invention determines the symbol size based on the amount of data in transit, the processing capacity of the sending end, and the link risk.

[0040] (1-3) in, The number of symbols in transit for the target.

[0041] The sending end determines the symbol size of RaptorQ according to formula (1-3). If the link is stable and the RDMA send request processing pressure is high, the symbol size should be increased appropriately. If the packet loss rate or decoding failure rate increases, the symbol size should be decreased appropriately to improve the erasure coding recovery granularity.

[0042] The initial redundancy rate can be calculated based on the initial state of the link, which is determined by the recent link packet loss rate in the current link state information. Normalized link jitter Minimum initial redundancy rate Determine the initial redundancy rate .

[0043] Step S20: Perform RaptorQ encoding on the file information to be transmitted based on the encoding parameters to obtain the source symbol and the initial repair symbol.

[0044] It should be noted that, to further illustrate the dynamic determination mechanism of RaptorQ encoding parameters for link and task awareness in this application, refer to... Figure 3 The diagram illustrates a link- and task-aware dynamic determination mechanism for RaptorQ encoding parameters. Based on the block size and symbol size in the encoding parameters, RaptorQ encoding is performed on the file information to be transmitted, enabling adaptive adjustment of encoding granularity under different long-distance link conditions across computing centers. When the link is stable, the system tends to use larger block sizes and moderate symbol sizes to improve throughput. When link jitter or packet loss increases, the system tends to reduce the block size or symbol size to improve recovery flexibility and decoding success rate.

[0045] In one implementation, step S20 further includes: performing RaptorQ encoding on the file information to be transmitted based on the encoding block size and the symbol size to determine the source symbols; determining the number of initial repair symbols based on the initial redundancy rate and the number of source symbols; and generating initial repair symbols based on the number of initial repair symbols.

[0046] Understandably, determining the code block size and sign size Then, the sending end according to and The number of source symbols can be calculated using formula (1-4): Then the file is divided into multiple coded blocks, each containing... The number of source symbols K in each coded block is: (1-4) in, For the floor function, if the last coded block is less than 100 byte units... A complete source symbol can be padded, or the last coded block can be set to a smaller value. The encoding parameters of the RaptorQ algorithm described above can be adjusted periodically, such as by the transmitter detecting the link status and the receiver's feedback status at a preset period and then recalculating them. , and Alternatively, event-triggered adjustments can be set to trigger immediate adjustments to encoding parameters when any of the following events occur: link RTT changes exceeding a threshold, effective throughput decreases exceeding a threshold, recent packet loss rate increases exceeding a threshold, multiple encoding blocks fail to decode, receiver buffer occupancy exceeds a threshold, or receiver decoding queue length exceeds a threshold.

[0047] It should be understood that this application employs an initial redundancy strategy, whereby the sender will send an initial redundancy block when it first transmits a certain coded block. The source symbols and a certain number of initial repair symbols are used. The sender calculates the initial redundancy rate using formula (2-5) based on the recent link status. And the number of initial repair symbols is calculated using formula (2-4). The sending end then performs RaptorQ encoding on each coded block to generate... Source symbols and The initial number of repair symbols is calculated using the following formula: (2-4) in, It is a rounding function. This represents the initial redundancy rate.

[0048] It can be determined based on the initial state information of the link using the following calculation formula: (2-5) in, Minimum initial redundancy rate; Recent packet loss rate; Normalized link jitter; These are weighting coefficients. When the link is stable, Smaller bandwidth reduces redundant bandwidth overhead. This is especially beneficial when there is slight packet loss or increased jitter in the link. Increase the number of cells to improve the probability of successful decoding on the first attempt.

[0049] Step S30: Receive feedback information sent by the receiving end based on the preset feedback channel, and determine additional repair symbols based on the feedback information.

[0050] It should be noted that the feedback information includes the decoding degree of freedom gap for each coding block, which is used to characterize the number of valid symbols that the coding block needs to supplement in order to meet the decoding conditions.

[0051] In one implementation, step S30 further includes: determining recent link risks based on the preset basic protection redundancy rate, recent link packet loss rate, normalized link jitter, and recent decoding failure rate in the link state information; determining protection redundancy based on the recent link risks and the number of source symbols; and determining additional repair symbols based on the decoding degree-of-freedom gaps and protection redundancy in the feedback information sent by the receiver through a preset feedback channel. The transmitter first sends source symbols and initial repair symbols, the receiver provides feedback on the degree-of-freedom gaps for each coding block, and the transmitter then generates additional RaptorQ repair symbols for coding blocks with gaps, prioritizing the supplementation of coding blocks that are close to being decoded.

[0052] It should be noted that the receiving end periodically or based on event triggers feeds back to the sending end the decoding degree of freedom gap for each coded block. After receiving the feedback, the sending end... For coded blocks with a value greater than 0, additional repair symbols are generated. The formula for calculating the number of additional repair symbols is: (2-6) in, To protect redundancy.

[0053] This can be dynamically determined based on recent link risks: (2-7) It can be represented as: (2-8) in, Basic protection redundancy rate; Packet loss rate weight; : Jitter weight; Decoding failure rate weight; Recent decoding failure rate.

[0054] Therefore, the number of additional repair symbols is: (2-9) This approach makes the number of additional repair symbols subject to both the actual degree-of-freedom gap and the link risk.

[0055] Step S40: Send the source symbol, the initial repair symbol, and the additional repair symbol to the receiving end through a preset RDMA data channel.

[0056] It should be noted that the preset RDMA data channel is a data transmission channel built on an RDMA network. The transmitting end writes the source symbol and the initial repair symbol into the RDMA transmit buffer and transmits them to the receiving end through the preset RDMA data channel. Each symbol carries symbol header information including: the header information of each coded symbol includes at least a file identifier (…). ), task identifier ( ), Code block identifier ( ), symbol identification ( ), symbol type ( ), number of source symbols Sign size Sending mode ( ) and check value ( ).in, Used to distinguish between source symbols and repair symbols. Used to distinguish between the initial send and subsequent repair sends. Used for symbol-level integrity verification at the receiving end.

[0057] In one embodiment, step S40 further includes: when multiple coding blocks have simultaneous degree-of-freedom gaps, determining the transmission priority of the additional repair symbol based on the decoding degree-of-freedom gap, coding block waiting time, the position of the coding block in the file reassembly order, and the receiving end buffer pressure state; and transmitting the source symbol, the initial repair symbol, and the additional repair symbol to the receiving end based on the transmission priority and a preset RDMA data channel.

[0058] It should be noted that when multiple coded blocks have simultaneous degree-of-freedom gaps, the sender needs to determine the transmission priority of additional repair symbols. The formula for calculating the total priority value of repair symbol transmission is as follows: (2-10) in, These are the weight coefficients for the four corresponding dimensions. It is used to measure the current gap in decoding degrees of freedom ( Priority variables include waiting time, proximity of consecutive file reassembly locations, and receiver cache pressure.

[0059] (2-11) (2-12) (2-13) (2-14) in, To preset the maximum waiting time, The last encoded block that has been decoded and written to the file is numbered. This represents the current occupancy of the receiver's buffer. This represents the total buffer capacity at the receiving end. The meaning of the above formula is: The smaller the value, the closer the encoded block is to being decoded. The higher the value, the higher the overall priority; The longer the waiting time, the longer the encoded block has been waiting. The higher the value, the higher the overall priority; the closer the encoded block is to the current file's consecutive reorganization position, The higher the value, the higher the overall priority; the greater the buffer pressure at the receiving end, the more priority should be given to padding nearly complete encoded blocks. The higher the value, the higher the overall priority. This strategy encourages the receiver to complete the decoding of some encoded blocks and release the buffer as soon as possible, rather than leaving a large number of encoded blocks in an incomplete state for a long time.

[0060] In one embodiment, to further illustrate the adaptive repair symbol generation mechanism based on decoding degree-of-freedom feedback in this application, refer to... Figure 4 The diagram illustrates an adaptive repair symbol generation mechanism based on decoding degree-of-freedom feedback. Through this adaptive redundancy strategy, the invention can reduce redundant symbols and improve effective throughput when the link condition is good; increase repair symbols and improve decoding success rate when the link condition is poor; the receiver does not need to request specific lost packets, reducing the waiting time for precise retransmission under long RTT; the sender only generates additional repair symbols for coded blocks with degree-of-freedom gaps, avoiding global redundancy waste; and prioritizes repairing coded blocks that are close to being decoded, accelerating the overall file recovery progress.

[0061] This embodiment provides a file transfer method based on an RDMA network. The sending end dynamically determines the encoding parameters corresponding to the encoding block based on the file information to be transmitted, the current link status information, and the transmission resource status information fed back by the receiving end. The encoding parameters include the encoding block size, symbol size, and initial redundancy rate. The sending end performs RaptorQ encoding on the file information to be transmitted based on the encoding parameters to obtain source symbols and initial repair symbols. The sending end receives feedback information sent by the receiving end through a preset feedback channel and determines additional repair symbols based on the feedback information. The feedback information includes the decoding degree of freedom gap of each encoding block, which is used to characterize the number of effective symbols required to supplement the encoding block to meet the decoding conditions. The sending end sends the source symbols, initial repair symbols, and additional repair symbols to the receiving end through a preset RDMA data channel, which is a channel for data transmission built based on the RDMA network. The receiving end receives encoded information from the sending end through a preset RDMA data channel. The receiving end counts the number of valid symbols received for each encoded block and determines the decoding degree-of-freedom gap for each encoded block based on a decoding threshold. The receiving end generates feedback information based on the decoding degree-of-freedom gap for each encoded block and sends this feedback information to the sending end through a preset RDMA feedback channel. The receiving end receives additional repair symbols sent by the sending end through the preset RDMA data channel and performs RaptorQ decoding to obtain the decoded encoded block. The decoded encoded block is then written to the target file according to the file offset position. This application establishes a closed-loop transmission control process at the file transfer application layer between the sending and receiving ends. The sending end dynamically determines the RaptorQ encoding parameters based on the file information to be transmitted, link status information, and the decoding degree-of-freedom gap fed back by the receiving end. The receiving end calculates the decoding degree-of-freedom gap for each encoded block based on the actual reception situation, and the sending end generates additional repair symbols as needed based on the decoding degree-of-freedom gap, thereby reducing reliance on network layer packet loss retransmission. This avoids the problem of balancing throughput, reliability, and redundancy overhead in large-scale file transfers across computing centers.

[0062] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, the embodiments of this application provide a file transfer method based on an RDMA network, referring to... Figure 5 , Figure 5 This is a flowchart illustrating a second embodiment of the file transfer method based on an RDMA network according to this application. In this embodiment, the file transfer method based on an RDMA network is applied at the receiving end, and includes steps S50-S90: Step S50: Receive encoded information from the transmitter through a preset RDMA data channel.

[0063] It should be noted that the encoded information received from the transmitter is received through a preset RDMA data channel, and the encoded information includes source symbols and initial repair symbols.

[0064] Step S60: Count the number of valid encoded symbols received for each encoded block, and determine the decoding degree of freedom gap for each encoded block based on the decoding threshold.

[0065] It should be noted that the receiving end receives symbols, updates the coded block state, and calculates the decoding degree of freedom gap. For a coded block based on the RaptorQ algorithm, suppose a coded block contains... Each source symbol. The receiver does not necessarily need to receive all of them. Given a source symbol, the original data can be recovered as long as a sufficient number of valid encoded symbols are received, including the source symbol and the repair symbol.

[0066] Understandably, this application defines the number of valid encoded symbols missing from the decodeable state at the receiver as the "decoding degree of freedom gap". The receiver maintains a series of states for each encoded block, including: : File identifier; : Encoded block identifier; The number of source symbols; : Sign size; : The number of valid encoded symbols received; : Number of failed verification symbols; : Number of repair symbols received; Decoding threshold; Decoding the gap in degrees of freedom; Decoding status; Waiting time. Decoding status is also included. The system includes six states: RECEIVING, READY, DECODING, SUCCESS, FAILED, and NEED_REPAIR. In this application, the decoding degree-of-freedom gap is not the link packet loss rate or the number of simple packet losses, but rather a state quantity regarding whether a RaptorQ encoded block at the application layer file encoding block level is decodeable. The receiver calculates how many valid encoded symbols are still missing from the decoding threshold for each encoded block based on the number of source symbols, the decoding safety margin, and the number of received valid encoded symbols. The feedback message does not require specifying specific lost symbols, but rather expresses how many decoding degrees of freedom are still missing. The feedback in this application is used to determine whether a specific file encoded block has met the RaptorQ decoding conditions and directly decides whether to generate additional repair symbols for that encoded block.

[0067] In one embodiment, step S60 further includes: performing integrity verification on the symbols in the encoded information sent by the transmitting end to obtain a verification result; if the verification result is successful, updating the number of valid encoded symbols in the corresponding encoded block; calculating a decoding threshold based on the number of source symbols sent by the transmitting end and the decoding security margin; and determining the decoding degree of freedom gap for each encoded block based on the number of valid encoded symbols and the decoding threshold.

[0068] In one embodiment, calculating the decoding threshold based on the number of source symbols sent by the sending end and the decoding security margin includes: determining the decoding security margin based on the number of source symbols sent by the sending end, the recent link packet loss rate in the link state information, the recent code block decoding failure rate, and a preset minimum security margin; and determining the decoding threshold based on the number of source symbols and the decoding security margin.

[0069] Understandably, the receiving end calculates the decoding threshold for each coded block, as follows: (2-1) in, The number of source symbols; This is a safety margin for decoding.

[0070] The calculation method is as follows: The adjustment can be dynamically made based on recent link status and decoding results. (2-2) in, It is a rounding function. This represents the recent packet loss rate. This represents the recent encoding block decoding failure rate; is the weighting coefficient; c is the preset minimum safety margin.

[0071] The receiver can calculate the decoding degree of freedom gap using the following formula: (2-3) in, This represents the number of valid encoded symbols that have been received and passed verification. When this condition is met, it indicates that the encoded block has met the decoding conditions, and the receiving end can enter the decoding process. If the condition is not met, it means that the encoded block has not yet met the decoding conditions and additional encoded symbols need to be received.

[0072] In one implementation, after receiving a symbol, the receiving end parses the symbol header and performs symbol verification. If the verification is successful, the number of valid received symbols for the corresponding coded block is updated. The receiving end calculates the decoding degree of freedom gap to determine whether the decoding conditions are met, and the receiving end calculates the decoding safety margin using formula (2-2). Then, the decoding threshold is calculated using formula (2-1). Then, the decoding degree of freedom gap is further calculated using equation (2-3). .like If so, the receiving end puts the encoded block into the decoding queue and performs RaptorQ decoding. If so, the receiving end continues to wait for subsequent symbols.

[0073] Step S70: Generate feedback information based on the decoding degree of freedom gap of each coding block, and send the feedback information to the transmitting end through a preset RDMA feedback channel.

[0074] In one implementation, the preset RDMA feedback channel is a separate channel from the preset RDMA data channel. The receiver reports a degree-of-freedom gap to the transmitter when the periodic feedback trigger time arrives or when an immediate feedback trigger event occurs (including: the waiting time for a certain coded block exceeds a threshold, the receiver's buffer occupancy exceeds a threshold, the current batch of coded blocks has finished receiving, decoding has failed, or the DoF_gap has not returned to zero for an extended period). The receiver's feedback information may include: : File identifier; : Encoded block identifier; The number of source symbols; : Number of valid symbols received; Decoding threshold; Decoding the gap in degrees of freedom; Decoding status; It is recommended to further reduce the number of symbols. Receiver buffer occupancy rate; : Decoding queue length. This feedback message does not need to list all lost symbol numbers. The sender only needs to continue generating new RaptorQ repair symbols and send the feedback information to the sender via a preset RDMA data channel.

[0075] Step S80: Receive the additional repair symbols sent by the transmitter through the preset RDMA data channel, and perform RaptorQ decoding to obtain the decoded coded block.

[0076] It should be noted that after the sending end receives the feedback, it uses formula (2-9) to... Additional repair symbols are generated for each coded block. When multiple coded blocks require repair, the sender selects the repair order based on priority. The total priority value for sending repair symbols can be calculated using formula (2-10), with higher values ​​sent first. After receiving enough valid coded symbols, the receiver performs RaptorQ decoding to obtain the decoded coded blocks.

[0077] Step S90: Write the decoded encoded block into the target file according to the file offset position.

[0078] It should be noted that after successful decoding, the encoded blocks are verified. If the verification passes, the recovered original data is written to the corresponding offset position in the target file. After all encoded blocks are completed, the receiving end calculates the file-level hash value and compares it with the FileHash provided by the sending end. If they match, the file transfer is confirmed to be complete.

[0079] In one implementation, traditional packet loss feedback is unsuitable for RaptorQ because RaptorQ does not require a specific lost symbol, but only a sufficient number of valid encoded symbols; therefore, a feedback mechanism centered on the decoding degree-of-freedom gap is needed. This application constructs a closed-loop mechanism based on the RaptorQ fountain code characteristics, including initial redundancy, decoding degree-of-freedom feedback redundancy, and additional symbol scheduling for repair. To further illustrate the complete RDMA file transfer process between the sender and receiver across computing centers in this application, refer to... Figure 6 The diagram shown illustrates the complete RDMA file transfer process, comprising 14 core steps: Step 1: Establish an RDMA transfer task; Step 2: Establish RDMA data and feedback channels; Step 3: Sensing link status and transmission resources; Step 4: Dynamically determining the coded block size; Step 5: Dynamically determining the symbol size; Step 6: Determining the number of source symbols; Step 7: Generating initial repair symbols; Step 8: Sending source symbols and initial repair symbols via RDMA; Step 9: The receiving end receives symbols and updates the coded block status; Step 10: The receiving end calculates the decoding degree-of-freedom gap to determine if the decoding conditions are met; Step 11: The receiving end feeds back decoding degree-of-freedom information; Step 12: The sending end generates additional repair symbols; Step 13: The sending end schedules additional repair symbols; Step 14: The receiving end completes decoding and file reassembly. This application addresses the scenarios of large files, high bandwidth, and long distances across computing centers. It innovatively proposes a high-speed RDMA file transfer method based on adaptive erasure coding. RaptorQ erasure coding is applied to RDMA file transfer, and a link- and task-aware dynamic determination mechanism for RaptorQ coding parameters and an adaptive repair symbol generation mechanism based on decoding degree of freedom feedback are integrated. This method can ensure transmission throughput, reliability, and resource utilization.

[0080] This embodiment provides a file transfer method based on an RDMA network. The sending end dynamically determines the encoding parameters corresponding to the encoding block based on the file information to be transmitted, the current link status information, and the transmission resource status information fed back by the receiving end. The encoding parameters include the encoding block size, symbol size, and initial redundancy rate. The sending end performs RaptorQ encoding on the file information to be transmitted based on the encoding parameters to obtain source symbols and initial repair symbols. The sending end receives feedback information sent by the receiving end through a preset feedback channel and determines additional repair symbols based on the feedback information. The feedback information includes the decoding degree of freedom gap of each encoding block, which is used to characterize the number of effective symbols required to supplement the encoding block to meet the decoding conditions. The sending end sends the source symbols, initial repair symbols, and additional repair symbols to the receiving end through a preset RDMA data channel, which is a channel for data transmission built based on the RDMA network. The receiving end receives encoded information from the sending end through a preset RDMA data channel. The receiving end counts the number of valid symbols received for each encoded block and determines the decoding degree-of-freedom gap for each encoded block based on a decoding threshold. The receiving end generates feedback information based on the decoding degree-of-freedom gap for each encoded block and sends this feedback information to the sending end through a preset RDMA feedback channel. The receiving end receives additional repair symbols sent by the sending end through the preset RDMA data channel and performs RaptorQ decoding to obtain the decoded encoded block. The decoded encoded block is then written to the target file according to the file offset position. This application establishes a closed-loop transmission control process at the file transfer application layer between the sending and receiving ends. The sending end dynamically determines the RaptorQ encoding parameters based on the file information to be transmitted, link status information, and the decoding degree-of-freedom gap fed back by the receiving end. The receiving end calculates the decoding degree-of-freedom gap for each encoded block based on the actual reception situation, and the sending end generates additional repair symbols as needed based on the decoding degree-of-freedom gap, thereby reducing reliance on network layer packet loss retransmission. This avoids the problem of balancing throughput, reliability, and redundancy overhead in large-scale file transfers across computing centers.

[0081] This application also provides a file transfer device based on an RDMA network; please refer to [reference needed]. Figure 7 , Figure 7 This is a schematic diagram of the module structure of a file transfer device based on an RDMA network, as provided in Embodiment 1. The file transfer device based on an RDMA network includes: The encoding parameter decision module 10 is used to dynamically determine the encoding parameters corresponding to the encoding block based on the file information to be transmitted, the current link status information, and the transmission resource status information fed back by the receiving end. The current link status information includes at least one of bandwidth, round-trip time, recent link packet loss rate, normalized link jitter, and current effective throughput. The transmission resource status information includes at least one of the following: the sending end's RDMA transmission request processing capability, registered memory capacity, receiving end's buffer occupancy status, and receiving end's decoding capability. The encoding parameters include at least one of the following: encoding block size, symbol size, number of source symbols, initial redundancy rate, and decoding security margin. RaptorQ encoding and redundancy generation module 20 is used to perform RaptorQ encoding on the file information to be transmitted based on the encoding parameters to obtain source symbols and initial repair symbols. The feedback processing and repair scheduling module 30 is used to receive feedback information sent by the receiving end based on a preset feedback channel, and to determine additional repair symbols based on the feedback information. The feedback information includes the decoding degree of freedom gap of each coding block. The decoding degree of freedom gap is used to characterize the number of effective symbols that the coding block needs to supplement in order to meet the decoding conditions. The RDMA transmission scheduling module 40 is used to send the source symbol, the initial repair symbol, and the additional repair symbol to the receiving end through a preset RDMA data channel. The preset RDMA data channel is a channel for data transmission built based on the RDMA network.

[0082] This application also provides a file transfer device based on an RDMA network; please refer to [reference needed]. Figure 8 , Figure 8 This is a schematic diagram of the module structure of a file transfer device based on an RDMA network, as provided in Embodiment 2. The file transfer device based on an RDMA network includes: The RDMA receiving and symbol management module 50 is used to receive encoded information from the transmitting end through a preset RDMA data channel. The encoded information includes source symbols and initial repair symbols. The coding block state management module 60 is used to count the number of valid coding symbols received for each coding block and determine the decoding degree of freedom gap for each coding block based on the decoding threshold. The decoding degree of freedom calculation and feedback module 70 is used to generate feedback information based on the decoding degree of freedom gap of each coding block, and send the feedback information to the transmitting end through a preset RDMA feedback channel. The preset RDMA feedback channel is a channel separate from the preset RDMA data channel. RaptorQ decoding module 80 is used to receive additional repair symbols sent by the transmitter through a preset RDMA data channel, and perform RaptorQ decoding to obtain the decoded coded block; The file reconstruction and integrity verification module 90 is used to write the decoded encoded block into the target file according to the file offset position.

[0083] The file transfer device based on an RDMA network provided in this application, employing the file transfer method based on an RDMA network in the above embodiments, can solve the technical problem of balancing throughput, reliability, and redundancy overhead in large-scale file transfers across computing centers. Compared with the prior art, the beneficial effects of the file transfer device based on an RDMA network provided in this application are the same as those of the file transfer method based on an RDMA network provided in the above embodiments, and other technical features in the file transfer device based on an RDMA network are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0084] This application provides a file transfer device based on an RDMA network. The file transfer device based on an RDMA network includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the file transfer method based on an RDMA network in the first embodiment described above.

[0085] The following is for reference. Figure 9 This document illustrates a structural diagram of a file transfer device based on an RDMA network suitable for implementing embodiments of this application. The file transfer device based on an RDMA network in these embodiments may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 9 The file transfer device based on the RDMA network shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0086] like Figure 9As shown, a file transfer device based on an RDMA network may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the file transfer device based on the RDMA network. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows RDMA-based file transfer devices to communicate wirelessly or wiredly with other devices to exchange data. While various systems are shown in the figure, it should be understood that implementation or possession of all shown systems is not required. More or fewer systems may be implemented alternatively.

[0087] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0088] The file transfer device based on an RDMA network provided in this application, employing the file transfer method based on an RDMA network in the above embodiments, can solve the technical problem of balancing throughput, reliability, and redundancy overhead in large-scale file transfers across computing centers. Compared with the prior art, the beneficial effects of the file transfer device based on an RDMA network provided in this application are the same as those of the file transfer method based on an RDMA network provided in the above embodiments, and other technical features of this file transfer device based on an RDMA network are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0089] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0090] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0091] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the file transfer method based on an RDMA network in the above embodiments.

[0092] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0093] The aforementioned computer-readable storage medium may be included in an RDMA-based file transfer device; or it may exist independently and not incorporated into an RDMA-based file transfer device.

[0094] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by a file transfer device based on an RDMA network, the RDMA-based file transfer device: determines whether to enable catalytic converter heating based on historical catalytic converter heating information and the target catalytic converter model temperature; when it is determined that catalytic converter heating should be enabled, it determines the target stage catalytic converter heating strategy from the multi-stage catalytic converter heating strategies based on the target catalytic converter model temperature; it optimizes the heating parameters corresponding to the target stage catalytic converter heating strategy based on historical catalytic converter heating information, and controls the heating of the catalytic converter based on the optimized heating parameters.

[0095] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Python, Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0096] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0097] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0098] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described file transfer method based on an RDMA network. This solves the technical problem of balancing throughput, reliability, and redundancy overhead in large-scale file transfers across computing centers. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the file transfer method based on an RDMA network provided in the above embodiments, and will not be repeated here.

[0099] All user-related data involved in this application (such as user privacy data, user behavior data, etc.) were obtained with the user's permission or consent; that is to say, when this application is used in a specific product or technology, user permission is required to obtain and process the relevant data, and the processing of the relevant data must comply with the relevant laws, regulations and regulatory standards of the relevant countries and regions.

[0100] The above description is only a part of the embodiments of this application and does not limit the scope of protection of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included in the scope of protection of this application.

Claims

1. A file transfer method based on an RDMA network, applied at the sending end, characterized in that, The file transfer method based on RDMA network includes: Calculate the bandwidth-delay product based on the link bandwidth and link round-trip time in the current link status information; The coding block size is dynamically determined based on the bandwidth-delay product, the coding block adjustment coefficient, the recent link packet loss rate in the current link state information, and the state adjustment factor. The symbol size is dynamically determined based on the bandwidth-delay product and the number of target symbols in transit. The initial redundancy rate is determined based on the recent link packet loss rate, normalized link jitter, and minimum initial redundancy rate in the current link status information. The coding parameters corresponding to the coding block are determined based on the coding block size, the symbol size, and the initial redundancy rate. The current link state information includes at least one of bandwidth, round-trip time, recent link packet loss rate, normalized link jitter, and current effective throughput. The coding parameters include at least one of coding block size, symbol size, number of source symbols, initial redundancy rate, and decoding security margin. Based on the block size and the symbol size, perform RaptorQ encoding on the file information to be transmitted to determine the source symbol; The number of initial repair symbols is determined based on the initial redundancy rate and the number of source symbols, and initial repair symbols are generated based on the number of initial repair symbols. The recent link risk is determined based on the preset basic protection redundancy rate, recent link packet loss rate, normalized link jitter, and recent decoding failure rate in the current link status information. The amount of protection redundancy is determined based on the recent link risks and the number of source symbols. Additional repair symbols are determined based on the decoding degree of freedom gap and the protection redundancy amount in the feedback information sent by the receiving end through the preset feedback channel. The feedback information includes the decoding degree of freedom gap of each coding block. The decoding degree of freedom gap is used to characterize the number of effective symbols that the coding block needs to supplement in order to meet the decoding conditions. The source symbol, the initial repair symbol, and the additional repair symbol are sent to the receiving end through a preset RDMA data channel, wherein the preset RDMA data channel is a channel for data transmission built based on an RDMA network.

2. The method as described in claim 1, characterized in that, The step of sending the source symbol, the initial repair symbol, and the additional repair symbol to the receiving end through a preset RDMA data channel includes: When multiple coding blocks have simultaneous degree-of-freedom gaps, the sending priority of the additional repair symbols is determined based on the decoding degree-of-freedom gap, the coding block waiting time, the position of the coding block in the file reassembly order, and the receiving end buffer pressure state. Based on the transmission priority and the preset RDMA data channel, the source symbol, the initial repair symbol, and the additional repair symbol are sent to the receiving end.

3. A file transfer method based on an RDMA network, applied at the receiving end, characterized in that, The file transfer method based on RDMA network includes: Receive encoded information from the transmitter through a preset RDMA data channel; Perform integrity verification on the symbols in the encoded information sent by the sending end, and obtain the verification result; If the verification result is successful, then update the number of valid coded symbols in the corresponding coded block; The decoding safety margin is determined based on the number of source symbols sent by the sending end, the recent link packet loss rate in the link status information, the recent code block decoding failure rate, and the preset minimum safety margin. The decoding threshold is determined based on the number of source symbols and the decoding security margin. The decoding degree of freedom gap for each coded block is determined based on the number of effective coded symbols and the decoding threshold; Feedback information is generated based on the decoding degree of freedom gap of each coding block, and the feedback information is sent to the transmitting end through a preset RDMA feedback channel, wherein the preset RDMA feedback channel is a channel separate from the preset RDMA data channel; The additional repair symbols sent by the transmitting end are received through a preset RDMA data channel, and RaptorQ decoding is performed to obtain the decoded coded block; The decoded encoded block is written to the target file according to the file offset position.

4. A file transfer device based on an RDMA network, applied at the sending end, characterized in that, The file transfer device based on the RDMA network includes: The encoding parameter decision module is used to dynamically determine the encoding parameters corresponding to the encoding block based on the information of the file to be transmitted, the current link status information, and the transmission resource status information fed back by the receiving end. The current link status information includes at least one of bandwidth, round-trip time, recent link packet loss rate, normalized link jitter, and current effective throughput. The encoding parameters include at least one of encoding block size, symbol size, number of source symbols, initial redundancy rate, and decoding security margin. The RaptorQ encoding and redundancy generation module is used to perform RaptorQ encoding on the file information to be transmitted based on the encoding parameters to obtain source symbols and initial repair symbols. The feedback processing and repair scheduling module is used to receive feedback information sent by the receiving end based on a preset feedback channel, and to determine additional repair symbols based on the feedback information. The feedback information includes the decoding degree of freedom gap of each coding block. The decoding degree of freedom gap is used to characterize the number of effective symbols that the coding block needs to supplement in order to meet the decoding conditions. The RDMA transmission scheduling module is used to send the source symbol, the initial repair symbol, and the additional repair symbol to the receiving end through a preset RDMA data channel. The preset RDMA data channel is a channel for data transmission built based on the RDMA network. The dynamic determination of encoding parameters corresponding to the encoding block based on the file information to be transmitted, the current link status information, and the transmission resource status information fed back by the receiving end includes: Calculate the bandwidth-delay product based on the link bandwidth and link round-trip time in the current link status information; The coding block size is dynamically determined based on the bandwidth-delay product, the coding block adjustment coefficient, the recent link packet loss rate in the current link state information, and the state adjustment factor. The symbol size is dynamically determined based on the bandwidth-delay product and the number of target symbols in transit. The initial redundancy rate is determined based on the recent link packet loss rate, normalized link jitter, and minimum initial redundancy rate in the current link status information. The coding parameters corresponding to the coding block are determined based on the coding block size, the symbol size, and the initial redundancy rate. The step of performing RaptorQ encoding on the file information to be transmitted based on the encoding parameters to obtain source symbols and initial repair symbols includes: The file information to be transmitted is RaptorQ encoded based on the block size and the symbol size to determine the source symbol. The number of initial repair symbols is determined based on the initial redundancy rate and the number of source symbols, and initial repair symbols are generated based on the number of initial repair symbols. The step of receiving feedback information sent by the receiving end based on a preset feedback channel and determining additional repair symbols based on the feedback information includes: The recent link risk is determined based on the preset basic protection redundancy rate, recent link packet loss rate, normalized link jitter, and recent decoding failure rate in the current link status information. The amount of protection redundancy is determined based on the recent link risks and the number of source symbols. Additional repair symbols are determined based on the decoding degree-of-freedom gap and the protection redundancy amount received from the feedback information sent by the receiving end through the preset feedback channel.

5. A file transfer device based on an RDMA network, applied at a receiving end, characterized in that, The file transfer device based on the RDMA network includes: The RDMA receiver and symbol management module is used to receive encoded information from the transmitter through a preset RDMA data channel; The coding block state management module is used to count the number of valid coded symbols received for each coding block and determine the decoding degree of freedom gap for each coding block based on the decoding threshold. The decoding degree of freedom calculation and feedback module is used to generate feedback information based on the decoding degree of freedom gap of each coding block, and send the feedback information to the transmitting end through a preset RDMA feedback channel. The preset RDMA feedback channel is a channel separate from the preset RDMA data channel. The RaptorQ decoding module is used to receive additional repair symbols sent by the transmitter through a preset RDMA data channel, and perform RaptorQ decoding to obtain the decoded encoded block; The file reconstruction and integrity verification module is used to write the decoded encoded block into the target file according to the file offset position; The process of counting the number of valid encoded symbols received for each encoded block and determining the decoding degree of freedom gap for each encoded block based on a decoding threshold includes: Perform integrity verification on the symbols in the encoded information sent by the sending end, and obtain the verification result; If the verification result is successful, then update the number of valid coded symbols in the corresponding coded block; The decoding safety margin is determined based on the number of source symbols sent by the sending end, the recent link packet loss rate in the link status information, the recent code block decoding failure rate, and the preset minimum safety margin. The decoding threshold is determined based on the number of source symbols and the decoding security margin. The decoding degree of freedom gap for each coded block is determined based on the number of effective coded symbols and the decoding threshold.

Images