A data transmission method, device and medium based on a network card system

Abstract

The embodiment of the specification discloses a data transmission method and device based on a network card system and a medium, comprising: when a first user sends transmission data, the network card system obtains a transmission command issued by the first user, and analyzes address information of the data storage according to the transmission command; an encoding decision submodule determines encoding behavior information of the transmission data; an encoding execution submodule encodes the transmission data in the encoding cache area; a transmission logic puts an encoding tag carrying the encoding behavior information in a message header; when a second user receives the transmission data, the unencoded data packet that passes the detection and the encoded data packet that passes the detection are obtained, the unencoded data packet that passes the detection is stored in a corresponding position of the memory, and the encoded data packet that passes the detection is sent to a decoding module; the decoding module decodes the received encoded data packet that passes the detection, obtains decoded data, and stores the decoded data in a corresponding position of the memory.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a data transmission method, device and medium based on a network interface card (NIC) system. Background Technology

[0002] With the continuous development of network infrastructure and forwarding equipment in data centers, the link transmission speed of data center networks has now reached 100Gbps and is trending towards 400Gbps. This has spurred the running of various online data-intensive applications, distributed computing services, and machine learning training tasks on data center infrastructures. These throughput-sensitive and latency-sensitive applications require low-latency, high-throughput networks and high CPU utilization servers for support. Therefore, most existing solutions utilize RoCEv2.

[0003] Traditional RoCEv2 is highly sensitive to packet loss and out-of-order delivery because the hardware-implemented transmission protocol on the network interface card (NIC) struggles to handle complex packet loss logic. The improved RoCE NIC (IRN) implements a new design, replacing the traditional Go-Back-N-Frames (GBN) protocol for out-of-order packets with a Selective Repeat (SR) protocol, thus achieving a more efficient loss recovery strategy. Figure 1 The diagram shown illustrates the RDMA Selective Repeat protocol process. In simple terms, when the receiver receives out-of-order data packets, it is considered that packet loss has occurred, and a retransmission notification will be sent to the receiver. After receiving the retransmission notification, the receiver retransmits the lost data packets. However, the retransmission behavior always has a lag, which may still cause excessive delay. Summary of the Invention

[0004] This specification provides one or more embodiments of a data transmission method, device, and medium based on a network interface card (NIC) system to solve the technical problems mentioned in the background art.

[0005] One or more embodiments of this specification employ the following technical solutions:

[0006] This specification provides one or more embodiments of a data transmission method based on a network interface card (NIC) system. The NIC system includes an encoding decision submodule, an encoding execution submodule, and a transmission logic and decoding module. The method includes:

[0007] When the first user sends data, the network card system obtains the transmission command issued by the first user, parses the address information where the data is stored according to the transmission command, and transmits the address information to the encoding decision submodule.

[0008] The encoding decision submodule obtains a data amount of the transmission data according to the address information, determines encoding behavior information of the transmission data if it is determined that the transmission data needs to be encoded based on the data amount of the transmission data, and calls a DMA controller to take the transmission data to an encoding cache area of the encoding execution submodule and respectively transmit the encoding behavior information to the encoding execution submodule and the transmission logic;

[0009] The encoding execution submodule encodes the transmission data in the encoding cache area according to the encoding behavior information when the encoding behavior information is received, and transmits the encoded data to a sending cache area of the transmission logic;

[0010] The transmission logic distinguishes and encapsulates the uncoded incoming data and the encoded incoming data to obtain data packets, the data packets include uncoded data packets and encoded data packets, and for the encoded data packets, an encoding label carrying the encoding behavior information is marked in a packet header;

[0011] When the second user receives the transmission data, the transmission logic performs anomaly detection on the received data packets, detects whether the encoding label is carried in the packet header of the data packets, obtains uncoded data packets that pass the detection and encoded data packets that pass the detection, and stores the uncoded data packets that pass the detection in a corresponding position of a memory and transmits the encoded data packets that pass the detection to a decoding module;

[0012] The decoding module decodes the received encoded data packets that pass the detection to obtain decoded data and stores the decoded data in a corresponding position of a memory.

[0013] It should be noted that the embodiments of the present specification have the following beneficial effects through the above content:

[0014] Improve data transmission reliability: traditional RoCEv2 is sensitive to packet loss and out-of-order, while the present method applies network coding for data transmission. When packet loss occurs in the network, the receiving end can recover the lost data packets through decoding operation, reducing data loss and transmission errors caused by packet loss, and improving the reliability of data transmission.

[0015] Reduce the delay caused by retransmission: although the improved RoCE network card adopts the selective repeat protocol (SR protocol), the retransmission behavior still has a lag, which may cause high delay. Through network coding, retransmission of lost data packets is avoided to some extent, thereby reducing transmission delay and improving real-time performance of data transmission.

[0016] Enhanced network adaptability: Network coding enables data transmission to have stronger adaptability in the face of packet loss. Even if the network environment is unstable and packet loss occurs, data can be recovered through decoding to ensure data integrity and accuracy.

[0017] Optimized network resource utilization: Reduces network resource waste caused by retransmission, improves network bandwidth utilization, and makes network resources more effectively utilized.

[0018] Improved overall system performance: The collaborative work of the encoding decision sub-module, encoding execution sub-module, transmission logic and decoding module, and the application of network coding improve the data processing efficiency and performance of the entire system, making data transmission more efficient and stable.

[0019] Further, the encoding decision sub-module obtains the data amount of the transmission data according to the address information, comprising:

[0020] The encoding decision sub-module obtains the address quantity according to the start address and the end address;

[0021] According to the data amount of each address and the address quantity, the data amount of the transmission data is obtained.

[0022] It should be noted that the embodiments of the present application have the following beneficial effects through the above content:

[0023] Accurate calculation of data amount: By obtaining the address quantity according to the start address and the end address, and combining the data amount of each address, the data amount of the transmission data can be accurately calculated. This helps to more accurately assess the data transmission requirements and resource allocation, and improve the efficiency and accuracy of data transmission.

[0024] Optimized resource allocation: After accurately knowing the data amount of the transmission data, system resources such as memory, bandwidth, etc. can be more reasonably allocated, avoiding resource waste or deficiency, and improving the overall performance and resource utilization of the system.

[0025] Supporting encoding decision: The encoding decision sub-module is provided with key information. Based on accurate data amount calculation, the encoding decision sub-module can more accurately determine whether the data needs to be encoded and determine the appropriate encoding behavior information, thereby improving the reliability and efficiency of data transmission.

[0026] Further, if it is determined based on the data amount of the transmission data that the transmission data needs to be encoded, the method further comprises:

[0027] Obtaining the current network card sending rate of the network card system;

[0028] Determine the round-trip time RTT of the current network and the packet loss rate of the current network according to the current received data packet.

[0029] determine whether the data volume of the transmission data needs network coding based on the data volume to be transmitted, the current network card transmission rate, the RTT of the current network, and the packet loss rate of the current network.

[0030] It should be noted that the embodiments of the present specification have the following beneficial effects through the above content:

[0031] Improving transmission efficiency: By obtaining the current network card transmission rate of the network card system, determining the round-trip time (RTT) and packet loss rate of the current network, and combining the data volume to be transmitted to determine whether network coding is needed. This can make optimization decisions based on actual network conditions and data volume, avoid unnecessary coding operations, and thus improve the efficiency of data transmission.

[0032] Optimizing network resource utilization: Considering factors such as current network card transmission rate, RTT, and packet loss rate, network resources can be more reasonably allocated. For good network conditions and low packet loss rate, the use of coding can be reduced to save computing resources; for poor network conditions and high packet loss rate, the reliability of data transmission can be improved by determining whether coding is needed, and network resources can be fully utilized.

[0033] Improving the reliability of data transmission: According to the RTT and packet loss rate of the current network, it is determined whether network coding is needed, which can handle packet loss in the network accordingly. If the packet loss rate is high, coding can increase the redundancy of data and improve the reliability of data transmission, reducing the impact of data loss on transmission.

[0034] Adapting to different network environments: This method can be adjusted according to real-time network conditions. Different network environments may have different network card transmission rates, RTTs, and packet loss rates. Through this dynamic judgment method, transmission data can better adapt to various network conditions, improving the stability and adaptability of data transmission.

[0035] Reducing transmission costs: By accurately determining whether network coding is needed, additional costs such as computing costs and time costs caused by excessive use of coding can be avoided. At the same time, reasonable coding decisions can also reduce network bandwidth consumption caused by data retransmission, thereby reducing transmission costs.

[0036] Further, the determination of whether the data volume of the transmission data needs network coding based on the data volume to be transmitted, the current network card transmission rate, the RTT of the current network, and the packet loss rate of the current network includes:

[0037] Setting a first threshold value and a second threshold value for the RTT and packet loss rate of the network, respectively;

[0038] Setting a third threshold value for the data transmission time;

[0039] determining a current data transmission time according to the current network card sending rate and the data volume of the transmission data;

[0040] judging whether the RTT of the current network meets the first threshold value, and / or judging whether the packet loss rate of the current network meets the second threshold value, and / or judging whether the current data transmission time meets the third threshold value, based on the data volume of the to-be-sent data.

[0041] It should be noted that the embodiments of the present specification have the following beneficial effects through the above content:

[0042] Optimizing network performance: by setting the thresholds of RTT, packet loss rate and data transmission time, and making judgments according to the current situation, problems in the network can be found in time, and corresponding measures can be taken to optimize the network performance and improve the efficiency and quality of data transmission.

[0043] Adapting to different data volume requirements: the thresholds change in steps with the change of the data volume of the to-be-sent data, which can better adapt to different data transmission requirements. For smaller data volume, the threshold requirements can be appropriately relaxed to avoid excessive resource consumption; for larger data volume, the strictness of the threshold can be increased to ensure the reliability and timeliness of data transmission.

[0044] Improving the reliability of data transmission: by monitoring and judging the RTT and packet loss rate, unstable factors in the network can be found in time, and measures can be taken to repair or adjust, thereby reducing the risk of data loss and improving the reliability of data transmission.

[0045] Reasonably allocating resources: determining the current data transmission time according to the current network card sending rate and the data volume of the transmission data, and comparing it with the set third threshold value, can help reasonably allocate network resources and avoid resource waste or shortage.

[0046] Enhancing the adaptability of the network: the scheme can dynamically adjust according to the actual situation of the network, so that the network can better adapt to different environments and requirements, and improve the overall adaptability and flexibility of the network.

[0047] Further, if it is determined based on the data volume of the transmission data that the transmission data does not need to be encoded, the method further comprises:

[0048] According to the address information, the transmission data is taken to the sending cache area of the transmission logic.

[0049] It should be noted that the embodiments of the present specification have the following beneficial effects through the above content:

[0050] Improving transmission efficiency: when it is determined that the transmission data does not need to be encoded, it is directly taken to the sending buffer area of the transmission logic according to the address information, reducing the time consumption caused by unnecessary encoding operations, thereby improving the overall efficiency of data transmission.

[0051] Saving system resources: avoids occupying system resources such as computing resources and memory resources during the encoding process, so that these resources can be better allocated to other tasks, improving the utilization rate of system resources.

[0052] Further, the encoding behavior information includes the data volume of the encoding batch and the redundant data volume generated by encoding, and the encoded data includes redundant data and initial data.

[0053] It should be noted that the embodiments of the present application have the following beneficial effects through the above content:

[0054] Precise evaluation of encoding effect: by recording the data volume of the encoding batch and the redundant data volume generated by encoding, the effect of the encoding behavior can be accurately evaluated. This helps to understand the changes of data in the encoding process and provides a basis for further optimizing the encoding strategy.

[0055] Improving data reliability: the encoded data includes redundant data and initial data, and the existence of redundant data can be used to recover lost data when packet loss occurs during data transmission, thereby improving the reliability of data transmission.

[0056] Optimizing network resource utilization: understanding the data volume of the encoding batch and the generated redundant data volume can help reasonably adjust the encoding parameters to minimize the generation of redundant data while ensuring data reliability, thereby optimizing the utilization of network resources and avoiding unnecessary bandwidth waste.

[0057] Facilitating error detection and recovery: the encoded data contains redundant data, and when the receiver receives the data, it can detect and recover possible errors by checking and processing the redundant data, improving the accuracy of the data.

[0058] Further, the transmission logic performs anomaly detection on the received data packet and detects whether the packet header of the data packet carries an encoding label, obtaining the unencoded data packet that passes the detection and the encoded data packet that passes the detection, including:

[0059] The transmission logic detects whether the packet header of the data packet carries an encoding label, obtaining the unencoded data packet and the encoded data packet;

[0060] The transmission logic performs packet loss detection and error detection on the uncoded data packet and the coded data packet, and obtains a detected passed uncoded data packet and a detected passed coded data packet.

[0061] It should be noted that the embodiments of the present specification have the following beneficial effects through the above content:

[0062] Improve data accuracy: by performing packet loss detection and error detection on the data packet, errors occurring in the data transmission process can be discovered and corrected in a timely manner, improving the accuracy and integrity of the data.

[0063] Ensure data reliability: detecting whether the message header of the data packet carries an encoding label can distinguish between uncoded data packets and coded data packets, and then perform targeted processing on different types of data packets to ensure the reliability of data transmission.

[0064] Enhance system stability: abnormality detection on received data packets can discover and handle abnormal situations in a timely manner, avoiding the impact of abnormal data on the system and enhancing the stability of the system.

[0065] Improve transmission quality: after packet loss detection and error detection, the detected passed uncoded data packet and the detected passed coded data packet are obtained, and the quality of these data packets is guaranteed, which helps to improve the quality of the entire data transmission.

[0066] Further, the transmission logic performs abnormality detection on the received data packet, and detects whether the message header of the data packet carries an encoding label, obtains a detected passed uncoded data packet and a detected passed coded data packet, and stores the detected passed uncoded data packet in a corresponding position in the memory, and sends the detected passed coded data packet to a decoding module, comprising:

[0067] The transmission logic performs packet loss detection and error detection on the uncoded data packet and the coded data packet;

[0068] If the detection result is an uncoded error data packet and / or an uncoded out-of-order data packet, a retransmission notification is sent back;

[0069] If the detection result is a coded error data packet, the coded data packet is discarded;

[0070] If the detection result is a detected passed uncoded data packet, the uncoded data packet is unpacked, and the unpacked data is stored in a corresponding position in the memory;

[0071] If the detection result is a detected passed coded data packet, when the number of coded data packets reaches the decoding requirement, the detected passed coded data packet is sent to the decoding module.

[0072] It should be noted that the embodiments of the present specification have the following beneficial effects through the above content:

[0073] Improve the accuracy and integrity of data transmission: through the packet loss detection and error detection of uncoded data packets and coded data packets, the problem data packets can be found and processed in time, such as retransmission notification or discarding error data packets, so as to ensure the accuracy and integrity of data transmission.

[0074] Optimize memory usage: the uncoded data packets that pass the detection are unpacked and stored in the corresponding position of the memory, so that the data can be reasonably stored and managed, and the memory usage efficiency is improved.

[0075] Ensure the smooth decoding: only when the number of coded data packets reaches the decoding requirement, it is sent to the decoding module, which ensures the smooth decoding process and avoids decoding failure or error caused by insufficient number of data packets.

[0076] Improve the reliability of the system: comprehensive detection and classification processing of data packets can effectively avoid the further propagation and processing of error data, and improve the reliability and stability of the system.

[0077] One or more embodiments of the present specification provide a data transmission device based on a network card system, the network card system comprising an encoding decision sub-module, an encoding execution sub-module, a transmission logic and a decoding module, the device comprising:

[0078] At least one processor; and,

[0079] The memory is in communication connection with the at least one processor; wherein,

[0080] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to:

[0081] When a first user sends transmission data, the network card system obtains the transmission command issued by the first user, parses the address information of the data storage according to the transmission command, and transmits the address information to the encoding decision sub-module;

[0082] The encoding decision sub-module obtains the data volume of the transmission data according to the address information, determines the encoding behavior information of the transmission data if it is determined that the transmission data needs to be encoded based on the data volume of the transmission data, calls the DMA controller, takes the transmission data to the encoding cache area of the encoding execution sub-module, and transmits the encoding behavior information to the encoding execution sub-module and the transmission logic respectively;

[0083] The encoding execution submodule receives the encoding behavior information, encodes the transmission data in the encoding cache area according to the encoding behavior information, and transmits the encoded data to the sending cache area of the transmission logic.

[0084] The transmission logic distinguishes and encapsulates the uncoded incoming data and the encoded incoming data to obtain data packets, the data packets including uncoded data packets and encoded data packets, and for the encoded data packets, an encoding label carrying the encoding behavior information is added in a packet header.

[0085] When the second user receives the transmission data, the transmission logic performs anomaly detection on the received data packets, detects whether the encoded label is carried in the packet header of the data packets, obtains uncoded data packets passing the detection and encoded data packets passing the detection, and stores the uncoded data packets passing the detection in a corresponding position of the memory and sends the encoded data packets passing the detection to a decoding module.

[0086] The decoding module decodes the received encoded data packets passing the detection to obtain decoded data and stores the decoded data in a corresponding position of the memory.

[0087] One or more embodiments of the present specification provide a non-volatile computer storage medium, a network card system including an encoding decision submodule, an encoding execution submodule, a transmission logic and a decoding module, and storing computer executable instructions, the computer executable instructions being executed by a computer to implement:

[0088] When a first user sends transmission data, the network card system obtains a transmission command issued by the first user, parses address information of data storage according to the transmission command, and transmits the address information to the encoding decision submodule;

[0089] The encoding decision submodule obtains the data volume of the transmission data according to the address information, determines the encoding behavior information of the transmission data if the transmission data needs to be encoded based on the data volume of the transmission data, calls a DMA controller, takes the transmission data to the encoding cache area of the encoding execution submodule, and transmits the encoding behavior information to the encoding execution submodule and the transmission logic respectively.

[0090] The encoding execution submodule receives the encoding behavior information, encodes the transmission data in the encoding cache area according to the encoding behavior information, and transmits the encoded data to the sending cache area of the transmission logic.

[0091] The transmission logic distinguishes and encapsulates the uncoded incoming data and the coded incoming data to obtain data packets, the data packets including uncoded data packets and coded data packets, and for the coded data packets, an encoding label carrying the encoding behavior information is marked in a packet header;

[0092] When the second user receives the transmission data, the transmission logic performs anomaly detection on the received data packets, detects whether the coded labels are carried in the packet headers of the data packets, obtains uncoded data packets that pass the detection and coded data packets that pass the detection, and stores the uncoded data packets that pass the detection in corresponding positions of a memory and sends the coded data packets that pass the detection to a decoding module;

[0093] The decoding module decodes the received coded data packets that pass the detection to obtain decoded data and stores the decoded data in corresponding positions of a memory.

[0094] The above at least one technical solution adopted by the embodiments of the present specification can achieve the following beneficial effects:

[0095] Improve data transmission reliability: Traditional RoCEv2 is sensitive to packet loss and out-of-order, while the present method applies network coding for data transmission. When packet loss occurs in the network, the receiving end can recover the lost data packets through decoding operations, reducing data loss and transmission errors caused by packet loss and improving the reliability of data transmission.

[0096] Reduce the delay caused by retransmission: Although the improved RoCE network card uses the selective repeat protocol (SR protocol), the retransmission behavior still has a lag, which may cause high delay. Through network coding, retransmission of lost data packets is avoided to some extent, thereby reducing transmission delay and improving real-time performance of data transmission.

[0097] Enhance network adaptability: Network coding can make data transmission have stronger adaptability when facing packet loss. Even if the network environment is unstable and packet loss occurs, data can be recovered through decoding to ensure data integrity and accuracy.

[0098] Optimize network resource utilization: Reduces network resource waste caused by retransmission, improves network bandwidth utilization, and makes network resources more effectively utilized.

[0099] Improve overall system performance: The collaborative work of the encoding decision sub-module, the encoding execution sub-module, the transmission logic and the decoding module, and the application of network coding improve the data processing efficiency and performance of the entire system, making data transmission more efficient and stable. BRIEF DESCRIPTION OF DRAWINGS

[0100] In order to more clearly illustrate the technical solutions in the embodiments of the present specification or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present specification, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0101] Figure 1 A flowchart of an RDMA selective repeat protocol process provided for one or more embodiments of the present specification;

[0102] Figure 2 A flowchart of a data transmission method based on a network card system provided for one or more embodiments of the present specification;

[0103] Figure 3 A network card system architecture diagram provided for one or more embodiments of the present specification;

[0104] Figure 4 A small data amount transmission flow completion time comparison diagram provided for one or more embodiments of the present specification;

[0105] Figure 5 A large data amount transmission flow completion time comparison diagram provided for one or more embodiments of the present specification;

[0106] Figure 6 A flow completion time comparison diagram under different packet loss rates provided for one or more embodiments of the present specification;

[0107] Figure 7 A flow completion time comparison diagram under different round-trip transmission delays provided for one or more embodiments of the present specification;

[0108] Figure 8 A structure diagram of a data transmission device based on a network card system provided for one or more embodiments of the present specification. DETAILED DESCRIPTION

[0109] The embodiments of the present specification provide a data transmission method, device and medium based on a network card system.

[0110] In order for those skilled in the art to better understand the technical solutions in the present specification, the technical solutions in the embodiments of the present specification will be described clearly and completely in conjunction with the drawings in the embodiments of the present specification. Obviously, the described embodiments are only some embodiments of the present specification, not all embodiments. Based on the embodiments of the present specification, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present specification.

[0111] Figure 2 A flowchart of a data transmission method based on a network card system is provided for one or more embodiments of the present specification, which can be executed by a data transmission system. The embodiments of the present specification are applied to a scenario of supporting RDMA transmission on a lossy network. The network card system can include an encoding decision submodule, an encoding execution submodule, a transmission logic, and a decoding module. Some input parameters or intermediate results in the flow allow manual intervention to adjust to help improve accuracy.

[0112] The method flow steps of the embodiments of the present specification are as follows:

[0113] S202, when the first user sends transmission data, the network card system obtains the transmission command issued by the first user, parses the address information of the data storage according to the transmission command, and transmits the address information to the encoding decision submodule.

[0114] In the embodiments of the present specification, when the first user sends transmission data, the network card system can first receive a transmission notification and then obtain the transmission command issued by the first user. Regarding the above, the following specific implementation can be used:

[0115] Obtaining of the transmission command: the network card system can set up a listening mechanism to monitor the operation of the first user in real time. When the first user sends the transmission data, the network card system can timely capture the event and obtain the transmission command issued by the first user.

[0116] Analysis of the transmission command: a special analysis module can be configured in the network card system to analyze the transmission command. The analysis module extracts the information related to the data storage address from the transmission command according to the preset rules and formats.

[0117] Transmission of the address information: after the address information is parsed, the network card system transmits the address information to the encoding decision submodule through an internal data channel. To ensure the stability and accuracy of data transmission, some verification mechanisms can be used to verify the integrity and correctness of the address information.

[0118] S204, the encoding decision submodule obtains the data volume of the transmission data according to the address information, determines the encoding behavior information of the transmission data if it is determined that the transmission data needs to be encoded based on the data volume of the transmission data, calls a DMA controller, takes the transmission data to the encoding cache area of the encoding execution submodule, and transmits the encoding behavior information to the encoding execution submodule and the transmission logic, respectively.

[0119] In the embodiments of the present specification, when determining the data amount of the transmission data, the encoding decision submodule can obtain the address quantity according to the start address and the end address; and obtain the data amount of the transmission data according to the data amount of each address and the address quantity. Wherein, the addresses of each data in the transmission data are consecutive addresses.

[0120] It should be noted that, regarding the determination of the data amount of the transmission data, the following specific implementation can be used:

[0121] Obtaining the start address and the end address: the encoding decision submodule extracts the start address and the end address from the received address information.

[0122] Calculating the address quantity: using a mathematical calculation method, subtracting the value of the start address from the value of the end address and then adding 1, the address quantity can be obtained.

[0123] Obtaining the data amount of each address: obtaining the data amount information corresponding to each address from the related configuration.

[0124] Calculating the data amount of the transmission data: multiplying the data amount of each address by the address quantity to obtain the data amount of the transmission data.

[0125] It should be noted that, through the above content, the embodiments of the present specification have the following beneficial effects:

[0126] Accurate calculation of data amount: the decision submodule needs to calculate the total sending amount and the to-be-sent data amount for encoding decision. The network card obtains the transmission command issued by the user, parses the data storage address, calculates the total transmission amount through the start address and the end address, and obtains the to-be-sent data amount through the current sending address and the end address.

[0127] Optimizing resource allocation: after accurately knowing the data amount of the transmission data, system resources such as memory and bandwidth can be more reasonably allocated, avoiding waste or shortage of resources, and improving the overall performance and resource utilization of the system.

[0128] Supporting encoding decision: the encoding decision submodule is provided with key information. Based on the accurate data amount calculation, the encoding decision submodule can more accurately judge whether the data needs to be encoded, and determine the appropriate encoding behavior information, thereby improving the reliability and efficiency of data transmission.

[0129] Further, if it is determined that the transmission data needs to be encoded based on the data amount of the transmission data, the current network card sending rate of the network card system can be obtained; the round-trip time RTT of the current network and the packet loss rate of the current network are determined according to the current received data packet; based on the to-be-sent data amount, the current network card sending rate, the RTT of the current network and the packet loss rate of the current network, it is judged whether the data amount of the transmission data needs network coding.

[0130] It should be noted that, in relation to the above, the following specific embodiments can be used:

[0131] Obtain the current network card sending rate: the sending rate information of the network card can be read by using the interface or tool provided by the network card system. This may require interaction with the driver program of the network card or related system API.

[0132] Determine the round-trip time (RTT) and packet loss rate of the current network: the RTT can be determined by sending test data packets and measuring their round-trip time. Data packets can be sent to the target address periodically, and the time of sending and the time of receiving the response are recorded, and the difference between the two is calculated as the measurement value of RTT. For the determination of packet loss rate, the number of sent data packets and the number of received data packets can be counted. A certain number of data packets are sent within a certain time, and then the actual number of received data packets is compared to calculate the packet loss rate.

[0133] Determine whether the data volume of the transmission data needs network coding: according to actual needs and experience, some judgment conditions are determined to decide whether network coding is needed. For example, a threshold can be set, if the data volume to be sent is large, the network card sending rate is low, the RTT is long or the packet loss rate is high, it is considered that network coding is needed; otherwise, it is not needed. The obtained data volume to be sent, the current network card sending rate, the RTT and the packet loss rate of the current network are substituted into the judgment rule for calculation and comparison. According to the judgment result, it is determined whether the data volume of the transmission data needs network coding.

[0134] It should be noted that, in relation to the above, the following specific embodiments can be used:

[0135] Improve transmission efficiency: by obtaining the current network card sending rate of the network card system, determining the round-trip time (RTT) and packet loss rate of the current network, and combining the data volume to be sent to determine whether network coding is needed. In this way, optimization decisions can be made according to the actual network conditions and data volume, avoiding unnecessary coding operations, thereby improving the efficiency of data transmission.

[0136] Optimize network resource utilization: considering the current network card sending rate, RTT and packet loss rate, network resources can be allocated more reasonably. For the case of good network condition and low packet loss rate, the use of coding can be reduced to save computing resources; for the case of poor network condition and high packet loss rate, the reliability of data transmission is improved by determining whether coding is needed, and network resources are fully utilized.

[0137] Improving the reliability of data transmission: According to the current network RTT and packet loss rate, it is judged whether network coding is needed, and the corresponding processing can be carried out according to the packet loss situation in the network. If the packet loss rate is high, the redundancy of data can be increased through coding, the reliability of data transmission can be improved, and the influence of data loss on transmission can be reduced.

[0138] Adapting to different network environments: This method can be adjusted according to real-time network conditions. Different network environments may have different network card sending rates, RTT and packet loss rates. Through this dynamic judgment method, the transmission data can better adapt to various network conditions, and the stability and adaptability of data transmission can be improved.

[0139] Reducing transmission cost: By accurately judging whether network coding is needed, the additional cost such as calculation cost and time cost caused by excessive use of coding can be avoided. At the same time, reasonable coding decision can also reduce the network bandwidth consumption caused by data retransmission, thereby reducing the transmission cost.

[0140] Further, when judging whether the data volume of the transmission data needs network coding based on the to-be-sent data volume, the current network card sending rate, the current network RTT and the current network packet loss rate, the first threshold value and the second threshold value can be set for the network RTT and the packet loss rate respectively; the third threshold value is set for the data transmission time; the current data transmission time is determined according to the current network card sending rate and the data volume of the transmission data; based on the to-be-sent data volume, it is judged whether the current network RTT meets the first threshold value, and / or whether the current network packet loss rate meets the second threshold value, and / or whether the current data transmission time meets the third threshold value.

[0141] It should be noted that the above content can be implemented through the following specific embodiments:

[0142] Setting thresholds: determine the range of the amount of data to be sent and divide it into multiple steps. For example, it can be divided into step 1, step 2, step 3, etc. in order from small to large according to the size of the data amount. For the amount of data to be sent in each step, the first threshold of RTT, the second threshold of packet loss rate and the third threshold of data transmission time are set respectively. These thresholds can be determined according to actual experience and network performance requirements. For example, when the amount of data to be sent is small, the first threshold of RTT can be set relatively high, the second threshold of packet loss rate can be set relatively low, and the third threshold of data transmission time can be set relatively long; as the amount of data to be sent increases, gradually reduce the first threshold of RTT, increase the strictness of the second threshold of packet loss rate, and shorten the third threshold of data transmission time. Determine the current data transmission time: obtain the current network card sending rate and the amount of data to be transmitted. Through the formula: data transmission time = amount of data to be transmitted / current network card sending rate, the current data transmission time is calculated.

[0143] Determine whether the thresholds meet the requirements: obtain the RTT and packet loss rate of the current network. According to the step to which the amount of data to be sent belongs, the corresponding first threshold of RTT, the second threshold of packet loss rate and the third threshold of data transmission time are determined. Determine whether the RTT of the current network is less than or equal to the first threshold, if yes, it is considered that the RTT meets the requirements; otherwise, it is considered that it does not meet the requirements. Determine whether the packet loss rate of the current network is less than or equal to the second threshold, if yes, it is considered that the packet loss rate meets the requirements; otherwise, it is considered that it does not meet the requirements. Determine whether the current data transmission time is less than or equal to the third threshold, if yes, it is considered that the data transmission time meets the requirements; otherwise, it is considered that it does not meet the requirements.

[0144] Take corresponding measures: if the RTT of the current network meets the first threshold, and the packet loss rate of the current network meets the second threshold, and the current data transmission time meets the third threshold, the data transmission can be carried out in a normal way without special processing. If the RTT of the current network does not meet the first threshold, or the packet loss rate of the current network does not meet the second threshold, or the current data transmission time does not meet the third threshold, corresponding measures need to be taken according to the specific situation. For example, the encoding mode can be adjusted, the redundant information can be increased, the transmission rate can be adjusted, etc. to improve the performance and reliability of data transmission. It should be noted that the embodiments of the present specification have the following beneficial effects through the above content:

[0145] Optimize network performance: by setting the thresholds of RTT, packet loss rate and data transmission time, and judging according to the current situation, the problems in the network can be found in time, and corresponding measures can be taken to optimize the network performance and improve the efficiency and quality of data transmission.

[0146] Adapt to different data volume requirements: the threshold changes with the change of the amount of data to be sent, which can better adapt to different sizes of data transmission requirements. For smaller data volume, the threshold requirement can be appropriately relaxed to avoid excessive resource consumption; for larger data volume, the strictness of the threshold can be improved to ensure the reliability and timeliness of data transmission.

[0147] Improve the reliability of data transmission: through the monitoring and judgment of RTT and packet loss rate, unstable factors in the network can be found in time, and measures can be taken to repair or adjust, so as to reduce the risk of data loss and improve the reliability of data transmission.

[0148] Reasonable allocation of resources: according to the current network card sending rate and the data volume of the transmission data to determine the current data transmission time, and compare with the set third threshold, which can help to reasonably allocate network resources and avoid resource waste or shortage.

[0149] Enhance the adaptability of network: the scheme can dynamically adjust according to the actual situation of network, so that the network can better adapt to different environments and requirements, and improve the overall adaptability and flexibility of network.

[0150] Further, if it is determined based on the data volume of the transmission data that the transmission data does not need to be encoded, the transmission data can be fetched to the sending cache area of the transmission logic according to the address information.

[0151] It should be noted that the embodiments of the present application have the following beneficial effects through the above content:

[0152] Improve transmission efficiency: when it is determined that the transmission data does not need to be encoded, it is directly fetched to the sending cache area of the transmission logic according to the address information, which reduces the time consumption caused by unnecessary encoding operation, thereby improving the overall efficiency of data transmission.

[0153] Save system resources: avoid the occupation of system resources such as computing resources and memory resources during the encoding process, so that these resources can be better allocated to other tasks, and the utilization rate of system resources is improved.

[0154] Further, regarding the determination of the encoding behavior information of the transmission data and the calling of the DMA controller to fetch the transmission data to the encoding cache area of the encoding execution submodule, the following specific implementation schemes can be used:

[0155] determining encoding behavior information: once it is determined that the transmission data needs to be encoded, further determine the encoding behavior information, which can include the data volume of the encoding batch and the redundant data volume generated by encoding. As for the data volume of the encoding batch, the data volume of each encoding process can be determined according to the total data volume of the transmission data and the requirement of encoding. As for the redundant data volume generated by encoding, the redundant data volume generated in the encoding process can be calculated according to the encoding algorithm and requirement.

[0156] calling DMA controller: starting the DMA (Direct Memory Access) controller to directly transmit data between memories through hardware, thereby improving the data transmission efficiency.

[0157] transferring the transmission data to the encoding cache area: the DMA controller directly transfers the transmission data from the original storage location to the encoding cache area of the encoding execution submodule according to the set data volume of the encoding batch, so as to perform subsequent encoding operation.

[0158] S206, when the encoding execution submodule receives the encoding behavior information, the encoding execution submodule encodes the transmission data in the encoding cache area according to the encoding behavior information, and transmits the encoded data to the sending cache area of the transmission logic.

[0159] In the embodiments of the present application, as for S206, the following specific implementation can be used:

[0160] receiving encoding behavior information: the encoding execution submodule continuously listens and waits to receive the encoding behavior information.

[0161] obtaining the transmission data of the encoding cache area: the encoding execution submodule reads the transmission data to be encoded from the encoding cache area after receiving the encoding behavior information.

[0162] performing encoding operation: encoding the transmission data in the encoding cache area according to the encoding mode specified in the encoding behavior information.

[0163] transmitting the encoded data: transmitting the encoded data to the sending cache area of the transmission logic, so as to perform subsequent data sending operation.

[0164] S208, the transmission logic distinguishes and encapsulates the uncoded incoming data and the encoded incoming data to obtain a data packet, the data packet includes uncoded data packet and encoded data packet, and for the encoded data packet, an encoding label carrying the encoding behavior information is marked in the packet header.

[0165] In the embodiments of the present application, the encoded data includes redundant data and initial data. As for S208, the following specific implementation can be used:

[0166] Data reception and classification: The transmission logic module receives incoming data. According to whether the data is encoded, the data is divided into unencoded incoming data and encoded incoming data.

[0167] Data encapsulation: The unencoded incoming data is encapsulated to obtain unencoded data packets. The encoded incoming data is encapsulated to obtain encoded data packets.

[0168] Encoding tag addition: For the encoded data packets, an encoding tag is added in the packet header. The encoding tag carries encoding behavior information, including the data volume of the encoding batch and the redundant data volume generated by encoding.

[0169] Data processing: For the encoded data, it is divided into redundant data and initial data for processing.

[0170] S210, when the second user receives the transmission data, the transmission logic detects the received data packet for anomaly detection, and detects whether the encoding tag is carried in the packet header of the data packet, to obtain the unencoded data packet passed the detection and the encoded data packet passed the detection, and store the unencoded data packet passed the detection in the corresponding position of the memory, and send the encoded data packet passed the detection to the decoding module.

[0171] In the embodiments of the present specification, when obtaining the unencoded data packet passed the detection and the encoded data packet passed the detection, the transmission logic can detect whether the encoding tag is carried in the packet header of the data packet, to obtain the unencoded data packet and the encoded data packet; the transmission logic performs packet loss detection and error detection on the unencoded data packet and the encoded data packet, to obtain the unencoded data packet passed the detection and the encoded data packet passed the detection.

[0172] It should be noted that, regarding the above content, the following specific implementation schemes can be used:

[0173] Detecting the encoding tag: After receiving the data packet, the transmission logic reads the packet header of the data packet. Check whether the encoding tag is carried in the packet header. According to whether the encoding tag exists, the data packet is divided into unencoded data packet and encoded data packet.

[0174] Packet loss detection: For unencoded data packets and encoded data packets, packet loss detection is performed respectively. Whether there is a packet loss situation can be judged by checking the sequence number or other identifier of the data packet. If it is found that the data packet is lost, according to the specific transmission protocol and requirements, the corresponding processing measures are taken, such as requesting retransmission or error handling.

[0175] Error detection: error detection is performed on uncoded data packets and coded data packets. Error detection methods such as checksum, cyclic redundancy check (CRC) and the like can be used. The checksum of the data packet is calculated and compared with the preset standard value. If the checksum does not match, there is an error, and appropriate error handling measures such as error correction or retransmission request are taken.

[0176] Get the detected data packet: after the packet loss detection and error detection, the uncoded data packet and the coded data packet that pass the detection are marked as the detected uncoded data packet and the detected coded data packet respectively.

[0177] It should be noted that the embodiments of the present application have the following beneficial effects through the above content:

[0178] Improve data accuracy: by performing packet loss detection and error detection on data packets, errors occurring in the data transmission process can be detected and corrected in time, improving the accuracy and integrity of the data.

[0179] Ensure data reliability: detecting whether the packet header of the data packet carries an encoding label can distinguish uncoded data packets and coded data packets, and then process different types of data packets accordingly, ensuring the reliability of data transmission.

[0180] Enhance system stability: abnormality detection on received data packets can detect and handle abnormal situations in time, avoid the impact of abnormal data on the system, and enhance the stability of the system.

[0181] Improve transmission quality: after packet loss detection and error detection, the detected uncoded data packet and the detected coded data packet are obtained, and the quality of these data packets is guaranteed, which helps to improve the quality of the entire data transmission.

[0182] Further, the transmission logic can perform packet loss detection and error detection on the uncoded data packet and the coded data packet; if the detection result is an uncoded error data packet and / or an uncoded out-of-order data packet, a retransmission notification is sent; if the detection result is a coded error data packet, the coded data packet is discarded; if the detection result is a detected uncoded data packet, the uncoded data packet is unpacked and the unpacked data is stored in the corresponding position in the memory; if the detection result is a detected coded data packet, when the number of the coded data packet reaches the decoding requirement, the detected coded data packet is sent to the decoding module.

[0183] It should be noted that the above content can be implemented through the following specific embodiments:

[0184] Packet loss detection and error detection: the transmission logic performs packet loss detection and error detection on the uncoded data packets and the coded data packets respectively. Packet loss detection can be performed by checking the sequence number, acknowledgement number and other information of the data packets to determine whether there is any missing data packet. Error detection can be performed by using checksum, cyclic redundancy check (CRC) and other methods to check whether there is any error in the data packets.

[0185] Processing according to the detection result: if the detection result is uncoded error data packet and / or uncoded out-of-order data packet, the transmission logic sends a retransmission request to the sending end; if the detection result is coded error data packet, the transmission logic directly discards the coded data packet; if the detection result is uncoded data packet that passes the detection, the transmission logic performs data packet unpacking on the uncoded data packet, extracts the data therein, and stores the unpacked data in the corresponding position in the memory; if the detection result is coded data packet that passes the detection, the transmission logic counts the coded data packet that passes the detection, and sends the data packets to the decoding module when the number of the coded data packets reaches the decoding requirement.

[0186] It should be noted that the embodiments of the present specification have the following beneficial effects through the above content:

[0187] Improve the accuracy and integrity of data transmission: by performing packet loss detection and error detection on uncoded data packets and coded data packets, the problematic data packets can be found and processed in time, such as sending a retransmission request or discarding error data packets, thereby ensuring the accuracy and integrity of data transmission.

[0188] Optimize memory usage: the uncoded data packet that passes the detection is unpacked and stored in the corresponding position in the memory, so that the data can be reasonably stored and managed, improving the efficiency of memory usage.

[0189] Ensure the smooth progress of decoding: only when the number of coded data packets reaches the decoding requirement, the coded data packets are sent to the decoding module, ensuring the smooth progress of the decoding process, avoiding decoding failure or error caused by insufficient number of data packets.

[0190] Improve the reliability of the system: comprehensive detection and classification processing of data packets can effectively avoid the further propagation and processing of error data, improving the reliability and stability of the system.

[0191] S212, the decoding module decodes the received coded data packet that passes the detection to obtain decoded data, and stores the decoded data in the corresponding position in the memory.

[0192] In the embodiments of the present specification, S212 can be implemented by the following specific implementation schemes:

[0193] Data receiving: the decoding module waits for the encoded data packet sent by the transmission logic to pass the detection.

[0194] Decoding operation: after receiving the encoded data packet, the data packet is decoded using the corresponding decoding algorithm. Ensure that the decoding algorithm corresponds to the algorithm used in the encoding process to correctly restore the data.

[0195] Data storage: store the decoded data obtained after decoding according to its corresponding position in the memory. The storage position planning of each data in the memory needs to be determined in advance to accurately store the data.

[0196] It should be noted that with the continuous development of network infrastructure and forwarding equipment in data centers, the link transmission speed of the current data center network has reached 100 Gbps and is developing towards 400 Gbps. This also promotes the running of various online data-intensive applications, distributed computing services and machine learning training tasks on the data center structure. These throughput-sensitive and latency-sensitive applications all need low-latency, high-throughput networks and high-CPU utilization server-side support. However, the traditional software-implemented TCP / IP protocol greatly reduces the CPU utilization and is difficult to provide low-latency guarantees. Therefore, the hardware offload transmission protocol Remote Direct Memory Access (RDMA) technology has been widely applied on data center Ethernet. Google, Microsoft, Alibaba and other data center operators have deployed RoCEv2 (RDMA over Converged Ethernet version 2).

[0197] RDMA technology has achieved preliminary success in deploying on Ethernet, however, when the scale of the RDMA network expands, it will be difficult to continue to maintain high bandwidth utilization and low transmission latency. This is because RoCEv2 runs on a lossless network guaranteed by the Priority Flow Control (PFC) mechanism. The negative effects such as head-of-line blocking, congestion propagation, and unfairness caused by the PFC mechanism will cause network performance loss, and the mechanism also faces the risk of network blocking due to pause frame storms and occasional deadlocks, and these disadvantages will be amplified when the network scale expands. Therefore, abandoning the lossless network guarantee and supporting RDMA on a lossy network becomes a feasible solution.

[0198] Supporting RDMA over lossy network also faces severe challenges. Unlike TCP / IP protocol, traditional RoCEv2 is very sensitive to packet loss and out-of-order, because the hardware implementation of the transport protocol on the NIC is difficult to implement complex packet loss processing logic. The improved RoCE NIC (for Improver RoCE NIC, IRN) implements a new NIC design, which replaces the traditional RoCE NIC's N-frame back protocol (GBN protocol) for out-of-order packets with a selective repeat protocol (SR protocol), which implements a more efficient loss recovery strategy. As shown in Figure 1 , simply put, when the receiver receives an out-of-order packet, it will be considered as a packet loss at this time, and a retransmission notification will be sent to the receiver. After receiving the retransmission notification, the receiver retransmits the missing packet. However, the retransmission behavior always has a lag, which will affect the low latency guarantee of the flow.

[0199] The application is applied to the scene of supporting RDMA transmission over lossy network (such as Ethernet), and when packet loss or failure occurs in the network, loss recovery can be completed through packet retransmission. The problems to be solved by the application are: 1, to reduce the negative impact of flow completion time delay caused by retransmission. 2, to achieve lower time delay without affecting throughput.

[0200] The loss recovery problem of deploying RDMA on a large-scale network aims to avoid packet retransmission as much as possible to reduce latency. The application innovatively applies network coding to RDMA network for data transmission. When packet loss occurs in the network, the receiving end can still recover the missing packet through decoding operation. First, the encoding and decoding time introduced by various network coding schemes are tested. The current experimental test confirms that the encoding and decoding speed of Cauchy coding can match the RDMA transmission speed, and of course other types of encoding (such as Raptor encoding, LT code, etc.) can also be used under the premise of performance guarantee. In the experiment, the influence factors of Cauchy coding encoding and decoding speed and the influence of encoding behavior on network performance are analyzed, and the key factors that affect cluster performance such as time overhead, network throughput, and decoding guarantee are studied. under the condition that the network congestion packet loss changes constantly. Further, a model is constructed to express the problem, quantify the influence of the redundant packets introduced by network coding on network effective throughput and the influence of reducing retransmission behavior on flow completion time. Further, a dynamic coding strategy that quickly responds to changes in network performance is designed, and an NIC with integrated network coding function and self-adjusting coding scheme is designed using FPGA technology, which is named as network coding supported RDMA NIC.

[0201] The system architecture diagram of the NIC is as shown in Figure 3As shown, the main functional modules include an encoding module, a decoding module and a transmission logic. The encoding module includes an encoding decision submodule and an encoding execution submodule. The specific functions of the modules are described as follows.

[0202] The encoding module is responsible for adjusting and executing the encoding scheme. The encoding decision submodule formulates a suitable encoding scheme according to the current network condition to reduce the flow completion time as much as possible. The influencing factors are complex and changeable, and the process of establishing the mathematical model is as follows:

[0203] The encoding execution submodule in the encoding module performs calculation according to the encoding scheme at this time.

[0204] The transmission logic is responsible for protocol stack encapsulation and sending of encoded or unencoded data, and calculating the RTT (round trip time) and packet loss rate of the network according to the current received data packet, and transmitting these information to the encoding module. For the received data packet, the loss and error conditions are checked, and for the lost or erroneous data packet, decoding or retransmission notification is performed according to the encoding condition.

[0205] The decoding module is responsible for decoding the transmitted data and placing the data in the corresponding position in the memory through DMA technology.

[0206] The above functional modules always play a role in the whole process of data stream transmission, and the data plane process of data transmission will be described in more detail.

[0207] For the sender, the network card first obtains the transmission command issued by the user, parses the address information of the data storage, and transmits the address information to the encoding decision submodule.

[0208] The encoding decision submodule obtains the transmission data volume according to the address information of the data storage, and makes a decision whether to perform network coding in combination with the recorded RTT and packet loss rate and the network card sending rate (whether to perform network coding is decided by considering these three factors. If the RTT is long, the packet loss rate is high, or the network card sending rate is limited, it may be considered that the current network condition is not ideal, and network coding at this time may help to improve the efficiency and reliability of data transmission.) If encoding is needed, the encoding behavior is formulated, including the data volume of the encoding batch (the data volume of the encoding batch refers to the data volume size processed each time) and the redundant data volume generated by the encoding (the redundant data volume is the data volume generated additionally to improve the reliability and fault tolerance of the data), the DMA controller is called to take the corresponding data to the encoding buffer of the encoding execution submodule, and then the previously generated encoding behavior information is transmitted to the encoding execution submodule and the transmission logic respectively. If no encoding is needed, the data is taken to the sending buffer of the transmission logic according to the address.

[0209] When the encoding execution sub-module receives the encoding behavior information (which means that the encoding execution sub-module receives relevant instructions and information on how to encode), the contents in the encoding buffer are encoded according to the description (after receiving the encoding behavior information, the encoding execution sub-module will encode the contents in the encoding buffer according to the requirements of the information. The encoding buffer is an area for storing data to be encoded, and the encoding execution sub-module will process the contents therein according to specific encoding algorithms and rules), and then the generated redundant data and initial data are transmitted to the sending buffer of the transmission logic (after the encoding operation is completed, redundant data (for improving data reliability and fault tolerance) and initial data are generated. These data are transmitted to the sending buffer of the transmission logic. The sending buffer is an area for temporarily storing data to be transmitted, so that subsequent data transmission operations can be more efficient).

[0210] The transmission logic distinguishes between directly transmitted data and encoded transmitted data, and performs inconsistent protocol encapsulation behaviors on the two types of data. For encoded data packets, an encoding label is added in the data packet message header and necessary encoding behavior information is carried.

[0211] The transmission logic sends the encapsulated data packets at the current sending rate.

[0212] For the receiving party, the transmission logic first performs packet loss detection and error detection on the received data packets and checks whether the encoding label is carried in the data packet message header, and performs corresponding processing: 1) for unencoded error data packets and out-of-order data packets, immediately retransmit a retransmission notification; 2) for encoded error data packets, discard without any processing; 3) for unencoded in-order data packets, perform data packet decapsulation and store the data in the corresponding memory location, and accumulate a certain number of data packets and then reply to the confirmation; 4) for encoded data packets, obtain the encoding behavior information and wait, when the number of received data packets reaches the decoding requirement, perform data packet decapsulation and send the data and data sequence number to the decoding module, and reply to the confirmation to the sender. For the redundant data packets of the encoding batch that arrive subsequently, immediately transmit a redundant notification to the sender to stop sending the current encoding data block and start sending the next data block. For the case where the data packets are not received in time to support decoding, a retransmission notification of the encoding data packets is sent. Note that for the encoded data packets, the retransmission notification and the redundant notification will immediately trigger the sender to develop a new encoding behavior.

[0213] The decoding module decodes the received data to recover lost or error data and places it in the correct memory address according to the incoming order.

[0214] For the sender, if a retransmission notification or acknowledgment is received during transmission, the receiving logic first determines whether the retransmission / acknowledgment data packet is a retransmission / acknowledgment of an encoded data packet. Based on this information, it then performs adjustments to key parameters such as transmission rate, packet loss rate, and RTT. For data lost even after encoding, it will be retransmitted immediately. For unencoded lost data, the loss information is sent to the encoding decision submodule, which decides whether to encode and transmit the retransmitted data packet.

[0215] The sender and receiver will communicate in accordance with this protocol until the transmission is complete.

[0216] Simulation experiments (primarily simulating random packet loss) show that network coding achieves lower latency for short flows, and through coding decisions, it does not reduce throughput for long flows. Furthermore, its advantages remain significant even when network reliability decreases or network size increases. The experiment first compared the flow completion time of the coding and non-coding schemes as the amount of transmitted data increased, with a fixed packet loss rate of 0.1%, an RTT of 8 microseconds, and a transmission rate of 100Gbps. Figure 4 The diagram illustrating the completion time comparison for small data volume transmission streams shows that when the data transmission volume is small, the encoding completion time is consistently shorter than the non-encoding completion time. Furthermore, when the data volume is 50KB, the transmission stream can be completed within one RTT. In this case, the encoding scheme, by avoiding the delay introduced by retransmissions, reduces the stream completion time by 56.5% compared to the non-encoding scheme. Additionally, as... Figure 5 The diagram showing the completion time comparison of large data volume transmission streams demonstrates that even with a large amount of data being transmitted, the method proposed in this invention can still achieve lower latency. This indicates that by controlling the coding decisions, low latency can be achieved without wasting excessive throughput.

[0217] like Figure 6 The diagram showing the comparison of transmission completion time under different packet loss rates, with a fixed data transmission volume of 500KB and an RTT of 8 microseconds, illustrates that the low latency advantage of the encoding scheme becomes more significant as the network packet loss rate increases. This demonstrates that even in large-scale network environments where reliability is difficult to guarantee, the scheme proposed in this invention can still achieve effective low-latency transmission. Finally, Figure 7 The diagram showing the comparison of different round-trip transmission delays and stream completion times illustrates that the stream completion time fluctuates dramatically with the increase of RTT. However, the stream completion time of the coded scheme is always lower than that of the non-coded scheme. This means that the scheme proposed in this invention still maintains its low latency advantage when facing network congestion or wide-area transmission scenarios.

[0218] Figure 8A structural schematic of a data transmission device based on a network card system is provided for one or more embodiments of the present specification, the network card system comprising an encoding decision submodule, an encoding execution submodule, a transmission logic and a decoding module, the device comprising:

[0219] at least one processor; and,

[0220] a memory in communication connection with the at least one processor; wherein,

[0221] the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to:

[0222] when a first user sends transmission data, the network card system obtains a transmission command issued by the first user, parses address information of data storage according to the transmission command, and transmits the address information to the encoding decision submodule;

[0223] the encoding decision submodule obtains the data volume of the transmission data according to the address information, determines the encoding behavior information of the transmission data if it is determined that the transmission data needs to be encoded based on the data volume of the transmission data, calls a DMA controller, takes the transmission data to the encoding cache area of the encoding execution submodule, and transmits the encoding behavior information to the encoding execution submodule and the transmission logic respectively;

[0224] the encoding execution submodule receives the encoding behavior information, encodes the transmission data in the encoding cache area according to the encoding behavior information, and transmits the encoded data to the sending cache area of the transmission logic;

[0225] the transmission logic distinguishes and encapsulates the uncoded incoming data and the encoded incoming data to obtain data packets, the data packets comprising uncoded data packets and encoded data packets, and for the encoded data packets, an encoding label carrying the encoding behavior information is marked in the packet header;

[0226] when a second user receives the transmission data, the transmission logic performs anomaly detection on the received data packets, detects whether the encoding label is carried in the packet header of the data packets, obtains the uncoded data packets that pass the detection and the encoded data packets that pass the detection, stores the uncoded data packets that pass the detection in the corresponding position of the memory, and sends the encoded data packets that pass the detection to the decoding module;

[0227] the decoding module decodes the received encoded data packets that pass the detection to obtain decoded data, and stores the decoded data in the corresponding position of the memory.

[0228] The one or more embodiments of the specification provide a non-volatile computer storage medium, a network card system comprising an encoding decision submodule, an encoding execution submodule, a transmission logic and a decoding module, and storing computer executable instructions, which can realize the following when executed by a computer:

[0229] When a first user sends transmission data, the network card system obtains a transmission command issued by the first user, parses address information of data storage according to the transmission command, and transmits the address information to the encoding decision submodule;

[0230] The encoding decision submodule obtains the data volume of the transmission data according to the address information, determines the encoding behavior information of the transmission data if it is determined that the transmission data needs to be encoded based on the data volume of the transmission data, calls a DMA controller, takes the transmission data to the encoding cache area of the encoding execution submodule, and transmits the encoding behavior information to the encoding execution submodule and the transmission logic respectively;

[0231] When the encoding execution submodule receives the encoding behavior information, it encodes the transmission data in the encoding cache area according to the encoding behavior information, and transmits the encoded data to the sending cache area of the transmission logic;

[0232] The transmission logic distinguishes and encapsulates the uncoded incoming data and the encoded incoming data to obtain a data packet, the data packet comprising uncoded data packets and encoded data packets, and for the encoded data packets, an encoding label carrying the encoding behavior information is marked in the packet header;

[0233] When a second user receives the transmission data, the transmission logic performs anomaly detection on the received data packet, detects whether the packet header of the data packet carries an encoding label, obtains the uncoded data packet that passes the detection and the encoded data packet that passes the detection, and stores the uncoded data packet that passes the detection in a corresponding position of the memory and sends the encoded data packet that passes the detection to the decoding module;

[0234] The decoding module decodes the received encoded data packet that passes the detection to obtain decoded data, and stores the decoded data in a corresponding position of the memory.

[0235] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for the device and non-volatile computer storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

Claims

1. A data transmission method based on a network card system, characterized in that, The network card system comprises a coding decision submodule, a coding execution submodule, a transmission logic and a decoding module, and the method comprises the following steps: When a first user sends transmission data, the network card system obtains a transmission command issued by the first user, parses address information of the data storage according to the transmission command, and transmits the address information to the coding decision submodule; The coding decision submodule obtains the data volume of the transmission data according to the address information, determines the coding behavior information of the transmission data if it is determined that the transmission data needs to be coded based on the data volume of the transmission data, calls a DMA controller, takes the transmission data to a coding cache area of the coding execution submodule, and transmits the coding behavior information to the coding execution submodule and the transmission logic respectively; When the coding execution submodule receives the coding behavior information, it codes the transmission data in the coding cache area according to the coding behavior information, and transmits the coded data to a sending cache area of the transmission logic; The transmission logic distinguishes and encapsulates uncoded incoming data and coded incoming data to obtain data packets, wherein the data packets comprise uncoded data packets and coded data packets, and a coding label carrying the coding behavior information is added to the uncoded data packets in the packet header; When a second user receives the transmission data, the transmission logic performs anomaly detection on the received data packets, detects whether the coding label is carried in the packet header of the data packets, obtains uncoded data packets that pass the detection and coded data packets that pass the detection, stores the uncoded data packets that pass the detection in a corresponding position of the memory, and sends the coded data packets that pass the detection to the decoding module; The decoding module decodes the received coded data packets that pass the detection to obtain decoded data, and stores the decoded data in a corresponding position of the memory; Before it is determined that the transmission data needs to be coded based on the data volume of the transmission data, the method further comprises the following steps: Obtain the current network card sending rate of the network card system; Determine the round-trip time (RTT) of the current network and the packet loss rate of the current network according to the current received data packet situation; Determine whether the data volume of the transmission data needs network coding based on the to-be-sent data volume, the current network card sending rate, the RTT of the current network and the packet loss rate of the current network; The determination of whether the data volume of the transmission data needs network coding based on the to-be-sent data volume, the current network card sending rate, the RTT of the current network and the packet loss rate of the current network comprises the following steps: Set a first threshold value and a second threshold value for the RTT and the packet loss rate of the network respectively; Set a third threshold value for the data transmission time; Determine the current data transmission time according to the current network card sending rate and the data volume of the transmission data; Determine whether the RTT of the current network meets the first threshold value, whether the packet loss rate of the current network meets the second threshold value, and whether the current data transmission time meets the third threshold value based on the to-be-sent data volume. The transmission logic performs anomaly detection on the received data packet, and detects whether an encoding label is carried in a packet header of the data packet, to obtain an unencoded data packet passing the detection and an encoded data packet passing the detection, including: The transmission logic detects whether an encoding label is carried in a packet header of the data packet, to obtain the unencoded data packet and the encoded data packet; The transmission logic performs packet loss detection and error detection on the unencoded data packet and the encoded data packet, to obtain an unencoded data packet passing the detection and an encoded data packet passing the detection; The transmission logic performs anomaly detection on the received data packet, and detects whether an encoding label is carried in a packet header of the data packet, to obtain an unencoded data packet passing the detection and an encoded data packet passing the detection, and stores the unencoded data packet passing the detection in a corresponding position in the memory and sends the encoded data packet passing the detection to a decoding module, including: The transmission logic performs packet loss detection and error detection on the unencoded data packet and the encoded data packet; If the detection result is an unencoded error data packet and / or an unencoded out-of-order data packet, a retransmission notification is sent back; If the detection result is an encoded error data packet, the encoded data packet is discarded; If the detection result is an unencoded data packet passing the detection, the unencoded data packet is unpacked, and the unpacked data is stored in a corresponding position in the memory; If the detection result is an encoded data packet passing the detection, when the number of the encoded data packets passing the detection reaches a decoding requirement, the encoded data packets passing the detection are sent to the decoding module.

2. The method of claim 1, wherein, The encoding decision submodule obtains the data amount of the transmission data according to the address information, including: The encoding decision submodule obtains the address amount according to the start address and the end address; The data amount of the transmission data is obtained according to the data amount of each address and the address amount.

3. The method of claim 1, wherein, If it is determined that the transmission data does not need to be encoded based on the data amount of the transmission data, the method further includes: The transmission data is taken to a sending buffer area of the transmission logic according to the address information.

4. The method of claim 1, wherein, The encoding behavior information includes the data amount of an encoding batch and the amount of redundant data generated by encoding, and the encoded data includes redundant data and initial data.

5. A data transmission device based on a network card system, characterized by The network card system includes an encoding decision submodule, an encoding execution submodule, a transmission logic, and a decoding module, and the device includes: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to: when a first user sends transmission data, the network card system obtains a transmission command issued by the first user, parses address information of data storage according to the transmission command, and transmits the address information to the encoding decision submodule; The encoding decision submodule obtains the data amount of the transmission data according to the address information, determines the encoding behavior information of the transmission data if it is determined that the transmission data needs to be encoded based on the data amount of the transmission data, and calls a DMA controller to take the transmission data to an encoding cache area of the encoding execution submodule and transmit the encoding behavior information to the encoding execution submodule and the transmission logic respectively; The encoding execution submodule encodes the transmission data in the encoding cache area according to the encoding behavior information when it receives the encoding behavior information, and transmits the encoded data to a sending cache area of the transmission logic; The transmission logic distinguishes and encapsulates the unencoded incoming data and the encoded incoming data to obtain data packets, the data packets include unencoded data packets and encoded data packets, and for the encoded data packets, an encoding label carrying the encoding behavior information is marked in a packet header; When the second user receives the transmission data, the transmission logic performs anomaly detection on the received data packets, detects whether the encoding label is carried in the packet header of the data packets, obtains unencoded data packets that pass the detection and encoded data packets that pass the detection, stores the unencoded data packets that pass the detection in a corresponding position of a memory, and transmits the encoded data packets that pass the detection to a decoding module; The decoding module decodes the received encoded data packets that pass the detection to obtain decoded data, and stores the decoded data in a corresponding position of a memory; If it is determined that the transmission data needs to be encoded based on the data amount of the transmission data, the method further comprises: obtaining a current network card sending rate of the network card system; determining a round trip time (RTT) of the current network and a packet loss rate of the current network according to a current received data packet condition; judging whether the data amount of the transmission data needs network coding based on the to-be-sent data amount, the current network card sending rate, the RTT of the current network, and the packet loss rate of the current network; The judgment of whether the data amount of the transmission data needs network coding based on the to-be-sent data amount, the current network card sending rate, the RTT of the current network, and the packet loss rate of the current network comprises: setting a first threshold value and a second threshold value for the RTT and the packet loss rate of the network respectively; setting a third threshold value for a data transmission time; determining a current data transmission time according to the current network card sending rate and the data amount of the transmission data; judging whether the RTT of the current network meets the first threshold value, whether the packet loss rate of the current network meets the second threshold value, and whether the current data transmission time meets the third threshold value based on the to-be-sent data amount; The transmission logic performs anomaly detection on the received data packets, detects whether the encoding label is carried in the packet header of the data packets, and obtains unencoded data packets that pass the detection and encoded data packets that pass the detection, comprising: The transmission logic detects whether the encoding label is carried in the packet header of the data packets to obtain the unencoded data packets and the encoded data packets. The transmission logic performs packet loss detection and error detection on the uncoded data packet and the coded data packet, and obtains a passed uncoded data packet and a passed coded data packet; The transmission logic performs abnormality detection on the received data packet, and detects whether the coded label is carried in the packet header of the data packet, to obtain a passed uncoded data packet and a passed coded data packet, and stores the passed uncoded data packet in a corresponding position of the memory, and sends the passed coded data packet to the decoding module, including: The transmission logic performs packet loss detection and error detection on the uncoded data packet and the coded data packet; If the detection result is an uncoded error data packet and / or an uncoded out-of-order data packet, a retransmission notification is sent back; If the detection result is a coded error data packet, the coded data packet is discarded; If the detection result is a passed uncoded data packet, the uncoded data packet is unpacked, and the unpacked data is stored in a corresponding position of the memory; If the detection result is a passed coded data packet, when the number of the passed coded data packets reaches the decoding requirement, the passed coded data packets are sent to the decoding module.

6. A non-transitory computer storage medium, comprising, The network card system includes a coding decision submodule, a coding execution submodule, transmission logic, and a decoding module, and stores computer executable instructions. When the computer executable instructions are executed by a computer, the following functions can be realized: When a first user sends transmission data, the network card system obtains a transmission command issued by the first user, analyzes address information of data storage according to the transmission command, and transmits the address information to the coding decision submodule; The coding decision submodule obtains a data volume of the transmission data according to the address information. If it is determined that the transmission data needs to be coded based on the data volume of the transmission data, the coding behavior information of the transmission data is determined, and the DMA controller is called to take the transmission data to the coding cache area of the coding execution submodule, and the coding behavior information is transmitted to the coding execution submodule and the transmission logic respectively; When the coding execution submodule receives the coding behavior information, the transmission data in the coding cache area is coded according to the coding behavior information, and the coded data is transmitted to the sending cache area of the transmission logic; The transmission logic distinguishes and encapsulates uncoded incoming data and coded incoming data to obtain data packets, the data packets include uncoded data packets and coded data packets, and for the coded data packets, a coded label carrying the coding behavior information is marked in the packet header; When a second user receives the transmission data, the transmission logic performs abnormality detection on the received data packet, and detects whether the coded label is carried in the packet header of the data packet, to obtain a passed uncoded data packet and a passed coded data packet, and stores the passed uncoded data packet in a corresponding position of the memory, and sends the passed coded data packet to the decoding module; The decoding module decodes the received encoded data packet that passes the detection to obtain decoded data, and stores the decoded data in a corresponding position of the memory; If it is determined that the transmission data needs to be encoded based on the data amount of the transmission data, the method further includes: obtaining a current network card sending rate of the network card system; determining a round trip time (RTT) of the current network and a packet loss rate of the current network according to a current received data packet condition; determining whether the data amount of the transmission data needs network coding based on the to-be-sent data amount, the current network card sending rate, the RTT of the current network, and the packet loss rate of the current network; The determination whether the data amount of the transmission data needs network coding based on the to-be-sent data amount, the current network card sending rate, the RTT of the current network, and the packet loss rate of the current network includes: setting a first threshold value and a second threshold value for the RTT and the packet loss rate of the network respectively; setting a third threshold value for a data transmission time; determining a current data transmission time according to the current network card sending rate and the data amount of the transmission data; determining whether the RTT of the current network meets the first threshold value, whether the packet loss rate of the current network meets the second threshold value, and whether the current data transmission time meets the third threshold value based on the to-be-sent data amount; The transmission logic performs anomaly detection on the received data packet, and detects whether an encoding label is carried in a packet header of the data packet to obtain a non-encoded data packet that passes the detection and an encoded data packet that passes the detection, and includes: The transmission logic detects whether an encoding label is carried in a packet header of the data packet to obtain the non-encoded data packet and the encoded data packet; The transmission logic performs packet loss detection and error detection on the non-encoded data packet and the encoded data packet to obtain a non-encoded data packet that passes the detection and an encoded data packet that passes the detection; The transmission logic performs anomaly detection on the received data packet, and detects whether an encoding label is carried in a packet header of the data packet to obtain a non-encoded data packet that passes the detection and an encoded data packet that passes the detection, and includes: The transmission logic performs packet loss detection and error detection on the non-encoded data packet and the encoded data packet; If the detection result is an unencoded error data packet and / or an unencoded out-of-order data packet, a retransmission notification is sent back; If the detection result is an encoded error data packet, the encoded data packet is discarded; If the detection result is a non-encoded data packet that passes the detection, the non-encoded data packet is packet unpacked, and the unpacked data is stored in a corresponding position of the memory; If the detection result is an encoded data packet that passes the detection, when the number of the encoded data packets reaches a decoding requirement, the encoded data packet that passes the detection is sent to the decoding module.

Images