Data sending method and device, equipment and storage medium

Abstract

The invention discloses a data sending method, the method is applied to sending end equipment, and the method comprises the following steps: obtaining a network state parameter; dividing a plurality of to-be-sent data packets corresponding to a frame of data generated by precoding into a plurality of first data packet groups in response to the condition that the network state parameter satisfies the first state; the number of data packets in each first data packet group is a first number; dividing a plurality of to-be-sent data packets corresponding to one frame of data into a plurality of second data packet groups in response to the condition that the network state parameter meets a second state; the number of the data packets in each second data packet group is a second number, the network state represented by the first state is superior to the network state represented by the second state, and the first number is greater than the second number; performing redundant coding on at least one to-be-sent data packet in each data packet group to obtain a first redundant data packet corresponding to each data packet group; and sending the plurality of data packet groups and the first redundant data packet corresponding to each data packet group.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data sending method, device, equipment and storage medium. Background Art

[0002] In data transmission, in order to improve data reliability and fault tolerance, forward error correction technology is usually used to perform redundant encoding on data packets.

[0003] However, in the case of high packet loss rate, redundant calculations need to be performed on all data packets, which has high computational complexity, resulting in increased encoding time and easily causing lag problems at the receiving end. Summary of the Invention

[0004] The embodiments of the present application provide a data sending method, apparatus, device, and storage medium, which can reduce computational complexity and transmission delay.

[0005] In a first aspect, an embodiment of the present application provides a data sending method, which is applied to a sending device and includes:

[0006] Get network status parameters;

[0007] In response to the network state parameter satisfying the first state, dividing the plurality of to-be-transmitted data packets corresponding to the pre-encoded frame of data into a plurality of first data packet groups; the number of data packets in each first data packet group is the first number;

[0008] In response to the network state parameter satisfying a second state, dividing a plurality of to-be-sent data packets corresponding to a frame of data into a plurality of second data packet groups; the number of data packets in each second data packet group is a second number, the network state represented by the first state is better than the network state represented by the second state, and the first number is greater than the second number;

[0009] Redundantly encode at least one to-be-sent data packet in each data packet group to obtain a first redundant data packet corresponding to each data packet group;

[0010] A plurality of data packet groups and a first redundant data packet corresponding to each data packet group are sent.

[0011] In a second aspect, an embodiment of the present application provides a method for recovering from packet loss, which is applied to a receiving device and includes:

[0012] receiving a plurality of data packet groups and a first redundant data packet corresponding to each of the plurality of data packet groups;

[0013] parsing the successfully received first redundant data packet to determine a third number of data packets in the corresponding data packet group;

[0014] When the sum of the number of successfully received data packets in the data packet group and the number of successfully received first redundant data packets in each data packet group is greater than or equal to a third number, decoding the successfully received data packets based on the successfully received first redundant data packets to obtain recovered data packets;

[0015] Among them, the first redundant data packet is obtained by redundantly encoding at least one data packet to be sent in each data packet group; the number of data packets in each data packet group is a first number or a second number, and the first number is greater than the second number; the first number is related to a first state of a network state parameter, and the second number is related to a second state of the network state parameter, and the network state represented by the first state is better than the network state represented by the second state.

[0016] In a third aspect, an embodiment of the present application provides a data sending device, which is applied to a sending end device. The data sending device includes:

[0017] Acquisition module, used to obtain network status parameters;

[0018] A first dividing module is configured to divide a plurality of to-be-transmitted data packets corresponding to a frame of data generated by precoding into a plurality of first data packet groups in response to a network state parameter satisfying a first state; the number of data packets in each first data packet group is a first number;

[0019] a second dividing module, configured to divide a plurality of to-be-sent data packets corresponding to a frame of data into a plurality of second data packet groups in response to the network state parameter satisfying a second state; the number of data packets in each second data packet group being a second number, the network state represented by the first state being better than the network state represented by the second state, and the first number being greater than the second number;

[0020] an encoding module, configured to perform redundant encoding on at least one to-be-sent data packet in each data packet group to obtain a first redundant data packet corresponding to each data packet group;

[0021] The sending module is used to send multiple data packet groups and a first redundant data packet corresponding to each data packet group.

[0022] In a fourth aspect, an embodiment of the present application provides a device for recovering data packet loss, the device being applied to a receiving device, and comprising:

[0023] A receiving module, configured to receive a plurality of data packet groups and a first redundant data packet corresponding to each of the plurality of data packet groups;

[0024] a determination module, configured to parse the successfully received first redundant data packet and determine a third number of data packets in the corresponding data packet group;

[0025] a decoding module, configured to, when the sum of the number of successfully received data packets in the data packet group and the number of successfully received first redundant data packets in each data packet group is greater than or equal to a third number, decode the successfully received data packets based on the successfully received first redundant data packets to obtain recovered data packets;

[0026] Among them, the first redundant data packet is obtained by redundantly encoding at least one data packet to be sent in each data packet group; the number of data packets in each data packet group is a first number or a second number, and the first number is greater than the second number; the first number is related to a first state of a network state parameter, and the second number is related to a second state of the network state parameter, and the network state represented by the first state is better than the network state represented by the second state.

[0027] In a fifth aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory storing instructions executable by the processor. When the instructions are executed by the processor, the method of the first or second aspect above is implemented.

[0028] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the method of the first or second aspect described above is implemented.

[0029] The embodiments of the present application provide a data transmission method, apparatus, device, and storage medium, wherein a transmitting end device obtains network status parameters; in response to the network status parameters satisfying a first state, multiple data packets to be transmitted corresponding to a frame of data generated through precoding are divided into multiple first data packet groups; in response to the network status parameters satisfying a second state, multiple data packets to be transmitted corresponding to a frame of data are divided into multiple second data packet groups; redundant encoding is performed on at least one data packet to be transmitted in each data packet group to obtain a first redundant data packet corresponding to each data packet group; and multiple data packet groups and the first redundant data packet corresponding to each data packet group are transmitted. In this way, the transmitting end device adjusts the data packet grouping size by dynamically adapting to the network status, adopts a larger group for redundant encoding when the network quality is good to reduce computing overhead, and reduces the group size when the network quality is poor to improve the timeliness of data recovery, thereby optimizing coding efficiency while ensuring error correction capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 Schematic diagram of the implementation process of the data sending method provided in the embodiment of the application Figure 1 ;

[0031] Figure 2 Schematic diagram of the format of the data packet to be sent provided in the embodiment of the application Figure 1 ;

[0032] Figure 3Schematic diagram of the format of the data packet to be sent provided in the embodiment of the application Figure 2 ;

[0033] Figure 4 Schematic diagram of data packet redundancy coding provided in an embodiment of the present application;

[0034] Figure 5 A schematic diagram of the format of a first redundant data packet provided in an embodiment of the present application;

[0035] Figure 6 Schematic diagram of the implementation process of the data sending method provided in the embodiment of the application Figure 2 ;

[0036] Figure 7 A schematic diagram of the structure of multi-way nested redundancy provided in an embodiment of the present application;

[0037] Figure 8 A schematic diagram of a process flow for implementing a method for recovering from packet loss provided in an embodiment of the present application;

[0038] Figure 9 A schematic diagram of data packet decoding provided in an embodiment of the present application;

[0039] Figure 10 A schematic diagram of the decoding process provided in an embodiment of the present application;

[0040] Figure 11 A schematic diagram of the structure of a data transmission device according to an embodiment of the present application;

[0041] Figure 12 A schematic diagram of the structure of a device for recovering data packet loss according to an embodiment of the present application;

[0042] Figure 13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. It should be understood that the specific embodiments described herein are only used to explain the different applications and are not intended to limit the application. It should also be noted that for ease of description, the drawings only show the parts that differ from the related applications.

[0044] In real-time audio and video transmission, forward error correction (FEC) technology is often used to redundancy encode data packets to improve data reliability and fault tolerance. This technology generates additional redundant data packets at the sender, allowing the receiver to recover the original data even if some packets are lost, thereby improving transmission quality.

[0045] In related technologies, data redundancy coding based on the Reed-Solomon Forward Error Correction (RS-FEC) algorithm is widely used. This method uses the Vandermonde reversible matrix to perform polynomial operations and encodes according to the number of original data packets and the number of redundant packets. However, in the case of high packet loss rate, RS-FEC needs to perform redundant calculations on all data packets. Its computational complexity is O(m×n), where m is the number of original packets and n is the number of redundant packets. This increases the encoding time and occupies a large amount of CPU resources. In real-time streaming scenarios, it is easy to cause problems such as freezing at the receiving end. Especially when processing large-size data frames (such as I frames), if a full redundant encoding strategy is adopted, it will further increase the system burden and limit its application effect in low-latency, high-concurrency scenarios.

[0046] For example, for a 150KB I-frame, if each packet size is 1500 bytes, 100 RTP packets are required. When all are redundant, the computational complexity is O(100×100), which takes a very long time to encode, resulting in serious waste of CPU resources and playback lag.

[0047] To address the aforementioned issues, embodiments of the present application provide a data transmission method, apparatus, device, and storage medium. A transmitting end determines whether it is in a first state or a second state based on current network state parameters, and accordingly divides data packets into different numbers of data packet groups. Subsequently, at least one to-be-sent data packet in each data packet group is redundantly encoded to generate a first redundant data packet, which is then sent along with the original data packet. In this way, the transmitting end device dynamically adapts to the network state to adjust the data packet grouping size. When network quality is good, larger groups are used for redundant encoding to reduce computational overhead, while when network quality is poor, the grouping size is reduced to improve the timeliness of data recovery, thereby optimizing coding efficiency while ensuring error correction capabilities.

[0048] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application.

[0049] In the embodiments of this application, Figure 1 Schematic diagram of the implementation process of the data sending method provided in the embodiment of the application Figure 1 , the data sending method can be applied to the sending end device, such as Figure 1 As shown, the data sending method may include the following steps:

[0050] Step 101: Obtain network status parameters.

[0051] The network status parameter is used to indicate the parameter reflecting the current network status obtained by the sending end device before or during data transmission.

[0052] For example, network status parameters may include but are not limited to bandwidth, packet loss rate, delay, etc.

[0053] In some embodiments, the network status parameters may be acquired by the transmitting device through real-time monitoring or from historical data cached locally, which is not limited in the embodiments of the present application.

[0054] Exemplarily, the sending device sends a test packet (such as Ping, UDP probe) to measure the round-trip time (RTT) and packet loss rate; or, the sending device calculates historical data statistics based on local cache (such as TCP RTT sampling).

[0055] Step 102: In response to the network state parameter satisfying a first state, a plurality of to-be-sent data packets corresponding to a frame of pre-coded data are divided into a plurality of first data packet groups.

[0056] The number of data packets in each first data packet group is a first number.

[0057] It is understood that the network status parameters satisfying the first state may include one or more of the following: the bandwidth in the network status parameters is greater than a preset bandwidth; the latency in the network status parameters is less than a preset latency; and the packet loss rate in the network status parameters is less than a preset packet loss rate. In other words, the network status parameters satisfying the first state can be used to indicate good network quality.

[0058] As another example, to determine whether the network status satisfies the first status, the bandwidth, delay, and packet loss rate can be subjected to fuzzy logic or weighted averaging to obtain a quantitative result, and compared with a preset threshold. If it is higher than the preset threshold, it is determined that the first status is satisfied; if it is lower than or equal to the preset threshold, it is determined that the second status is not satisfied.

[0059] In some embodiments, a frame of data generated through precoding refers to a process where the transmitter pre-codes the original data before transmission to generate a data unit suitable for transmission. For example, the data may be audio data, video data, sensor data, text data, etc., which is not limited in the present embodiments.

[0060] For example, if the data is video data, an image frame (I frame / P frame) encoded by H.264 / HEVC can be segmented into multiple RTP packets. If the data is audio data, an audio frame (e.g., 20ms data) encoded by Opus can be segmented into multiple UDP packets.

[0061] It can also be understood that one frame of data may contain multiple data packets to be sent, that is, one frame of data after precoding may include multiple RTP packets.

[0062] In some embodiments, after obtaining the network status parameters, the sending device can determine whether the network status parameters meet the first state. If the first state is met, that is, the network state is good, the sending device will divide all the data packets to be sent after a frame of data is segmented into multiple first data packet groups, each group containing a first number of data packets to be sent.

[0063] It should be noted that the first quantity can be calculated based on the quantization results of the network status parameters, or can be set by the developer, and this is not limited in the embodiments of the present application.

[0064] It should be understood that if the number of data packets in a frame of data is less than the first number, or the number of data packets to be sent in the last first data packet group of a frame of data is less than the first number, they can be grouped separately or filled with invalid packets.

[0065] In some embodiments, a fixed-size data packet format can be used. For example, the size of the RTP packet is repackaged into 1408 bytes (maximum transmission unit <1500), and the insufficient part is filled with 0. Subsequent processing (RS-FEC encoding and decoding) is performed on the entire packet data unit to avoid delays caused by the mismatch between bytes and packets after decoding and recovery.

[0066] For example, Figure 2 Schematic diagram of the format of the data packet to be sent provided in the embodiment of the application Figure 1 .from Figure 2 It can be seen that the data packet may include a 2-byte payload length (payload-size), a payload (payload field), and a padding portion (Padding).

[0067] As another example, Figure 3 Schematic diagram of the format of the data packet to be sent provided in the embodiment of the application Figure 2 .from Figure 3 It can be seen that the data packet may include a 2-byte RTP header and the total length of the payload (rtp-header-and-payload-total-size), an RTP header (rtp-header), a payload (payload field), and a padding part (Padding).

[0068] Step 103: In response to the network status parameter satisfying the second status, a plurality of to-be-sent data packets corresponding to a frame of data are divided into a plurality of second data packet groups.

[0069] The number of data packets in each second data packet group is a second number.

[0070] It is understood that the network status parameter meeting the second state may include one or more of the following: the bandwidth in the network status parameter is less than a preset bandwidth; the delay in the network status parameter is greater than a preset delay; or the packet loss rate in the network status parameter is greater than a preset packet loss rate. In other words, the network status parameter meeting the second state can be used to indicate poor network quality.

[0071] In other words, the network state represented by the first state is better than the network state represented by the second state.

[0072] In some embodiments, after obtaining the network status parameters, the sending device can determine whether the network status parameters meet the second state. If it does not meet the first state but meets the second state, that is, the network state is poor, the sending device will divide all the data packets to be sent after a frame of data is segmented into multiple second data packet groups, each group containing a second number of data packets to be sent.

[0073] It should be noted that the first number is greater than the second number. That is, when the network status is poor, to improve the success rate of data recovery, the transmitting device may divide the same frame of data into more groups, each of which contains the second number of data packets to be sent. When the network status is good, the transmitting device may divide the same frame of data into fewer groups, each of which contains the first number of data packets to be sent, thereby reducing redundancy overhead and improving transmission efficiency.

[0074] For example, in the case of a high packet loss rate, that is, when the network state parameters meet the second state, each group may contain only two data packets to be sent, thereby further reducing the redundant coding complexity of a single group and thus improving decoding efficiency. For another example, in the case of a good network state, that is, when the network state parameters meet the first state, each group may contain four data packets to be sent, thereby reducing redundancy overhead and improving transmission efficiency.

[0075] It should also be noted that the second number can be calculated based on the quantization results of the network status parameters, or can be set by the developer, which is not limited in the embodiments of the present application.

[0076] It should be understood that if the number of data packets in a frame of data is less than the second number, or the number of data packets to be sent in the last second data packet group of a frame of data is less than the second number, they can be grouped separately or filled with invalid packets.

[0077] Step 104: Redundancy encoding is performed on at least one to-be-sent data packet in each data packet group to obtain a first redundant data packet corresponding to each data packet group.

[0078] Step 105: Send multiple data packet groups and a first redundant data packet corresponding to each data packet group.

[0079] In some embodiments, after completing the data packet grouping, the transmitting device performs redundant encoding on at least one data packet in each data packet group to generate a corresponding first redundant data packet. After completing the data packet grouping and redundant encoding, the transmitting device transmits the original data packet group and the first redundant data packet to the receiving end.

[0080] Redundant coding can generate additional redundant data packets by performing mathematical operations on the original data packets, allowing the receiver to recover the original data without retransmission even if some data packets are lost. For example, redundant coding methods include but are not limited to the RS-FEC algorithm and XOR coding.

[0081] The first redundant data packet is generated through redundant coding and is used to protect a data packet group. Its function is to help the receiver recover the lost data when some original data packets in the group are lost.

[0082] It can be understood that the FEC encoding parameters can be recorded as (N1, K1), where N1 is the data packet in the data packet group and K1 is the first redundant data packet.

[0083] It should be noted that redundant data packets can also be called verification data packets and error correction data packets. The three can be equivalent or interchangeable, and this is not limited in the embodiments of the present application.

[0084] It should also be noted that the number of redundant data packets generated can be dynamically adjusted based on network conditions. For example, when network conditions are poor, the number of redundant data packets can be increased to improve data recovery capabilities; when network conditions are good, the number of redundant data packets can be reduced to conserve bandwidth resources.

[0085] For example, Figure 4 Schematic diagram of redundant encoding of data packets provided in the embodiment of the present application. Figure 4 As shown, the transmitting end device may perform redundant encoding (eg, RS-encoding) on ​​data packet 1 , data packet 2 , data packet 3 , and data packet 4 to obtain a first redundant data packet 1 and a first redundant data packet 2 .

[0086] It should be noted that the sizes of the above data packets 1, 2, 3 and 4 are all 1048 bytes, and the sizes of the first redundant data packets 1 and 2 generated at the same time are also 1048 bytes.

[0087] As another example, Figure 5 Schematic diagram of the format of the first redundant data packet provided in the embodiment of the present application. Figure 5As shown, the first redundant data packet may include an RTP header (rtp-header), an FEC header (protected_packets_masksFEC-header), and a payload (parity-packet-payload) of the first redundant data packet. The payload of the first redundant data packet has a size of 1048 bytes.

[0088] An embodiment of the present application provides a data transmission method, wherein a transmitting end determines whether it is in a first state or a second state based on current network state parameters, and accordingly divides data packets into different numbers of data packet groups; then, redundantly encoding at least one to-be-sent data packet in each data packet group generates a first redundant data packet, and transmits the first redundant data packet along with the original data packet during transmission. In this way, the transmitting end device adjusts the data packet grouping size by dynamically adapting to the network state, using larger groups for redundant encoding to reduce computational overhead when network quality is good, and reducing the grouping size to improve the timeliness of data recovery when network quality is poor, thereby optimizing coding efficiency while ensuring error correction capabilities.

[0089] In some embodiments, the first redundant data packet may include first configuration information, and the first configuration information may be used by the receiving end to determine the number of data packets in the data packet group corresponding to the first redundant data packet.

[0090] The first configuration information refers to information added to the first redundant data packet.

[0091] Exemplarily, the first configuration information may be a field; or, the first configuration information may also be mask information (mask), which is not limited in the embodiments of the present application.

[0092] In one embodiment, if the first configuration information is field information, the number of data packets in the data packet group can be explicitly indicated by the field. Assuming that the size of the field is 3 bits, if the first configuration information is 101, it indicates that there are 5 data packets in the data packet group. If the first configuration information is 011, it indicates that there are 3 data packets in the data packet group.

[0093] In another embodiment, if the first configuration information is mask information, the number of packets in the packet group can be implicitly represented by a bitmap. Assuming the mask field is M bits long, each bit (bit_i) indicates whether the packet group contains a packet with a specific sequence number. The packet group uses sequence_number_base as the starting sequence number. If bit_i is 1, it indicates that the redundant packet protects the packet with sequence number sequence_number_base+i. In other words, the original number of packets in the packet group equals the number of 1s in the mask.

[0094] For example, assuming sequence_number_base = 100, mask (i.e., first configuration information) = 0b1101 (binary), bit_0 = 1, meaning the packet with sequence number 100+0 = 100 is protected; bit_1 = 1, meaning the packet with sequence number 100+1 = 101 is protected; bit_2 = 0, meaning the packet with sequence number 102 is not protected; and bit_3 = 1, meaning the packet with sequence number 100+3 = 103 is protected. In other words, the number of packets in this packet group is 3 (i.e., 100, 101, 103).

[0095] It should also be noted that the first configuration information may also include information such as an identifier of the data packet group, a timestamp, a data packet sequence number, etc., which is not limited in the embodiments of the present application.

[0096] In some embodiments, each data packet group corresponds to multiple first redundant data packets, and the first configuration information of the multiple first redundant data packets is the same, so that the receiving end can determine the number of data packets in the data packet group based on any received first redundant data packet.

[0097] It is understood that, in order to improve decoding efficiency and reduce computational complexity, the number of first redundant data packets corresponding to each data packet group is set to multiple, rather than one or a fixed number. This improves system fault tolerance. Even if some first redundant data packets are lost, data recovery can still be completed using other first redundant data packets.

[0098] It is also understood that the transmitting device may add the same first configuration information to the headers of multiple first redundant data packets. That is, in a data packet group, all first redundant data packets carry the same configuration information. This allows the receiving end to determine the total number of data packets in the group after receiving any first redundant data packet.

[0099] In an embodiment of the present application, by setting multiple first redundant data packets and using the same first configuration information, the receiving end can immediately determine the size of the data packet group after receiving any first redundant data packet, so that the decoding process can be started in advance, thereby reducing the decoding delay.

[0100] In some embodiments, Figure 6 Schematic diagram of the implementation process of the data sending method provided in the embodiment of the application Figure 2 For the above step 104, redundant encoding is performed on at least one data packet to be sent in each data packet group to obtain the first redundant data packet corresponding to each data packet group, and the following steps may be further included:

[0101] Step 601: Determine the packet loss rate of data packets within a first historical duration.

[0102] The first historical duration refers to the length of the time window used to count network packet loss, which can be a configurable time period from the last few seconds to tens of seconds. For example, it can be set to count packet loss within the past 1 second, 5 seconds, or 10 seconds.

[0103] The packet loss rate refers to the ratio of the number of data packets lost in the first historical period to the total number of data packets sent, expressed as a percentage.

[0104] It is understood that the packet loss rate can reflect the current network transmission quality to a certain extent. For example, if the packet loss rate is high (e.g., over 30%), the number of first redundant data packets needs to be increased to improve the recovery success rate. Conversely, if the packet loss rate is low, the number of first redundant data packets can be appropriately reduced to save bandwidth resources.

[0105] Step 602: When a first preset condition is met, determine a first redundancy rate according to the packet loss rate.

[0106] The first redundancy rate refers to the ratio of the first redundant data packets generated by the sending end device to the original data packets in order to ensure that the data packet group can be successfully restored under the current network conditions.

[0107] It is understandable that the first redundancy rate is related to the number of first redundant data packets. That is, the higher the first redundancy rate, the more first redundant data packets, the stronger the recovery capability, but also consumes more bandwidth resources.

[0108] For example, if the first redundancy rate is 20%, each group of 4 original data packets will generate 1 first redundant data packet; if the first redundancy rate is 50%, each group of 4 original data packets will generate 2 first redundant data packets.

[0109] In some embodiments, the first preset condition may be that the actual bandwidth does not exceed the predicted available bandwidth; the actual bandwidth is determined based on the packet loss rate and the data packet encoding rate; the predicted available bandwidth is the predicted value of the available bandwidth in the current network transmission; the predicted available bandwidth is determined based on the network status parameters.

[0110] The actual bandwidth is the actual bandwidth occupied by the data packets in the current network transmission; that is, the bandwidth resources actually occupied by all the data packets being sent in the current network transmission process.

[0111] The predicted available bandwidth is the predicted value of the available bandwidth in the current network transmission, that is, the system estimates the maximum bandwidth that can be provided in the future based on the current network status parameters.

[0112] In one embodiment, the actual bandwidth may be determined based on the packet loss rate and the data packet encoding rate.

[0113] The packet encoding bitrate is the bit rate used when encoding a video or audio stream, which determines the amount of information carried in each packet. A higher packet encoding bitrate means higher image or sound quality, but also requires greater bandwidth. For example, in real-time video transmission, if a packet encoding bitrate of 1000kbps is used, 1000Kb of data must be transmitted per second.

[0114] For example, the calculation formula of the actual bandwidth in the embodiment of the present application can be expressed as:

[0115] actual_bandwidth=(1+loss_rate)×video_bitrate (1)

[0116] Among them, actual_bandwidth is the actual bandwidth; loss_rate is the packet loss rate of the data packet in the first historical length; video_bitrate is the data packet encoding rate.

[0117] Of course, the calculation formula for the actual bandwidth can also be obtained by modifying the above formula (1), and the embodiment of the present application does not limit this.

[0118] In one embodiment, the predicted available bandwidth may be determined based on network status parameters.

[0119] Exemplarily, the transmitting device may determine the current network status according to the time interval between sending and receiving packets, and adjust the bandwidth prediction value to obtain the predicted available bandwidth.

[0120] For example, the transmitting device can determine the interval difference between adjacent sent packets. After one RRT, the transmitting device can determine the interval difference between adjacent received packets, and then use the difference between the two interval differences to perform trendline filtering. The trend line can identify the changing trend of network conditions. For example, if the interval difference between received packets continues to increase, it may indicate that network congestion is increasing; conversely, if the interval difference between received packets decreases, it may indicate that the network condition is improving. Based on the results of the network condition judgment, adjustments are made based on the last predicted bandwidth: the adjustment methods include increasing or decreasing the bandwidth. That is, if the network condition is good and the bandwidth utilization is high, the predicted bandwidth can be appropriately increased to improve data transmission efficiency; if the network congestion is serious and the bandwidth utilization is low, the predicted bandwidth should be reduced to avoid further increasing the network burden.

[0121] For example, the formula of the first preset condition in the embodiment of the present application can be expressed as:

[0122] (1+loss_rate)×video_bitrate<=estimated_bandwidth (2)

[0123] Among them, estimated_bandwidth is the predicted available bandwidth.

[0124] Of course, the formula of the first preset condition can also be obtained by modifying the above formula (2), and the embodiment of the present application does not limit this.

[0125] In some embodiments, after determining the actual bandwidth and the predicted available bandwidth, the receiving device can determine whether the actual bandwidth is less than or equal to the predicted available bandwidth. If so, the current packet loss rate is used directly as the first redundancy rate. If the actual bandwidth is greater than the predicted available bandwidth, the transmitting device can reduce the packet encoding rate to ensure redundancy and avoid bandwidth overload and lag.

[0126] Step 603: Based on the first redundancy rate, perform redundancy coding on at least one to-be-sent data packet in each data packet group to obtain a first redundant data packet corresponding to each data packet group.

[0127] In this embodiment, the transmitting device may perform redundant encoding on at least one to-be-transmitted data packet in each data packet group based on the calculated first redundancy rate to generate a corresponding first redundant data packet. Furthermore, the transmitting device may transmit the data packet group and the first redundant data packet together to the receiving end.

[0128] In an embodiment of the present application, by dynamically monitoring the packet loss rate of data packets within a first historical period, the first redundancy rate is adjusted in real time based on the packet loss rate when a first preset condition is met, thereby avoiding the disadvantages of using a fixed redundancy rate. If the redundancy rate is too high, bandwidth will be wasted, while if the redundancy rate is too low, the original data cannot be restored. Furthermore, the terminal device can perform redundant encoding on the data packet group based on the first redundancy rate. In this way, the current network conditions can be more accurately matched, thereby optimizing bandwidth utilization and data recovery efficiency, and significantly reducing the jamming rate and end-to-end latency.

[0129] It should be noted that in the current FEC protection strategy, when the packet loss rate is high (greater than 30%), the first redundant data packet of FEC can only be partially received, which will make the original data packet unrecoverable. Therefore, in order to maximize the recovery of lost data in a far-field high packet loss and weak network environment, the first redundant data packet can be further redundantly protected.

[0130] Based on this, in some embodiments, after step 104, the method further includes:

[0131] Redundancy encoding is performed on the first redundant data packet corresponding to each data packet group to obtain a second redundant data packet corresponding to the first redundant data packet.

[0132] Correspondingly, for the above step 105, the process of sending multiple data packet groups and the first redundant data packet corresponding to each data packet group further includes: sending multiple data packet groups, the first redundant data packet corresponding to each data packet group, and the second redundant data packet corresponding to the first redundant data packet.

[0133] The second redundant data packet refers to a group of check packets generated by the first redundant data packet through a redundant coding algorithm. The second redundant data packet is used to restore the first redundant data packet when a large number of the first redundant data packets are lost, thereby completing the recovery of the original data packet.

[0134] It can be understood that the FEC encoding parameters can be recorded as (K1, R2), where K1 is the first redundant data packet and R2 is the second redundant data packet.

[0135] For example, during video streaming, if a key frame (such as an I-frame) is split into multiple RTP packets and the network experiences a packet loss rate exceeding 50%, traditional single-layer redundant FEC may not be able to effectively recover all the original packets. However, by introducing a second redundant packet, the system can achieve a higher probability of recovery within a limited time, thereby reducing the rate of interruptions and end-to-end latency.

[0136] In addition, the generation of the second redundant data packets does not significantly increase the overall bandwidth overhead because the number of the second redundant data packets is usually much smaller than the number of the original data packets, which ensures the reliability of data recovery and avoids the waste of bandwidth resources.

[0137] For example, Figure 7 A schematic diagram of the structure of multi-way nested redundancy provided in an embodiment of the present application; Figure 7 As shown, data packets S1-S5 are multiple data packets in a data packet group; the first redundant data packets R1-R3 are the first redundant data packets corresponding to the data packet group; the second redundant data packets RR1 and RR2 are the second redundant data packets corresponding to the first redundant data packet in the data packet group.

[0138] In some embodiments, after generating the first redundant data packet and the second redundant data packet, the transmitting device may send multiple data packet groups, the first redundant data packet corresponding to each data packet group, and the second redundant data packet corresponding to the first redundant data packet to the receiving device.

[0139] In the embodiment of the present application, the second redundant data packet is obtained by redundancy encoding the first redundant data packet again. This can further improve the probability of data recovery in a high packet loss environment, thereby shortening the time required for data recovery, significantly reducing the jam rate and end-to-end latency, and improving the user experience.

[0140] In some embodiments, the second redundant data packet includes second configuration information, and the second configuration information is used by the receiving end to determine the number of redundant data packets in the first redundant data packet corresponding to the second redundant data packet.

[0141] The second configuration information refers to information added to the second redundant data packet.

[0142] Exemplarily, the second configuration information may be a field; or, the second configuration information may also be mask information (mask), which is not limited in the embodiments of the present application.

[0143] In one embodiment, if the second configuration information is field information, the number of redundant data packets in the first redundant data packet can be explicitly indicated by the field. Assuming that the size of the field is 3 bits, if the second configuration information is 001, it indicates that there is one redundant data packet in the data packet group. If the second configuration information is 011, it indicates that there are three redundant data packets in the data packet group.

[0144] In another embodiment, if the second configuration information is mask information, the number of redundant data packets in the first redundant data packet can be implicitly represented by a bitmap. Assuming that the length of the mask field is N bits, the number of first redundant data packets in the data packet group = the number of 1s in the mask.

[0145] In the embodiment of the present application, by adding the second configuration information to the second redundant data packet, the receiving end can quickly determine the required number of first redundant data packets, thereby reducing the waiting time required for redundancy recovery and improving encoding and decoding efficiency.

[0146] In some embodiments, the first redundant data packet corresponds to multiple second redundant data packets, and the second configuration information of the multiple second redundant data packets is the same, so that the receiving end can determine the number of redundant data packets in the first redundant data packet based on any received second redundant data packet.

[0147] It is understood that, to improve decoding efficiency and reduce computational complexity, the number of second redundant data packets corresponding to a first redundant data packet is multiple, rather than just one or a fixed number. This improves system fault tolerance. Even if some second redundant data packets are lost, the first redundant data packets can still be recovered using the remaining second redundant data packets.

[0148] It is also understood that the transmitting device may add the same second configuration information to the headers of multiple second redundant data packets. That is, in a data packet group, all second redundant data packets carry the same configuration information. This allows the receiving end to determine the total number of first redundant data packets in the group after receiving any second redundant data packet.

[0149] In an embodiment of the present application, by setting multiple second redundant data packets and using the same second configuration information, the receiving end can immediately determine the size of the first redundant data packet after receiving any second redundant data packet, so that the first redundant data packet can be recovered when the packet loss rate is high, thereby reducing the decoding delay.

[0150] In some embodiments, the process of performing redundant encoding on the first redundant data packet corresponding to each data packet group to obtain a second redundant data packet corresponding to the first redundant data packet may further include the following steps:

[0151] In some embodiments, a recovery rate of data packets within a second historical duration is determined.

[0152] The second historical duration refers to a time window length set by the system, which is used to count the data packet recovery status that has been completed within this time period.

[0153] It should be noted that the second historical duration can be dynamically adjusted or set by the system, and this is not limited in the embodiments of the present application.

[0154] The recovery rate refers to the ratio of the number of data packets successfully recovered within the second historical time period to the total number of data packets that should be recovered.

[0155] For example, the calculation formula of the recovery rate can be expressed as:

[0156] recovery-rate=recovered_packets / total-lost-packets×100% (3)

[0157] Among them, recovery-rate is the recovery rate, recovered_packets is the successfully recovered data packets; total-lost-packets is the total number of lost data packets, that is, the data packets that should be recovered.

[0158] It should be noted that when the sum of the number of received original data packets and the number of received first redundant data packets is greater than or equal to the total number of data packets, it is recorded as a successful recovery.

[0159] In some embodiments, the second redundancy rate is determined based on the recovery rate.

[0160] The second redundancy rate refers to the percentage of the second redundant data packets generated by the sending end device to the first redundant data packets in the current network environment.

[0161] It is understandable that the second redundancy rate is related to the number of second redundant data packets. That is, the higher the second redundancy rate, the more second redundant data packets, the stronger the recovery capability, but also consumes more bandwidth resources.

[0162] For example, if the second redundancy rate is 25%, each group of four first redundant data packets will generate one second redundant data packet; if the second redundancy rate is 50%, each group of four first redundant data packets will generate two second redundant data packets.

[0163] In some embodiments, the second redundancy rate is related to a recovery rate.

[0164] Exemplarily, the calculation formula of the second redundancy rate can be expressed as:

[0165] redundancy_rate=1-recovery_rate (4)

[0166] Here, redundancy_rate is the second redundancy rate.

[0167] Furthermore, for each data packet group, the calculation formula for the number of second redundant data packets can be expressed as:

[0168] w=(1-recovery_rate)×K1 (5)

[0169] Wherein, w represents the number of second redundant data packets; K1 represents the number of first redundant data packets in the data packet group.

[0170] In some embodiments, based on the second redundancy rate, redundant encoding is performed on the first redundant data packet corresponding to each data packet group to obtain a second redundant data packet corresponding to the first redundant data packet.

[0171] The redundant coding refers to processing the first redundant data packets in each group of data packets according to the currently determined second redundant rate to generate a certain number of second redundant data packets.

[0172] Through the above embodiment, the second redundancy rate is dynamically adjusted according to the historical recovery rate, and the redundancy strength can be flexibly configured under different network conditions, which not only avoids the waste of redundant resources but also improves the data recovery capability in high packet loss scenarios.

[0173] In the embodiments of this application, Figure 8 The embodiment of the present invention provides a method for recovering data packet loss. The method for recovering data packet loss can be applied to a receiving device, such as Figure 8 As shown, the method for recovering from data packet loss may include the following steps:

[0174] Step 801: Receive multiple data packet groups and a first redundant data packet corresponding to each of the multiple data packet groups.

[0175] It is understood that after the transmitting device groups multiple to-be-transmitted data packets corresponding to a frame of precoded data based on the network state parameter, it may perform redundant encoding on at least one to-be-transmitted data packet in each data packet group to obtain a first redundant data packet corresponding to each data packet group. The transmitting device transmits the multiple data packet groups and the first redundant data packet corresponding to each of the multiple data packet groups to the receiving device.

[0176] Accordingly, the receiving device can receive multiple data packet groups and the first redundant data packet corresponding to each of the multiple data packet groups.

[0177] The number of data packets in each data packet group is a first number or a second number, the first number is greater than the second number, the first number is related to a first state of the network state parameter, the second number is related to a second state of the network state parameter, and the network state represented by the first state is better than the network state represented by the second state.

[0178] It should be noted that the relevant contents of the data packet group and the first redundant data packet in step 801 have been described in detail in the above-mentioned application embodiments, and for the sake of brevity, they will not be repeated here.

[0179] Step 802: parse the successfully received first redundant data packet to determine a third number of data packets in the corresponding data packet group.

[0180] It is understandable that during the data packet transmission process, data packet loss may occur due to poor network conditions or other reasons.

[0181] Based on this, in some embodiments, the receiving device may parse the first configuration information in the first redundant data packet upon receiving the first redundant data packet, thereby determining the number of data packets in the data packet group corresponding to the first redundant data packet.

[0182] It can be seen from the above embodiments that the first configuration information in the first redundant data packet may be a field; or the first configuration information may also be mask information.

[0183] Exemplarily, if the first configuration information is field information, and the field information obtained by the receiving device after parsing the first configuration information is 101, the receiving device can determine that there are a total of 5 data packets in the data packet group, that is, the third number is 5.

[0184] In another example, if the first configuration information is mask, and the field information obtained by the receiving device after parsing the first configuration information is 11010001, the receiving device can determine that there are a total of 4 data packets in the data packet group, that is, the third number is 4.

[0185] Step 803: When the sum of the number of successfully received data packets in the data packet group and the number of successfully received first redundant data packets in each data packet group is greater than or equal to a third number, decode the successfully received data packets based on the successfully received first redundant data packets to obtain recovered data packets.

[0186] It is understandable that due to transmission over unreliable networks (such as UDP or wireless channels), some data packets may be lost. The receiving device does not need to receive all first redundant data packets to recover the original data packets. In other words, the receiving device only needs to ensure that the sum of the number of successfully received data packets and the number of successfully received first redundant data packets in each data packet group is greater than or equal to the third number to recover all data packets.

[0187] For example, if there are 4 data packets in a data packet group, it can be expressed as [P1, P2, P3, P4]; based on the data packet group, 2 first redundant data packets are generated, which can be expressed as [R1, R2], that is, the data packet sequence finally sent by the sending device is [P1, P2, P3, P4, R1, R2].

[0188] Furthermore, the current situation of the receiving device receiving data packets is: three data packets P1, P2, and P4 are successfully received, and one first redundant data packet R1 is successfully received. At this time, the receiving device can parse R1 and determine that the number of data packets is 4; the receiving device determines that the sum of the number of successfully received data packets and the number of successfully received first redundant data packets in each data packet group is 4, which meets the recovery conditions. At this time, the receiving device does not need to continue waiting for the next first redundant data packet, but can directly recover the data packets based on P1, P2, P4, and R1, so as to obtain the recovered P3.

[0189] For example, Figure 9 A schematic diagram of data packet decoding provided in an embodiment of the present application is provided. Figure 9 The process of RS-FEC decoding and recovering lost data packets under 50% redundancy rate (2 first redundant data packets, 4 data packets) is shown. Figure 9As shown in the figure, when data packet 2 (RTP-packet-2) and data packet 4 (RTP-packet-4) are set to zero due to packet loss, the receiving end uses the received data packet 1 (RTP-packet-1) and data packet 3 (RTP-packet-3) and the first redundant data packet 1 (parity-packet-1) and the first redundant data packet 2 (parity-packet-2) to reconstruct the lost two data packets (that is, data packet 2 and data packet 4) through the RS decoding algorithm.

[0190] An embodiment of the present application proposes a method for recovering from packet loss, comprising: receiving multiple packet groups and a first redundant packet corresponding to each of the multiple packet groups; parsing the successfully received first redundant packet to determine a third number of packets in the corresponding packet group; and, if the sum of the number of successfully received packets in the packet group and the number of successfully received first redundant packets in each packet group is greater than or equal to the third number, decoding the successfully received packet based on the successfully received first redundant packet to obtain a recovered packet. This avoids the problem in related arts of having to wait for all first redundant packets to arrive before initiating decoding, thereby significantly reducing decoding latency. This method can effectively reduce lag, particularly in networks with high packet loss rates, while also reducing CPU resource consumption.

[0191] For example, Figure 10 The decoding process diagram provided in the embodiment of the present application may include the following steps:

[0192] Step 1001: receiving a first redundant data packet;

[0193] Step 1002: parsing the first configuration information in the received first redundant data packet;

[0194] Step 1003, determining received first redundant data packets with the same group identifier, and classifying them into the same processing queue;

[0195] Step 1004, determining whether there are any unreceived data packets (i.e., lost packets) in the current group;

[0196] It should be noted that if there is packet loss, the recovery process is triggered; otherwise, decoding is skipped;

[0197] Step 1005: Determine whether the number of successfully received first redundant data packets is greater than or equal to the number of lost packets. If so, execute step 1006; otherwise, execute step 1001 and continue waiting for the next first redundant data packet.

[0198] Step 1006: merge the successfully received data packets (packets that are not lost) and the first redundant data packet into the same buffer;

[0199] Step 1007, using a decoder to perform decoding;

[0200] Step 1008, obtaining the restored data packet;

[0201] Step 1009: parse the actual payload.

[0202] In one embodiment, the process of receiving multiple data packet groups and the first redundant data packet corresponding to each of the multiple data packet groups in step 801 may further include the following steps:

[0203] A plurality of data packet groups, a first redundant data packet corresponding to each of the plurality of data packet groups, and a second redundant data packet corresponding to the first redundant data packet are received, wherein the second redundant data packet is obtained by redundantly encoding the first redundant data packet corresponding to each data packet group.

[0204] It can be understood from the above embodiment that when the packet loss rate is high (greater than 30%), the first redundant data packet of the FEC can only be partially received, which may make the original data packet unrecoverable. Therefore, the first redundant data packet is further subjected to secondary redundant protection, that is, the transmitting device can perform redundant encoding based on the first redundant data packet corresponding to each data packet group to obtain a second redundant data packet.

[0205] Accordingly, after the transmitting device sends multiple data packet groups, the first redundant data packet corresponding to each of the multiple data packet groups, and the second redundant data packet corresponding to the first redundant data packet to the receiving device, the receiving device can receive multiple data packet groups, the first redundant data packet corresponding to each of the multiple data packet groups, and the second redundant data packet corresponding to the first redundant data packet.

[0206] It should be noted that some data packets may be lost during the receiving process of the receiving device due to network reasons.

[0207] It should be noted that the relevant content of the above-mentioned second redundant data packet has been described in detail in the above-mentioned application embodiment, and for the sake of brevity, it will not be repeated here.

[0208] In one embodiment, in step 803 above, based on the successfully received first redundant data packet, the process of decoding the successfully received data packet to obtain a recovered data packet may further include the following steps:

[0209] Obtain a packet loss rate of data packets within a third historical time period and a recovery rate of data packets within a fourth historical time period; when the packet loss rate is greater than a first preset threshold and the recovery rate is less than a second preset threshold, decode the successfully received first redundant data packet based on the second redundant data packet to obtain a recovered first redundant data packet; and decode the successfully received data packet based on the recovered first redundant data packet to obtain a recovered data packet.

[0210] It is understandable that when the packet loss rate is high (greater than 30%), only part of the FEC first redundant data packet can be received, which may make it impossible to recover the original data packet. In other words, when the packet loss rate is low, the second redundant data packet is not needed and the original data packet can be recovered based on the first redundant data packet alone.

[0211] In some embodiments, the receiving device can obtain the packet loss rate of data packets within a preset time length and the recovery rate of data packets within the preset time length. When the packet loss rate is greater than a first preset threshold and the recovery rate is less than a second preset threshold, the receiving device can decode the successfully received first redundant data packet based on the second redundant data packet to obtain a recovered first redundant data packet. Further, based on the recovered first redundant data packet, the successfully received data packet is decoded to obtain a recovered data packet. For example, when the packet loss rate is greater than 40% and the recovery rate is less than 50%, the receiving device can decode the successfully received first redundant data packet based on the second redundant data packet to obtain a recovered first redundant data packet. Further, based on the recovered first redundant data packet, the successfully received data packet is decoded to obtain a recovered data packet.

[0212] It should also be noted that the recovery rate within a preset time period can be determined and then filtered to calculate the average recovery rate. This can avoid false triggering due to fluctuations in the recovery rate. Specifically, the successfully received first redundant data packet can be decoded based on the second redundant data packet. Similarly, the packet loss rate within a preset time period can be determined and then filtered to calculate the average packet loss rate.

[0213] Through the above embodiment, by monitoring the packet loss rate within the third historical time period and the recovery rate within the fourth historical time period, when the packet loss rate exceeds a preset threshold and the recovery rate falls below the preset threshold, the second redundant data packet is used to decode and recover the successfully received first redundant data packet, and the original data packet is further decoded and recovered based on the recovered first redundant data packet. This effectively addresses data transmission issues in high packet loss environments, significantly improving the packet recovery success rate and transmission reliability, thereby enhancing overall transmission efficiency.

[0214] In one embodiment, in step 802 above, the process of decoding the successfully received first redundant data packet based on the second redundant data packet to obtain a recovered first redundant data packet may further include the following steps:

[0215] In some implementations, the successfully received second redundant data packet is parsed to determine a fourth number of first redundant data packets in the corresponding data packet group.

[0216] It is understandable that during the data packet transmission process, the first redundant data packet may be lost due to poor network conditions or other reasons.

[0217] Based on this, in some embodiments, the receiving device may parse the second configuration information in the second redundant data packet upon receiving the second redundant data packet, thereby determining the number of data packets in the data packet group corresponding to the second redundant data packet.

[0218] As can be seen from the above embodiments, the second configuration information in the second redundant data packet may be a field; or, the second configuration information may also be mask information.

[0219] Exemplarily, if the second configuration information is field information, and the field information obtained by the receiving device after parsing the second configuration information is 101, the receiving device can determine that there are a total of 5 first redundant data packets in the data packet group, that is, the fourth quantity is 5.

[0220] In another example, if the second configuration information is mask, and the field information obtained by the receiving device after parsing the second configuration information is 0001, the receiving device can determine that there is a total of 1 first redundant data packet in the data packet group, that is, the fourth quantity is 1.

[0221] In some embodiments, when the sum of the number of successfully received first redundant data packets in each data packet group and the number of successfully received second redundant data packets in each data packet group is greater than or equal to a fourth number, the successfully received first redundant data packet is decoded based on the successfully received second redundant data packet to obtain a recovered first redundant data packet.

[0222] It is understandable that due to transmission over an unreliable network (such as UDP or a wireless channel), some first redundant data packets may be lost. The receiving device does not need to receive all second redundant data packets to recover the first redundant data packet. In other words, the receiving device only needs to ensure that the sum of the number of successfully received first redundant data packets and the number of successfully received second redundant data packets in each data packet group is greater than or equal to the third number to recover the first redundant data packet.

[0223] For example, if there are 4 first redundant data packets in a data packet group, it can be expressed as [R1, R2, R3, R4]; based on the data packet group, 2 second redundant data packets are generated, which can be expressed as [RR1, RR2], that is, the data packet sequence finally sent by the sending device is [R1, R2, R3, R4, RR1, RR2].

[0224] Furthermore, the current situation of the receiving end device receiving the first redundant data packet is: two first redundant data packets R1 and R2 are successfully received, and two second redundant data packets RR1 and RR2 are successfully received. At this time, the receiving end device can parse RR1 or RR2 to determine that the number of first redundant data packets is 4; the receiving end device determines that the sum of the number of successfully received first redundant data packets and the number of successfully received second redundant data packets in each data packet group is 4, which meets the recovery condition. At this time, the receiving end device does not need to continue waiting for the next first redundant data packet, but can directly recover the first redundant data packet according to R1, R2RR1, RR2, so as to obtain the recovered R3, R4.

[0225] An embodiment of the present application proposes a data transmission method, comprising: a transmitting end determining whether it is in a first state or a second state based on current network state parameters, and accordingly dividing data packets into different numbers of data packet groups; then, redundantly encoding at least one to-be-transmitted data packet in each data packet group to generate a first redundant data packet, and transmitting the first redundant data packet along with the original data packet during transmission. In this way, the transmitting end device adjusts the data packet grouping size by dynamically adapting to the network state, using larger groups for redundant encoding to reduce computational overhead when network quality is good, and reducing the grouping size to improve the timeliness of data recovery when network quality is poor, thereby optimizing coding efficiency while ensuring error correction capabilities.

[0226] The present application also provides a method for recovering from packet loss, comprising: receiving multiple packet groups and a first redundant packet corresponding to each of the multiple packet groups; parsing the successfully received first redundant packet to determine a third number of packets in the corresponding packet group; and, if the sum of the number of successfully received packets in the packet group and the number of successfully received first redundant packets in each packet group is greater than or equal to the third number, decoding the successfully received packet based on the successfully received first redundant packet to obtain a recovered packet. This avoids the problem in the related art of having to wait for all first redundant packets to arrive before initiating decoding, thereby significantly reducing decoding delay. This method can effectively reduce lag and reduce CPU resource consumption, particularly in network environments with high packet loss rates.

[0227] The following examples describe possible implementations of the data sending method and data packet recovery method of one or more of the above embodiments.

[0228] It should be noted that due to the algorithmic characteristics of Reed-Solomon forward error correction (RSFEC), the algorithm complexity is O(m×n), where M is the original packet and N is the number of check packets. When packet loss is high (>50%) and full redundancy is required, for a 150KB I-frame, each RTP packet is 1500 bytes, 100 RTP packets are required. The full redundancy computational complexity is O(100×100). This encoding time is very high and is almost unacceptable in real-time streaming. It consumes a lot of CPU resources, and RS-FEC causes encoding and recovery delays, resulting in noticeable lag at the receiving end. In some embodiments, the original packets are evenly grouped to reduce the packet group size, with a maximum of 4 packets per group. In this way, even when all are redundant, the maximum complexity of each group of data is O(4×4), and the encoding is completed within 1ms. This avoids timeouts caused by waiting for redundant packets for too long at the decoding end, and there is no need to wait for all parity check packets to be received before decoding and recovery. Recovery can be performed as long as the sum of the received original packets and check packets is greater than or equal to the number of original packets.

[0229] In some embodiments, the same mask (the first configuration information) is added to the headers of all parity_packets (the first redundant data packet) in the same packet group, ensuring that the number of protected packets (the number of data packets in the data packet group) and the number of check packets can be determined when each FEC packet is received. There is no need to wait until all parity_packets are received. As long as the conditions are ripe, the process can be restored immediately. Subsequent parity_packets will detect that all protected packets have been restored and end early.

[0230] In some embodiments, the packet size is fixed, and the size of an RTP packet is repackaged into 1408 bytes (<1500 maximum transmission unit (MTU)), the insufficient data is padded with 0, and RS-FEC encoding and decoding is performed on the entire packet data unit to avoid the delay caused by byte and packet matching after RS-FEC decoding is restored.

[0231] In some embodiments, the redundancy rate of RS-FEC is adjusted in real time based on a comprehensive consideration of three factors: the current packet loss rate loss-rate, the video_bitrate encoding bitrate (the above-mentioned data packet encoding bitrate), and the estimated_bandwidth predicted bandwidth (the above-mentioned predicted available bandwidth), to avoid the disadvantages of using a fixed redundancy rate. If the redundancy rate is too high, bandwidth will be wasted, and if the redundancy rate is too low, the original data cannot be restored.

[0232] Through the optimizations described above, smooth playback is possible even under 50% packet loss conditions. The lag rate before optimization was 22.8%, while after optimization, it was 1.5%. The end-to-end latency before optimization was 320ms, while after optimization, it was reduced to 150ms.

[0233] It should be noted that in the current FEC protection strategy, when the packet loss rate is high (greater than 30%), the FEC parity packet can only be partially received, which makes the original packet unrecoverable. Therefore, in order to maximize the recovery of lost data in a far-field high packet loss and weak network environment, the FEC parity packet is further protected with secondary redundancy.

[0234] In some embodiments, a redundant FEC packet (the second redundant data packet) is nested on the basis of the original FEC to protect the first redundant packet. When the first redundant packet is partially lost, the loss_rate reaches a certain threshold (40%) and the recovery_rate is lower than 50%.

[0235] In some embodiments, the recovery-rate within 3 seconds is counted and then filtered to calculate the average recovery-rate to avoid false triggering due to fluctuations in the recovery-rate.

[0236] In some embodiments, when the packet loss rate is greater than 40% and the recovery-rate is lower than 50%, the second FEC stream is enabled in real time to protect the first redundant packet (the first redundant data packet mentioned above). When the first FEC stream cannot restore the original packet due to packet loss, the second FEC stream is used to restore the packet of the first FEC stream, and then the restored first FEC packet is used to restore the original packet.

[0237] In some embodiments, the redundancy rate of the second FEC path is based on the recovery-rate of the first path, using (1-recovery_rate) as the second redundancy rate:

[0238] In this way, after enabling nested FEC, the overall recovery rate is increased from 70% to 90% in a weak network environment with 40% packet loss, effectively reducing playback stuttering.

[0239] Based on the above embodiment, in another embodiment of the present application, Figure 11 This is a schematic diagram of the structure of the data sending device proposed in the embodiment of the present application. The data sending device is applied to the sending end device, such as Figure 11 As shown, the data sending device 110 proposed in the embodiment of the present application may include:

[0240] Acquisition module 1101, used to obtain network status parameters;

[0241] A first dividing module 1102 is configured to divide a plurality of to-be-transmitted data packets corresponding to a frame of data generated by precoding into a plurality of first data packet groups in response to a network state parameter satisfying a first state; the number of data packets in each first data packet group is a first number;

[0242] A second division module 1103 is configured to, in response to the network state parameter satisfying a second state, divide a plurality of to-be-sent data packets corresponding to a frame of data into a plurality of second data packet groups; the number of data packets in each second data packet group is a second number, the network state represented by the first state is better than the network state represented by the second state, and the first number is greater than the second number;

[0243] The encoding module 1104 is configured to perform redundant encoding on at least one to-be-sent data packet in each data packet group to obtain a first redundant data packet corresponding to each data packet group;

[0244] The sending module 1105 is configured to send multiple data packet groups and a first redundant data packet corresponding to each data packet group.

[0245] In some embodiments, the first redundant data packet includes first configuration information, and the first configuration information is used by the receiving end to determine the number of data packets in the data packet group corresponding to the first redundant data packet.

[0246] In some embodiments, each data packet group corresponds to multiple first redundant data packets, and the first configuration information of the multiple first redundant data packets is the same, so that the receiving end can determine the number of data packets in the data packet group based on any received first redundant data packet.

[0247] In some embodiments, the data sending device 110 may further include a determination module for determining a packet loss rate of data packets within a first historical period; the determination module is further used to determine a first redundancy rate based on the packet loss rate when a first preset condition is met; the first redundancy rate is related to the number of first redundant data packets; the encoding module 1104 is further used to perform redundant encoding on at least one to-be-sent data packet in each data packet group based on the first redundancy rate to obtain a first redundant data packet corresponding to each data packet group.

[0248] In some embodiments, the encoding module 1104 is further configured to perform redundant encoding on the first redundant data packet corresponding to each data packet group to obtain a second redundant data packet corresponding to the first redundant data packet;

[0249] Correspondingly, the sending module 1105 is further configured to send multiple data packet groups, a first redundant data packet corresponding to each data packet group, and a second redundant data packet corresponding to the first redundant data packet.

[0250] In some embodiments, the second redundant data packet includes second configuration information, and the second configuration information is used by the receiving end to determine the number of redundant data packets in the first redundant data packet corresponding to the second redundant data packet.

[0251] Based on the above embodiment, in another embodiment of the present application, Figure 12 This is a schematic diagram of the structure of the data packet loss recovery device proposed in an embodiment of the present application. The data packet loss recovery device is applied to a receiving device, such as Figure 12 As shown, the data packet loss recovery device 120 proposed in the embodiment of the present application may include:

[0252] A receiving module 1201 is configured to receive a plurality of data packet groups and a first redundant data packet corresponding to each of the plurality of data packet groups;

[0253] A determination module 1202 is configured to parse the successfully received first redundant data packet and determine a third number of data packets in the corresponding data packet group;

[0254] a decoding module 1203 configured to, when the sum of the number of successfully received data packets in the data packet group and the number of successfully received first redundant data packets in each data packet group is greater than or equal to a third number, decode the successfully received data packets based on the successfully received first redundant data packets to obtain recovered data packets;

[0255] Among them, the first redundant data packet is obtained by redundantly encoding at least one data packet to be sent in each data packet group; the number of data packets in each data packet group is a first number or a second number, and the first number is greater than the second number; the first number is related to a first state of a network state parameter, and the second number is related to a second state of the network state parameter, and the network state represented by the first state is better than the network state represented by the second state.

[0256] In some embodiments, the receiving module 1201 is further used to receive multiple data packet groups, a first redundant data packet corresponding to each data packet group in the multiple data packet groups, and a second redundant data packet corresponding to the first redundant data packet; wherein the second redundant data packet is obtained by redundantly encoding the first redundant data packet corresponding to each data packet group.

[0257] The description of the above device embodiment is similar to the description of the above method embodiment and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of this application, please refer to the description of the method embodiment of this application for understanding.

[0258] It should be noted that the division of modules in the embodiments of the present application is schematic and is merely a logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units. They may also be implemented in the form of a combination of software and hardware.

[0259] It should be noted that, in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling an electronic device to execute all or part of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0260] In the embodiments of the present application, further, Figure 13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 13 As shown, the electronic device 130 proposed in the embodiment of the present application may include a processor 1301, a memory 1302, a communication interface 1303, and a bus 1304 for connecting the processor 1301, the memory 1302 and the communication interface 1303.

[0261] In an embodiment of the present application, the processor 1301 may be at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, and a microprocessor. It is understandable that for different devices, the electronic device used to implement the above-mentioned processor function may also be other, and the embodiment of the present application is not specifically limited. The electronic device 130 may further include a memory 1302, which may be connected to the processor 1301, wherein the memory 1302 is used to store executable program code, the program code including computer operating instructions, and the memory 1302 may include a high-speed RAM memory, and may also include a non-volatile memory, for example, at least two disk memories.

[0262] In the embodiment of the present application, the bus 1304 is used to connect the communication interface 1303, the processor 1301 and the memory 1302, as well as the mutual communication between these devices.

[0263] In actual applications, the above-mentioned memory 1302 can be a volatile memory (volatile memory), such as random-access memory (RAM); or a non-volatile memory (non-volatile memory), such as read-only memory (ROM), flash memory, hard disk drive (HDD) or solid-state drive (SSD); or a combination of the above types of memory, and provide instructions and data to the processor 1301.

[0264] It should be noted that the memory 1302 is configured to store instructions and applications executable by the processor 1301, and can also cache data to be processed or processed by the processor 1301 and various modules in the electronic device 130 (for example, image data, audio data, voice communication data and video communication data), which can be implemented through flash memory (FLASH) or random access memory (RAM).

[0265] In the embodiments of the present application, there is no restriction on the type of electronic device, and the electronic device can be a variety of devices with image processing capabilities. For example, the electronic device can be various personal computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. It should be noted that the electronic device can be a terminal device or a server.

[0266] An embodiment of the present application also provides a computer-readable storage medium for storing a computer program.

[0267] Optionally, the computer-readable storage medium can be applied to the electronic device in the embodiments of the present application, and the computer program enables the processor or electronic device to execute the various methods of the embodiments of the present application. For the sake of brevity, they are not described here.

[0268] An embodiment of the present application also provides a computer program product, including computer program instructions.

[0269] Optionally, the computer program product can be applied to the electronic device in the embodiments of the present application, and the computer program instructions enable the processor or electronic device to execute the various methods of the embodiments of the present application. For the sake of brevity, they are not described here in detail.

[0270] The embodiment of the present application also provides a computer program.

[0271] Optionally, the computer program can be applied to the electronic device in the embodiments of the present application. When the computer program runs on a processor or electronic device, the processor or electronic device executes the various methods of the embodiments of the present application. For the sake of brevity, they are not described here in detail.

[0272] It should be noted that the descriptions of the electronic device, storage medium, computer program product, and computer program embodiments described above are similar to the descriptions of the method embodiments described above and have similar beneficial effects as the method embodiments. For technical details not disclosed in the electronic device, storage medium, computer program product, and computer program embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0273] It should be understood that "one embodiment" or "an embodiment" or "some embodiments" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" or "in some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments. The above description of the various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other. For the sake of brevity, they will not be repeated here.

[0274] The term "and / or" in this article is only a description of the association relationship between associated objects, indicating that there can be three relationships. For example, object A and / or object B can mean: object A exists alone, object A and object B exist at the same time, and object B exists alone.

[0275] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0276] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of the modules is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or modules can be electrical, mechanical or other forms.

[0277] The modules described above as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules; they may be located in one place or distributed across multiple network units; some or all of the modules may be selected according to actual needs to achieve the purpose of this embodiment.

[0278] In addition, all functional modules in the embodiments of the present application can be integrated into one processing unit, or each module can be a separate unit, or two or more modules can be integrated into one unit; the above-mentioned integrated modules can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0279] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, and other media that can store program codes.

[0280] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling an electronic device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks or optical disks.

[0281] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0282] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0283] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0284] The above is merely an embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A data transmission method, applied to a transmitting device, comprising: Get network status parameters; In response to the network state parameter satisfying a first state, dividing a plurality of to-be-sent data packets corresponding to a frame of data generated through precoding into a plurality of first data packet groups; The number of data packets in each first data packet group is a first number; In response to the network state parameter satisfying a second state, dividing the plurality of to-be-sent data packets corresponding to the one frame of data into a plurality of second data packet groups; The number of data packets in each second data packet group is a second number, the network state represented by the first state is better than the network state represented by the second state, and the first number is greater than the second number; performing redundant encoding on at least one to-be-sent data packet in each data packet group to obtain a first redundant data packet corresponding to each data packet group; The multiple data packet groups and the first redundant data packet corresponding to each data packet group are sent.

2. The method according to claim 1, wherein the first redundant data packet includes first configuration information, and the first configuration information is used by a receiving end to determine the number of data packets in the data packet group corresponding to the first redundant data packet.

3. The method according to claim 2, wherein each data packet group corresponds to multiple first redundant data packets, and the first configuration information of the multiple first redundant data packets is the same, so that the receiving end can determine the number of data packets in the data packet group based on any one of the received first redundant data packets.

4. The method according to claim 3, wherein the redundant encoding of at least one to-be-sent data packet in each data packet group to obtain a first redundant data packet corresponding to each data packet group comprises: Determining a packet loss rate of data packets within a first historical time period; When a first preset condition is met, determining a first redundancy rate according to the packet loss rate; The first redundancy rate is related to the number of the first redundant data packets; Based on the first redundancy rate, redundant encoding is performed on at least one to-be-sent data packet in each data packet group to obtain a first redundant data packet corresponding to each data packet group.

5. The method according to any one of claims 1 to 4, wherein after performing redundant encoding on at least one to-be-sent data packet in each data packet group to obtain a first redundant data packet corresponding to each data packet group, the method further comprises: performing redundant encoding on the first redundant data packet corresponding to each data packet group to obtain a second redundant data packet corresponding to the first redundant data packet; Correspondingly, the sending of the multiple data packet groups and the first redundant data packet corresponding to each data packet group includes: The multiple data packet groups, the first redundant data packet corresponding to each data packet group, and the second redundant data packet corresponding to the first redundant data packet are sent.

6. The method according to claim 5, wherein the second redundant data packet comprises second configuration information, and the second configuration information is used by the receiving end to determine the number of redundant data packets in the first redundant data packet corresponding to the second redundant data packet.

7. A method for recovering from packet loss, the method being applied to a receiving device, the method comprising: receiving a plurality of data packet groups and a first redundant data packet corresponding to each of the plurality of data packet groups; parsing the successfully received first redundant data packet to determine a third number of data packets in the corresponding data packet group; When the sum of the number of successfully received data packets in the data packet group and the number of successfully received first redundant data packets in each data packet group is greater than or equal to the third number, decoding the successfully received data packets based on the successfully received first redundant data packets to obtain recovered data packets; The first redundant data packet is obtained by redundantly encoding at least one data packet to be sent in each data packet group; the number of data packets in each data packet group is a first number or a second number, and the first number is greater than the second number; the first number is related to a first state of a network state parameter, and the second number is related to a second state of a network state parameter, and the network state represented by the first state is better than the network state represented by the second state.

8. The method according to claim 7, wherein receiving a plurality of data packet groups and a first redundant data packet corresponding to each of the plurality of data packet groups comprises: receiving a plurality of data packet groups, a first redundant data packet corresponding to each of the plurality of data packet groups, and a second redundant data packet corresponding to the first redundant data packet; The second redundant data packet is obtained by redundantly encoding the first redundant data packet corresponding to each data packet group.

9. A data sending device, applied to a sending end device, comprising: Acquisition module, used to obtain network status parameters; a first dividing module, configured to divide a plurality of to-be-sent data packets corresponding to a frame of data generated by precoding into a plurality of first data packet groups in response to the network state parameter satisfying a first state; The number of data packets in each first data packet group is a first number; A second division module is configured to divide the plurality of to-be-sent data packets corresponding to the one frame of data into a plurality of second data packet groups in response to the network state parameter satisfying a second state; The number of data packets in each second data packet group is a second number, the network state represented by the first state is better than the network state represented by the second state, and the first number is greater than the second number; an encoding module, configured to perform redundant encoding on at least one to-be-sent data packet in each data packet group to obtain a first redundant data packet corresponding to each data packet group; The sending module is configured to send the multiple data packet groups and the first redundant data packet corresponding to each data packet group.

10. A device for recovering data packet loss, the device being applied to a receiving device, the device comprising: A receiving module, configured to receive a plurality of data packet groups and a first redundant data packet corresponding to each of the plurality of data packet groups; a determination module, configured to parse the successfully received first redundant data packet and determine a third number of data packets in the corresponding data packet group; a decoding module, configured to, when the sum of the number of successfully received data packets in the data packet group and the number of successfully received first redundant data packets in each data packet group is greater than or equal to the third number, decode the successfully received data packets based on the successfully received first redundant data packets to obtain recovered data packets; The first redundant data packet is obtained by redundantly encoding at least one data packet to be sent in each data packet group; the number of data packets in each data packet group is a first number or a second number, and the first number is greater than the second number; the first number is related to a first state of a network state parameter, and the second number is related to a second state of a network state parameter, and the network state represented by the first state is better than the network state represented by the second state.

Images