Adaptive video transmission method based on fountain code, electronic device and storage medium

Abstract

The application provides a fountain code-based adaptive video transmission method, an electronic device and a storage medium, and the method comprises the following steps: receiving historical packet loss rates of all data packets corresponding to video image sent frames which are counted and fed back by a receiving end of a video transmission system; updating coding redundancy for fountain code encoding of a current frame of a video image based on the historical packet loss rates of the video image sent frames; compressively encoding the current frame of the video image to be transmitted to generate a compressed code stream data frame; fountain code encoding the compressed code stream data frame according to the coding redundancy to generate coding data packets corresponding to the current frame of the video image; and transmitting the coding data packets corresponding to the current frame of the video image to the receiving end of the video transmission system. The application can realize real-time video transmission with low computational complexity, low performance consumption and high decoding success rate.

Description

Technical Field

[0001] This invention relates to the field of real-time video transmission technology, and in particular to an adaptive video transmission method, electronic device, and storage medium based on fountain codes. Background Technology

[0002] In real-time video transmission, video data is prone to packet loss due to adverse network conditions, which manifests at the receiving end as frequent stuttering and jitter during playback. For example, when network quality is poor, insufficient coding redundancy cannot provide effective packet loss protection; when the network is good, excessive coding redundancy reduces bandwidth utilization and introduces latency. Furthermore, if network fluctuations are significant, adjusting future fountain code encoding parameters based on past packet loss rates often results in excessive packet loss and decoding failures.

[0003] Erasure coding originated in the field of communication transmission. Its main function is to solve the problems of data loss and data error in data transmission. The principle of erasure coding can be simply summarized as follows: The sending end encodes k source data packets into n data packets using an erasure algorithm, and then transmits the n data packets over the network. When the receiving end receives any m (m≥k) data packets from the n data packets, it can obtain the k source data packets through a certain decoding algorithm. k / n is called the coding rate of the erasure coding, and (nk) / k is the coding redundancy of the erasure coding. Fountain codes (Raptor codes), as a forward error correction technique, can improve the packet loss problem during transmission. Fountain codes are rate-free codes. Compared with traditional fixed-rate erasure codes, they can generate any number of encoded packets. They can adaptively adjust the coding redundancy according to the packet loss situation of the channel and dynamically adjust the code rate according to the channel conditions to adapt to different network conditions and ensure sufficient decoding success rate. The two IETF standards for fountain codes are RFC 5053 and RFC 6330. Bitrate refers to the amount of data used by a video file per unit time, also called bit rate. A higher bitrate indicates a higher sampling rate per unit time, resulting in higher data stream accuracy.

[0004] Video transmission congestion control technology is a technique that dynamically adjusts the transmission bitrate of the sending end based on network congestion during video transmission. Congestion control mainly consists of two parts: the transmission protocol and the video encoder. The transmission protocol estimates the available bandwidth of the network based on network congestion signals, and the video encoder compresses the original video data based on the available network bandwidth so that the bitrate of the compressed video does not exceed the available network bandwidth measured by the transmission protocol.

[0005] In recent years, adaptive coding redundancy algorithms based on deep learning methods have been continuously developing. Historical packet loss patterns implicitly indicate network conditions. Deep neural networks can fully utilize the relationship between history and the future. The receiver feeds back the observed packet loss patterns, and the sender collects the fed-back packet loss patterns and uses a long short-term memory network algorithm to predict the packet loss rate in a short period of time, and adjusts the coding redundancy according to the predicted packet loss rate.

[0006] Although predicting short-term packet loss rate and adjusting coding redundancy based on deep learning methods can effectively balance the relationship between decoding success rate and decoding overhead, there are the following problems: (1) Deep learning-based algorithms have high computational complexity and are not suitable for deployment on devices with limited computing power; (2) The scale requirements for deep learning models are high, the performance consumption is large, and it is difficult to apply them to real-time video transmission systems with limited computing resources.

[0007] Therefore, how to provide a real-time video transmission method with low computational complexity, low performance consumption, and high decoding success rate is a technical problem that urgently needs to be solved. Summary of the Invention

[0008] In view of this, embodiments of the present invention provide an adaptive video transmission method, electronic device, and storage medium based on fountain codes to eliminate or improve one or more defects existing in the prior art.

[0009] One aspect of the present invention provides an adaptive video transmission method based on fountain codes. This method is applied at the transmitting end of a video transmission system, where the video image to be transmitted consists of multiple frames. The method includes the following steps: receiving the historical packet loss rate of all data packets corresponding to the transmitted frames of the video image, as statistically analyzed and fed back by the receiving end of the video transmission system; updating the coding redundancy of fountain code encoding on the current frame of the video image based on the historical packet loss rate of the transmitted frames; compressing and encoding the current frame of the video image to be transmitted to generate a compressed bitstream data frame; performing fountain code encoding on the compressed bitstream data frame according to the coding redundancy to generate an encoded data packet corresponding to the current frame of the video image; and sending the encoded data packet corresponding to the current frame of the video image to the receiving end of the video transmission system.

[0010] In some embodiments of the present invention, the method further includes: adjusting the target size of the generated compressed bitstream data frame based on coding redundancy; performing fountain code encoding on the compressed bitstream data frame according to coding redundancy to generate an encoded data packet corresponding to the current frame of the video image, including: slicing the current frame of the video image according to a preset slice length and the target size of the compressed bitstream data frame to obtain a number of slice data frame slices; generating a fountain code encoding corresponding to the current frame of the video image using a fountain code encoding algorithm according to the coding redundancy and the number of slice data frame slices; wherein the fountain code encoding has the same slice length as the current frame of the video image; and encapsulating the fountain code encoding into an encoded data packet corresponding to the current frame of the video image according to a preset data packet message format.

[0011] In some embodiments of the present invention, the preset data packet format includes a packet header and fountain code encoding. The packet header includes at least the data frame sequence number, data frame length, slice length, number of slices, and encoding redundancy.

[0012] In some embodiments of the present invention, before the step of adjusting the target size of the generated compressed bitstream data frame based on coding redundancy, the method further includes: adjusting the resolution of the video image to be sent based on the most recent historical packet loss rate; and / or adjusting the target compression bitrate of the compressed encoding based on the most recent historical packet loss rate.

[0013] In some embodiments of the present invention, the step of adjusting the resolution of the video image to be sent based on the most recent historical packet loss rate includes: reducing the resolution within a preset adjustment range when the two most recent consecutive historical packet loss rates are greater than a preset first packet loss rate threshold; increasing the resolution within a preset adjustment range when the two most recent consecutive historical packet loss rates are less than a preset second packet loss rate threshold; otherwise, keeping the resolution unchanged. The step of adjusting the target compression bitrate based on the most recent historical packet loss rate includes: reducing the target compression bitrate within a preset adjustment range when the two most recent consecutive historical packet loss rates are greater than a preset first packet loss rate threshold; increasing the target compression bitrate within a preset adjustment range when the two most recent consecutive historical packet loss rates are less than a preset second packet loss rate threshold; otherwise, keeping the target compression bitrate unchanged.

[0014] In some embodiments of the present invention, the step of updating the coding redundancy of the current frame of the video image based on the historical packet loss rate of the transmitted frames of the video image, wherein the coding redundancy of the fountain code encoding is the number of redundant symbols used for fountain code encoding, includes: analyzing the changing trend and rate of change of the packet loss rate based on the acquired multiple historical packet loss rates closest to the current time; calculating the number of redundant symbols by combining the changing trend of the packet loss rate with a preset calculation formula that includes two parts: a general coding redundancy number and a supplementary coding redundancy number; wherein, when the changing trend of the packet loss rate is decreasing, the supplementary coding redundancy number is set to 0.

[0015] In some embodiments of the present invention, when the rate of change of the packet loss rate is greater than 0, that is, the packet loss rate changes with an increasing trend and the rate of change is getting faster and faster, the value of the packet loss rate growth rate contained in the supplementary coding redundancy number is taken as the first value; otherwise, the value of the packet loss rate growth rate contained in the supplementary coding redundancy number is taken as the second value.

[0016] In some embodiments of the present invention, the preset calculation formula is as follows:

[0017]

[0018] Where R is the number of redundant symbols used for fountain code encoding. The general coding redundancy part is α(β+ΔP)K, which is the supplementary coding redundancy part, where K is the number of slices and P is the number of slices. lost Let P be the packet loss rate, and α be a parameter value that adjusts the redundant coding portion to a value of 0 when the packet loss rate decreases. i-2 P i-1 P i It refers to the packet loss rate information received by the sending end three times consecutively, where ΔP represents the rate of change of the packet loss rate, and ΔP = (P i -P i-1 )-(P i-1 -P i-2 ), let P i -P i-1 The value is Δ i P i-1 -P i-2 The value is Δ i-1 Δ i This indicates the trend of packet loss rate, when Δ i When the value of Δ is negative, the value of α is 0; when Δ i-1 and Δ i When all values ​​are positive, α takes the value of 1, and the encoding will increase the number of redundant symbols in (β+ΔP)K; β is the packet loss rate growth rate, and this value takes effect when the packet loss rate increases. When ΔP>0, the value of β is the first value, and when ΔP≤0, the value of β takes the second value.

[0019] In some embodiments of the present invention, the step of compressing and encoding the current frame of the video image to be transmitted to generate a compressed bitstream data frame uses the H.264 algorithm as an example of the compression encoding algorithm.

[0020] One aspect of the present invention provides an adaptive video transmission method based on fountain codes. This method is applied at the receiving end of a video transmission system, where the video image to be received consists of multiple frames. The method includes the following steps: receiving encoded data packets corresponding to the current frame of the video image from the sending end; parsing the encoded data packets according to a preset data packet format; calculating the packet loss rate corresponding to the current frame of the video image based on the parsed encoded data packets; and feeding the packet loss rate back to the sending end so that the sending end updates the encoding redundancy of the fountain code encoding of the current frame of the video image based on the historical packet loss rate of the previously transmitted video image frames; and performing fountain code decoding and compression decoding on the parsed encoded data packets to obtain the video image to be received.

[0021] In some embodiments of the present invention, the preset data packet format includes a packet header and fountain code encoded data. The packet header includes at least a data frame sequence number, a data frame length, a slice length, a slice count, and encoding redundancy. The step of calculating the packet loss rate of the current frame of a video image based on the parsed encoded data packets includes: obtaining all encoded data packets of the current frame of the video image to be received; determining whether the encoded data packets of the current frame of the video image are lost based on the data frame sequence number; and calculating the packet loss rate of the current channel based on the matching relationship between the current encoded data packet and the previous encoded data packet.

[0022]

[0023] Where, N 00 This indicates that both the previously encoded data packet and the currently encoded data packet were received correctly; N 01 This indicates that the current encoded data packet was not received correctly, assuming the previous encoded data packet was received correctly; N 10 This indicates that the current encoded data packet was received correctly even though the previous encoded data packet was not received correctly; N 11 This indicates that neither the previous encoded data packet nor the current encoded data packet was received correctly.

[0024] Another aspect of the present invention provides an electronic device including a processor and a memory, wherein the memory stores computer instructions, and the processor is configured to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the electronic device implements the steps of any of the methods described in the above embodiments.

[0025] Another aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of any of the methods described in the above embodiments.

[0026] The adaptive video transmission method, electronic device, and storage medium based on fountain codes proposed in this invention adjust the coding redundancy according to the changes in packet loss rate in the channel, appropriately increasing the redundancy to ensure decoding success rate. Compared with deep learning-based adaptive coding redundancy algorithms, this invention significantly reduces computational complexity and performance consumption, thereby shortening the processing time. Systems based on this method can be deployed in devices with limited computing power, especially in real-time video transmission systems with limited actual computing resources, achieving real-time video transmission with low computational complexity, low performance consumption, and high decoding success rate.

[0027] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the description, or may be learned by practice of the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures specifically pointed out in the description and drawings.

[0028] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to the specific ones described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0029] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, are not intended to limit the scope of the invention. In the drawings:

[0030] Figure 1 This is a flowchart of video transmission at the sending end in one embodiment of the present invention.

[0031] Figure 2 This is a flowchart of fountain coding of compressed bitstream data frames in one embodiment of the present invention.

[0032] Figure 3 This is a schematic diagram of an adaptive video transmission system structure in one embodiment of the present invention.

[0033] Figure 4 This is a schematic diagram of adaptive coding redundancy adjustment in another embodiment of the present invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0035] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0036] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0037] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0038] In the following description, embodiments of the invention will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0039] To address the limitations of existing adaptive video transmission methods in real-time transmission scenarios due to their low complexity and high performance requirements, this invention provides an adaptive video transmission method, electronic device, and storage medium based on fountain codes. This method offers the advantage of low performance overhead. It adjusts the coding redundancy based on changes in the packet loss rate in the channel, appropriately increasing redundancy to ensure decoding success rate. Simultaneously, it estimates a bandwidth value based on packet loss and adjusts the sending bit rate at the transmitting end to adapt to the transmission bandwidth, achieving better video transmission performance. Fountain codes are a type of forward error correction technique used to improve packet loss during transmission. Specifically, fountain codes are rate-free codes that adaptively adjust coding redundancy based on channel packet loss conditions to adapt to different network conditions. Two IETF standards related to fountain codes are RFC 5053 and RFC 6330.

[0040] Figure 1 This is a flowchart of a video transmission process at the transmitting end in one embodiment of the present invention. The method is applied to the transmitting end of a video transmission system. The video image to be transmitted consists of multiple frames. The method includes the following steps:

[0041] Step S110: Receive the historical packet loss rate of all data packets corresponding to the video image sent frames as statistically analyzed and fed back by the receiving end of the video transmission system, and update the coding redundancy of the fountain code encoding of the current frame of the video image based on the historical packet loss rate of the video image sent frames.

[0042] The encoding redundancy has an initial value that is used when the sending end first transmits a video image to the receiving end, at which time there is no historical packet loss rate as a reference.

[0043] In step S110, the coding redundancy of fountain code encoding is the number of redundant symbols used for fountain code encoding. The step of updating the coding redundancy of the current frame of the video image based on the historical packet loss rate of the transmitted video image frames includes: analyzing the changing trend and rate of change of the packet loss rate based on the acquired historical packet loss rates closest to the current time; calculating the number of redundant symbols by combining the changing trend of the packet loss rate with a preset calculation formula that includes a general coding redundancy number and a supplementary coding redundancy number; wherein, when the changing trend of the packet loss rate is decreasing, the supplementary coding redundancy number is set to 0.

[0044] Step S120: Compress and encode the current frame of the video image to be transmitted to generate a compressed bitstream data frame.

[0045] Optionally, the compression encoding method may be MPEG encoding compression technology, H.264 algorithm, H.265 encoding compression and Huffman coding, etc., but the present invention is not limited to these.

[0046] Step S130: Compress and encode the current frame of the video image to be transmitted to generate a compressed bitstream data frame.

[0047] Step S140: Perform fountain code encoding on the compressed bitstream data frames according to the encoding redundancy to generate the encoded data packet corresponding to the current frame of the video image. Wherein,

[0048] Step S150: Send the encoded data packet corresponding to the current frame of the video image to the receiving end of the video transmission system.

[0049] The adaptive video transmission method based on fountain codes provided by this invention improves the packet loss rate, adjusts the coding redundancy, and appropriately increases redundancy to ensure decoding success rate by changing the processing parameters of the acquired image and video data at the transmitting end of the video transmission system. This reduces computational complexity and performance consumption, thereby shortening the processing time and achieving better video transmission results.

[0050] In some embodiments of the present invention, before step S120, the method further includes: adjusting the target size of the generated compressed bitstream data frame based on the coding redundancy.

[0051] Furthermore, before adjusting the target size of the generated compressed bitstream data frame based on the coding redundancy, the method further includes: adjusting the resolution of the video image to be sent according to the most recent historical packet loss rate; and / or adjusting the target compression bitrate of the compressed encoding according to the most recent historical packet loss rate.

[0052] The step of adjusting the resolution of the video image to be sent based on the most recent historical packet loss rate includes: when the two most recent consecutive historical packet loss rates are greater than a preset first packet loss rate threshold, reducing the resolution within a preset adjustment range; when the two most recent consecutive historical packet loss rates are less than a preset second packet loss rate threshold, increasing the resolution within a preset adjustment range; otherwise, keeping the resolution unchanged. The step of adjusting the target compression bitrate based on the most recent historical packet loss rate includes: when the two most recent consecutive historical packet loss rates are greater than a preset first packet loss rate threshold, reducing the target compression bitrate within a preset adjustment range; when the two most recent consecutive historical packet loss rates are less than a preset second packet loss rate threshold, increasing the target compression bitrate within a preset adjustment range; otherwise, keeping the target compression bitrate unchanged. Wherein, if the rate of change of the packet loss rate is greater than 0, i.e., the packet loss rate is increasing and the rate of change is accelerating, the value of the packet loss rate growth rate included in the supplementary coding redundancy is taken as the first value; otherwise, the value of the packet loss rate growth rate included in the supplementary coding redundancy is taken as the second value. It should be noted that the present invention is not limited thereto. The preset threshold and adjustment strategy mentioned above are only examples. For example, the adjustment condition can also be set to reduce the resolution within a preset adjustment range when the three consecutive historical packet loss rates closest to the current time are greater than the preset first packet loss rate threshold.

[0053] Specifically, the preset calculation formula is:

[0054]

[0055] Where R is the number of redundant symbols used for fountain code encoding. The general coding redundancy part is α(β+ΔP)K, which is the supplementary coding redundancy part, where K is the number of slices and P is the number of slices. lost Let P be the packet loss rate, and α be a parameter value that adjusts the redundant coding portion to a value of 0 when the packet loss rate decreases. i-2 P i-1 P i It refers to the packet loss rate information received by the sending end three times consecutively, where ΔP represents the rate of change of the packet loss rate, and ΔP = (P i -P i-1 )-(P a-1 -P i-2 ), let P i -P i-1 The value is Δ i P i-1 -P i-2 The value is Δ i-1 Δ i This indicates the trend of packet loss rate, when Δ iWhen the value of Δ is negative, the value of α is 0; when Δ i-1 and Δ i When all values ​​are positive, α is 1, increasing the number of redundant symbols in the encoding by (β+ΔP)K. β is the packet loss rate growth rate, which takes effect when the packet loss rate increases. When ΔP>0, β has the first value, and when ΔP≤0, β has the second value. These first and second values ​​are custom-designed, with the basic principle being to ensure that the coding redundancy is just the minimum value that satisfies the packet loss rate threshold. This prevents insufficient redundancy from causing excessively high packet loss rates that could affect the image quality at the receiving end, and also prevents excessive redundancy from increasing the transmission pressure on the channel for real-time video images (i.e., video data), as high channel transmission pressure also affects the real-time video transmission effect.

[0056] Another aspect of the present invention provides an adaptive video transmission method based on fountain codes. This method is applied to the receiving end of a video transmission system, where the video image to be received consists of multiple frames. The method includes: receiving an encoded data packet corresponding to the current frame of the video image from a sending end; parsing the encoded data packet according to a preset data packet message format; calculating the packet loss rate corresponding to the current frame of the video image based on the parsed encoded data packet; and feeding the packet loss rate back to the sending end, so that the sending end updates the encoding redundancy of the fountain code encoding of the current frame of the video image based on the historical packet loss rate of the previously transmitted video image frames; and performing fountain code decoding and compression decoding on the parsed encoded data packet to obtain the video image to be received.

[0057] In some embodiments of the present invention, the preset data packet format includes a packet header and fountain code encoded data. The packet header includes at least the data frame sequence number, data frame length, slice length, number of slices, and encoding redundancy.

[0058] The step of calculating the packet loss rate of the current frame of the video image based on the parsed encoded data packets includes:

[0059] (1) Obtain all the encoded data packets of the current frame of the video image to be received;

[0060] (2) Determine whether the encoded data packet of the current frame of the video image is lost based on the data frame sequence number;

[0061] (3) Calculate the packet loss rate of the current channel based on the matching relationship between the current encoded data packet and the previous encoded data packet to determine whether they were received correctly:

[0062]

[0063] Where, N 00 This indicates that both the previously encoded data packet and the currently encoded data packet were received correctly; N01 This indicates that the current encoded data packet was not received correctly, assuming the previous encoded data packet was received correctly; N 10 This indicates that the current encoded data packet was received correctly even though the previous encoded data packet was not received correctly; N 11 This indicates that neither the previous encoded data packet nor the current encoded data packet was received correctly.

[0064] Figure 2 This is a flowchart illustrating fountain coding of compressed bitstream data frames according to an embodiment of the present invention. In step S140, this step includes:

[0065] Step S141: Based on the preset slice length and the target size of the compressed bitstream data frame, slice the current frame of the video image to obtain the number of slices (data frame slices).

[0066] Step S142: According to the coding redundancy and the number of data frame slices, a fountain code encoding algorithm is used to generate the fountain code encoding corresponding to the current frame of the video image; wherein, the fountain code encoding has the same slice length as the current frame of the video image. Further, in some embodiments of the present invention, step S142 also includes a slice length supplementation step during the process of dividing the current frame of the video image into multiple slices.

[0067] Step S143: According to a preset data packet format, the fountain code is encoded and encapsulated into an encoded data packet corresponding to the current frame of the video image. The preset data packet format includes a header and the fountain code encoding. The header includes at least the data frame sequence number, data frame length, slice length, number of slices, and encoding redundancy.

[0068] Figure 3 This is a schematic diagram of an adaptive video transmission system structure according to an embodiment of the present invention, including a transmitting end 210 and a receiving end 220. The overall method steps are described below based on the different design modules in the process:

[0069] (1) Video acquisition module 211: In this embodiment of the invention, the video acquisition module 211 acquires video data. The acquired video data consists of many frames of video images. The video images are preset with parameters such as resolution, acquisition frame rate and image format.

[0070] Video data is essentially a media stream, representing a continuous segment of data on the timeline, such as audio data, video data, or subtitle data. It can be compressed or uncompressed; compressed data requires a specific codec. In one embodiment of this invention, the codec used is the FFMPEG codec.

[0071] Typically, a media stream consists of a large number of data frames / packets. For compressed data, a frame corresponds to the smallest processing unit of the codec. Data frames belonging to different media streams are interleaved and stored in containers.

[0072] (2) H.264 compression encoding module 212: In this embodiment of the invention, H.264 encoding module 212 is used for compression encoding. The acquired video images are H.264 compressed and encoded, and the encoded bitstream (i.e., the compressed video data stream) is output. That is, each frame of the video image is compressed into a compressed bitstream data frame. In this step, the compression ratio is adjusted by the channel packet loss rate fed back by the receiving end. The compression ratio can be understood as the coding redundancy of the compression encoding.

[0073] The meaning of coding redundancy is: in the field of image or video transmission, in order to minimize the amount of data required to represent digital images, the basic principle of reducing the amount of data is to remove redundant data, which requires the use of image compression technology. Coding redundancy is a parameter used to describe the degree of redundant data contained in the transmitted information after data compression.

[0074] H.264 is a video data compression standard, consisting of a Network Data Abstraction Layer (VCL) and a Data Coding Layer (VCL). The VCL is used for unpacking and reassembling the raw video data before transmission, while the VCL compresses the raw video data. Its compression principle lies in using inter-frame compression technology to record only one complete frame from a group of highly similar images within a given time frame, while the other frames only retain the pixels that differ from each other.

[0075] Alternatively, other compression coding standards include the MPEG video coding standard and FFMPEG (Fast Forward). FFMPEG (Fast Forward) is an open-source computer program that can be used to record, convert digital audio and video, and stream them, and can easily convert between various video formats.

[0076] (3) System Raptor Encoding Module 213: A module based on fountain code encoding technology, which performs fountain code encoding on the compressed video data stream. In this step, the coding redundancy of the fountain code encoding is adaptively adjusted based on the channel packet loss rate fed back from the receiver. The specific formula is shown in Formula (1).

[0077] The receiving end receives all the data packets of a video frame and then calculates the packet loss rate, feeding the calculated packet loss rate back to the sending end via the UDP protocol.

[0078] (4) Video transmission module 214: It is used to add header information to the encoded data and send it to the receiving end. The receiving end counts the channel packet loss rate and feeds it back to the sending end.

[0079] Specifically, in this step, each frame of the Raptor-encoded video data is split into multiple slices, and header information is added to each slice. These slices are then encapsulated into multiple data packets corresponding to each frame of the video data. These multiple data packets corresponding to the video data are then sent to the receiving end. Subsequently, the receiving end calculates the channel packet loss rate and feeds it back to the sending end.

[0080] (5) Video reception and packet loss statistics module 224: Receives each frame of the video image, counts the packet loss rate, and feeds back the packet loss rate to the sending end.

[0081] (6) System Raptor Decoding Module 223: Performs fountain code decoding (i.e., system Raptor decoding) on ​​each frame of the received video image, and all frames form an H.264 bitstream. The H.264 bitstream is the bitstream that requires H.264 decoding processing. The concept of a bitstream is that the data generated after video data is compressed is called a bitstream. A bitstream can also be understood as multiple frames of image data, only compressed, and needs to be decoded to be recognized.

[0082] The received video data packets are decoded by the system Raptor to obtain the H.264 bitstream.

[0083] (7) H.264 decompression module 222: decodes the original image data through H.264 data decoding module 222.

[0084] (8) Video display module 221: Displays the decoded data on the screen.

[0085] The system Raptor code is the foundation of this invention. The basic principle of the system Raptor code is as follows: the original video data is divided into K data blocks of fixed length, each data block is considered a coded symbol, and the K source symbols are subjected to system Raptor encoding to generate N Raptor code symbols. When the receiving end receives M (M>N) coded symbols, system Raptor decoding can be performed to recover the original data. The larger M is, the higher the decoding success rate.

[0086] The N encoded symbols after the system Raptor encoding consist of two parts: K source symbols and R redundant symbols. The redundant symbols are generated from the source symbols, and the number R of redundant symbols (also known as repair symbols) needs to be specified during encoding.

[0087] When the number of received encoded symbols N is 5 more than the number of source symbols K, the successful decoding rate of the system's Raptor code can reach 99%. During video transmission, if the number of received encoded symbols is ensured to be slightly greater than the number of source symbols by 5 each time, a sufficient decoding success rate can be guaranteed with low redundancy, while simultaneously reducing decoding overhead. Knowing the channel's packet loss rate before encoding allows determining the value of the redundant symbol R.

[0088]

[0089] Among them, P lost This refers to the packet loss rate of the channel. Theoretically, the transmitter can determine the number of redundant symbols for the next encoding based on the packet loss rate reported by the receiver. However, this method is based on the premise that the packet loss rate of the channel is fixed, and it cannot adapt to complex network environments. The actual network environment is complex and changeable, but network fluctuations have certain regularities in a short period of time. That is, there is a certain correlation between historical packet loss patterns and future packet loss patterns. The coding redundancy can be adjusted according to the changes in the historical packet loss rate as shown in formula (1):

[0090] Formula (1) represents the improved value of R, where α(β+ΔP)K is an additional coding redundancy added based on the changes in packet loss rate. This redundancy value is determined based on the changes in packet loss rate.

[0091] Changes in packet loss rate can reflect network fluctuations to some extent, where R is the number of redundant symbols used for fountain code encoding. α(β+ΔP)K is the general coding redundancy part, α(β+ΔP)K is the supplementary coding redundancy part, K ​​is the number of slices, and P is the number of slices. lost Let P be the packet loss rate, α be a parameter value that adjusts the value of the supplementary coding redundancy part to 0 when the packet loss rate changes in a decreasing trend. i-2 P i-1 P i It refers to the packet loss rate information received by the sending end three times consecutively, where ΔP represents the rate of change of the packet loss rate, and ΔP = (P i -P i-1 )-(P i-1 -P i-2 ), let P i -P i-1 The value is Δ i P i-1 -P i-2 The value is Δ i-1 Δ i This indicates the trend of packet loss rate, when Δ i When the value of Δ is negative, the value of α is 0; when Δ i-1 and Δ iWhen all values ​​are positive, α takes the value of 1, and the encoding will increase the number of redundant symbols in (β+ΔP)K; β is the packet loss rate growth rate, which takes effect when the packet loss rate increases. When ΔP>0, β takes the first value, and when ΔP≤0, β takes the second value.

[0092] β is a growth rate that comes into play when the packet loss rate increases. Its specific values ​​are as follows:

[0093]

[0094] In this embodiment of the invention, the first value is 0.1 and the second value is 0.5. It can be seen from this formula that when the packet loss rate increases, the redundancy increases at a greater rate; when the packet loss rate decreases, the redundancy increases at a smaller rate. This invention is not limited to this; the values ​​of the first and second values ​​mentioned above are merely examples.

[0095] Based on the adaptive fountain code coding redundancy adjustment method described above, the redundancy (i.e., redundant symbols R) is maintained at a level that ensures the number of received encoded symbols is slightly greater than the number of source symbols by a factor of 5. This maintains the system's Raptor code successful decoding rate above 99%, ensuring sufficient decoding success rate even with low redundancy while reducing decoding overhead. This avoids increasing transmission pressure on the channel and prevents the degradation of real-time image transmission quality that could result from high packet loss. The redundancy threshold of this invention is not limited to this; for example, it can be modified to ensure the number of received encoded symbols is slightly greater than the number of source symbols by a factor of 4.

[0096] Figure 4 This is a schematic diagram of adaptive coding redundancy adjustment in another embodiment of the present invention, illustrating an adaptive redundancy adjustment model based on packet loss rate. At the transmitting end, the compressed video data is sliced, and all data slices are sent as a coding unit to the fountain code encoder for encoding. The receiving end counts the channel packet loss rate and feeds it back to the transmitting end. The transmitting end determines the coding redundancy according to the change in packet loss rate and updates the value of R for the next encoding, which is used for the next fountain code encoding.

[0097] In this embodiment of the invention, at the transmitting end 310, Figure 4 The image data in the image is compressed and encoded by H.264. One frame of the image is divided into multiple slices. The multiple slices enter the fountain code encoder 311. The redundancy scheduling and adjustment module 312 calculates the number of redundant symbols based on the packet loss rate fed back by the receiver 321 and formula (1). At the receiver 320, the packet loss statistics module 321 calculates the packet loss rate based on formula (2). The fountain code decoder 322 decodes the data packets of the video image for each frame and sends them to the H.264 decoding module for further decoding.

[0098] Since packet loss rate reflects not only network conditions but also, to some extent, the actual available bandwidth of the channel, the available bandwidth can be estimated based on the packet loss rate, allowing for adjustment of the transmitter's bitrate. The adjustment strategy is as follows: if the packet loss rate is low, it indicates sufficient available bandwidth, and the transmitter's bitrate can be appropriately increased within the bitrate adjustment range; if the packet loss rate is high, it suggests network congestion, and the bitrate exceeds the available bandwidth, requiring a corresponding reduction in bitrate. After data is compressed using the H.264 standard, it has a compression ratio V. The H.264 compression standard offers two compression modes: constant bitrate and variable bitrate. In constant bitrate mode, the compression ratio V is a fixed value, which can be adjusted to control the size of the compressed data, thereby adjusting the video data flow in the channel.

[0099] In another embodiment of the present invention, a sending end transmission rate adjustment strategy is designed: the initial compression ratio V is set to 2%; when the packet loss rate is less than 2% for two consecutive times, the H.264 compression ratio is increased to 1.08 times the previous value; when the packet loss rate is between 2% and 10%, the compression ratio remains unchanged; when the packet loss rate exceeds 10% for two consecutive times, the H.264 compression ratio is adjusted to 95% of the previous value.

[0100] The adaptive video transmission method and system based on fountain codes proposed in this invention adjust the coding redundancy according to changes in the packet loss rate in the channel, appropriately increasing the redundancy to ensure decoding success rate. Simultaneously, a bandwidth value is estimated based on the packet loss situation to control the compression ratio of the compression coding, i.e., adjusting the transmission bitrate at the sending end to adapt to the transmission bandwidth, achieving better video transmission performance. Compared with deep learning-based adaptive coding redundancy algorithms, this invention significantly reduces computational complexity and performance consumption, thereby shortening the processing time. Systems based on this method can be deployed in devices with limited computing power, especially in real-time video transmission systems with limited actual computing resources, achieving real-time video transmission with low computational complexity, low performance consumption, and a high decoding success rate.

[0101] The adaptive video transmission method and system based on fountain codes proposed in this invention combine packet loss recovery and congestion control. It utilizes the redundancy adjustment of fountain code encoding to recover from packet loss, and adjusts the transmission flow of the sending end according to the packet loss situation to reduce packet loss events caused by channel congestion, which helps to improve the quality of real-time video transmission.

[0102] This invention can ensure the lowest possible packet loss rate and channel transmission pressure while maintaining the same transmitted image quality, and it also considers reducing the clarity of the transmitted image as an alternative, further ensuring the quality of real-time video transmission.

[0103] Corresponding to the above method, the present invention also provides an adaptive video transmission system based on fountain codes, comprising a transmitter, the transmitter including:

[0104] (1) The first module of the transmitting end is used to compress and encode the current frame of the video image to be transmitted to generate a compressed bitstream data frame;

[0105] (2) The second module of the transmitting end is used to perform fountain code encoding on the compressed bitstream data frame according to the encoding redundancy, and generate the encoded data packet corresponding to the current frame of the video image;

[0106] (3) The third module of the sending end is used to update the coding redundancy of the fountain code encoding of the current frame of the video image based on the historical packet loss rate of the video image sent frames;

[0107] (4) The fourth module of the transmitting end is used to receive the historical packet loss rate of all data packets corresponding to the video image sent frames as statistically reported and fed back by the receiving end of the video transmission system, and to send the encoded data packet corresponding to the current frame of the video image to the receiving end of the video transmission system.

[0108] In one embodiment of the present invention, the first module of the transmitting end is a compression encoding module, which may optionally be an H.264 video encoding module, etc. Figure 3 As shown. The second module of the sending end is a fountain code encoder, used to encode the video data after compression by the compression module using fountain codes. The third module of the sending end is a redundancy adjustment module, or a packet loss rate analysis module. The fourth module of the sending end is the transceiver module located at the sending end.

[0109] Corresponding to the above method, the present invention also provides an adaptive video transmission system based on fountain codes, comprising a receiver, the receiver including:

[0110] (1) The first module of the receiving end is used to receive the encoded data packet corresponding to the current frame of the video image from the sending end, and feed back the packet loss rate to the sending end so that the sending end can update the encoding redundancy of the fountain code encoding of the current frame of the video image based on the historical packet loss rate of the video image already sent frames.

[0111] (2) The receiving end second module is used to parse the encoded data packet according to the preset data packet message format, and to calculate the packet loss rate corresponding to the current frame of the video image based on the parsed encoded data packet;

[0112] (3) The third module of the receiving end is used to perform fountain code decoding and compression decoding on the parsed encoded data packets in order to obtain the video image to be received.

[0113] In one embodiment of the present invention, the first module of the receiving end is an encoding module located at the receiving end, the second module of the receiving end is a packet loss rate statistics module, and the third module of the receiving end is a compression decoding module. The compression decoding module of the receiving end corresponds to the compression encoding module of the sending end. For example, the H.264 encoding module corresponds to the H.264 decoding module.

[0114] Corresponding to the above method, the present invention also provides an adaptive video transmission system based on fountain codes, comprising a transmitter and a receiver. The transmitter includes: (1) a first transmitter module, used to compress and encode the current frame of the video image to be transmitted to generate a compressed bitstream data frame; (2) a second transmitter module, used to perform fountain code encoding on the compressed bitstream data frame according to the encoding redundancy, generating an encoded data packet corresponding to the current frame of the video image; (3) a third transmitter module, used to update the encoding redundancy of the fountain code encoding of the current frame of the video image based on the historical packet loss rate of the previously transmitted video image frames; and (4) a fourth transmitter module, used to receive the historical packet loss rate of all data packets corresponding to the previously transmitted video image frames as statistically analyzed and fed back by the receiver of the video transmission system, and to send the encoded data packet corresponding to the current frame of the video image to the receiver of the video transmission system. The receiving end includes: (1) a first receiving end module, used to receive the encoded data packet corresponding to the current frame of the video image from the sending end, and feed back the packet loss rate to the sending end so that the sending end can update the encoding redundancy of the fountain code encoding of the current frame of the video image based on the historical packet loss rate of the video image sent frames; (2) a second receiving end module, used to parse the encoded data packet according to the preset data packet message format, and calculate the packet loss rate corresponding to the current frame of the video image based on the parsed encoded data packet; (3) a third receiving end module, used to perform fountain code decoding and compression decoding on the parsed encoded data packet to obtain the video image to be received.

[0115] Corresponding to the above method, the present invention also provides an electronic device, which includes a computer device, the computer device including a processor and a memory, the memory storing computer instructions, the processor executing the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the electronic device performs the steps of the method as described above.

[0116] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, performs the steps of the method as described above. The computer-readable storage medium may be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, floppy disk, hard disk, removable storage disk, CD-ROM, or any other form of storage medium known in the art.

[0117] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0118] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0119] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0120] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An adaptive video transmission method based on fountain codes, characterized in that, The method is applied to the transmitting end of a video transmission system, where the video image to be transmitted consists of multiple frames. The method includes: The receiving end of the video transmission system counts and reports the historical packet loss rate of all data packets corresponding to the sent frames of the video image, and updates the coding redundancy of the fountain code encoding of the current frame of the video image based on the historical packet loss rate of the sent frames of the video image. The current frame of the video image to be transmitted is compressed and encoded to generate a compressed bitstream data frame; Fountain code encoding is performed on the compressed bitstream data frame according to the aforementioned coding redundancy to generate the encoded data packet corresponding to the current frame of the video image; Send the encoded data packet corresponding to the current frame of the video image to the receiving end of the video transmission system; The step of updating the coding redundancy of the current frame of the video image based on the historical packet loss rate of the sent frames of the video image, wherein the coding redundancy of the fountain code encoding is the number of redundant symbols used for fountain code encoding, includes: analyzing the changing trend and rate of change of the packet loss rate based on the acquired multiple historical packet loss rates closest to the current time; calculating the number of redundant symbols by combining the changing trend of the packet loss rate with a preset calculation formula that includes two parts: a general coding redundancy number and a supplementary coding redundancy number; wherein, when the changing trend of the packet loss rate is decreasing, the supplementary coding redundancy number is set to 0. The preset calculation formula is: ； in, The number of redundant symbols used for fountain code encoding, This refers to the general coding redundancy portion. For the supplementary coding redundancy part, Number of slices The current packet loss rate of the channel. To adjust the parameter value to 0 for the supplementary coding redundancy part when the packet loss rate changes in a decreasing trend, let's call it... , , These are the packet loss rate information received consecutively by the sending end. This indicates the rate of change of the packet loss rate. ,remember The value is , The value is , This indicates the trend of packet loss rate. When the value is negative The value of is 0; when and When all values ​​are positive When the value is 1, the encoding will increase. The number of redundant symbols; This is the packet loss rate growth rate; this value takes effect when the packet loss rate increases. hour The value is the first value, when hour The value is the second value, the first value is 0.1, and the second value is 0.

05.

2. The method according to claim 1, characterized in that, The method further includes: adjusting the target size of the generated compressed bitstream data frame based on the coding redundancy; The step of performing fountain code encoding on the compressed bitstream data frame according to the encoding redundancy to generate the encoded data packet corresponding to the current frame of the video image includes: Based on the preset slice length and the target size of the compressed bitstream data frame, the current frame of the video image is sliced ​​to obtain a number of data frame slices; According to the coding redundancy and the number of data frame slices, the fountain code encoding algorithm is used to generate the fountain code encoding corresponding to the current frame of the video image; wherein, the fountain code encoding has the same slice length as the current frame of the video image; According to the preset data packet format, the fountain code is encoded and encapsulated into an encoded data packet corresponding to the current frame of the video image.

3. The method according to claim 2, characterized in that, The preset data packet format includes a packet header and the fountain code encoding. The packet header includes at least the data frame sequence number, data frame length, slice length, number of slices, and encoding redundancy.

4. The method according to claim 2, characterized in that, Before the step of adjusting the target size of the generated compressed bitstream data frame based on the coding redundancy, the method further includes: adjusting the resolution of the video image to be sent according to the most recent historical packet loss rate; and / or The target compression bitrate of the compression code is adjusted based on the most recent historical packet loss rate.

5. The method according to claim 4, characterized in that, The step of adjusting the resolution of the video image to be sent based on the most recent historical packet loss rate includes: when the two most recent consecutive historical packet loss rates are greater than a preset first packet loss rate threshold, reducing the resolution within a preset adjustment range; when the two most recent consecutive historical packet loss rates are less than a preset second packet loss rate threshold, increasing the resolution within a preset adjustment range; otherwise, keeping the resolution unchanged. The step of adjusting the target compression bitrate of the compression code based on the most recent historical packet loss rate includes: when the two most recent consecutive historical packet loss rates are greater than a preset first packet loss rate threshold, reducing the target compression bitrate within a preset adjustment range; when the two most recent consecutive historical packet loss rates are less than a preset second packet loss rate threshold, increasing the target compression bitrate within a preset adjustment range; otherwise, keeping the target compression bitrate unchanged.

6. The method according to claim 1, characterized in that, When the rate of change of packet loss rate is greater than 0, that is, the trend of packet loss rate change is increasing and the rate of change is getting faster and faster, the value of the packet loss rate growth rate contained in the supplementary coding redundancy number is taken as the first value; otherwise, the value of the packet loss rate growth rate contained in the supplementary coding redundancy number is taken as the second value.

7. The method according to claim 1, characterized in that, The step of compressing and encoding the current frame of the video image to be transmitted to generate a compressed bitstream data frame uses the H.264 algorithm as an example of a compression encoding algorithm.

8. An adaptive video transmission method based on fountain codes, characterized in that, The method is applied at the receiving end of a video transmission system, where the video image to be received consists of multiple frames. The method includes: Receive the encoded data packet corresponding to the current frame of the video image from the sending end; According to the preset data packet format, the encoded data packet is parsed, the packet loss rate corresponding to the current frame of the video image is calculated based on the parsed encoded data packet, and the packet loss rate is fed back to the sending end so that the sending end updates the encoding redundancy of the fountain code encoding of the current frame of the video image based on the historical packet loss rate of the video image frames that have been sent. The parsed encoded data packets are then subjected to fountain code decoding and compression decoding to obtain the video image to be received.

9. The method according to claim 8, characterized in that, The preset data packet format includes a packet header and fountain code encoded data. The packet header includes at least the data frame sequence number, data frame length, slice length, number of slices, and encoding redundancy. The step of calculating the packet loss rate of the current frame of the video image based on the parsed encoded data packets includes: Obtain all encoded data packets of the current frame of the video image to be received; Determine whether the encoded data packet of the current frame of the video image has been lost based on the data frame sequence number; Based on the matching relationship between the current encoded data packet and the previous encoded data packet to determine whether they were correctly received, calculate the packet loss rate of the current channel: ； in, This indicates that both the previously encoded data packet and the currently encoded data packet were received correctly; This indicates that the current encoded data packet was not received correctly, assuming the previous encoded data packet was received correctly. This indicates that the current encoded data packet was received correctly when the previous encoded data packet was not received correctly. This indicates that neither the previous encoded data packet nor the current encoded data packet was received correctly.

10. An electronic device comprising a processor and a memory, characterized in that, The memory stores computer instructions, and the processor executes the computer instructions stored in the memory. When the computer instructions are executed by the processor, the electronic device implements the steps of the method as described in any one of claims 1 to 9.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in any one of claims 1 to 9.

Images