Video transmission method, device and system

Abstract

The video transmission method, device, and system provided by the embodiments of the present application can reduce the end-to-end transmission delay of video data. The method includes: a transmitting end obtains a first frame image of video data, the first frame image including multiple sub-images, the multiple sub-images including a first sub-image and a second sub-image; the transmitting end performs layered encoding on the first sub-image to obtain a multi-layer code stream of the first sub-image; the transmitting end sends the multi-layer code stream of the first sub-image to a receiving end; the receiving end decodes the multi-layer code stream of the first sub-image to obtain the first sub-image; the receiving end displays the first sub-image; the transmitting end performs layered encoding on the second sub-image to obtain a multi-layer code stream of the second sub-image; the transmitting end sends the multi-layer code stream of the second sub-image to the receiving end; the receiving end decodes the multi-layer code stream of the second sub-image to obtain the second sub-image; and the receiving end displays the second sub-image.

Description

Technical Field

[0001] The present application relates to the field of multimedia, and more specifically, to a video transmission method, device, and system in the field of multimedia. Background Art

[0002] Short-range wireless screen projection technology involves encoding and compressing video data generated by a powerful mobile terminal, such as game graphics rendered by a graphics processing unit (GPU), and wirelessly transmitting it to a device with less processing power but better display quality, such as a TV or virtual reality (VR) headset. This technology can solve the problem of connecting and sharing information between mobile devices and other devices.

[0003] Short-range wireless screen projection applications, such as game screen projection and VR glasses, require data interaction, and therefore require the end-to-end latency of the projection system to be as low as possible. Furthermore, since wireless fidelity (WIFI) wireless channels are easily affected by multiple factors such as noise and interference signals, transmission bandwidth instability can result in data loss and image quality issues. Therefore, the projection system must be as resistant to channel changes as possible. In summary, end-to-end latency and channel interference resistance are both important factors affecting the performance of the projection system and the user experience.

[0004] In short-range wireless projection scenarios, scalable coding is currently used to encode and transmit video. Scalable coding involves encoding the source video into multiple layers of varying quality / resolution that reference each other. The higher-quality / higher-resolution layer references, and can only reference, layers of equal or lower quality / lower resolution during encoding. During transmission, the video is transmitted from lower-quality / lower-resolution layers to higher-quality / higher-resolution layers. The more layers the decoder receives, the better the quality of the reconstructed image.

[0005] However, in existing scalable coding methods, the transmitter encodes a frame of image into multiple layers of code streams with different qualities / resolutions that have a reference relationship, and sends each layer of code stream to the receiver layer by layer. The receiver needs to wait until the transmission of multiple layers of code streams is completed before it can reconstruct the frame of image and display the output. This results in high end-to-end latency, which affects the user experience. Summary of the Invention

[0006] Embodiments of the present application provide a video transmission method, apparatus, and system that can reduce the end-to-end transmission delay of video data and improve robustness to channel changes.

[0007] In a first aspect, embodiments of the present application provide a video transmission method. The method can be applied to a video transmission system, the video transmission system including a transmitter and a receiver, and the method can include the following steps: the transmitter obtains a first frame image of video data, the first frame image including multiple sub-images, the multiple sub-images including a first sub-image and a second sub-image; the transmitter performs layered encoding on the first sub-image to obtain a multi-layer code stream of the first sub-image; the transmitter sends the multi-layer code stream of the first sub-image to a receiver; the receiver receives the multi-layer code stream of the first sub-image from the transmitter; the receiver decodes the multi-layer code stream of the first sub-image to obtain the first sub-image; the receiver displays the first sub-image; the transmitter performs layered encoding on the second sub-image to obtain a multi-layer code stream of the second sub-image; the transmitter sends the multi-layer code stream of the second sub-image to the receiver; the receiver receives the multi-layer code stream of the second sub-image from the transmitter; the receiver decodes the multi-layer code stream of the second sub-image to obtain the second sub-image; and the receiver displays the second sub-image.

[0008] It should be noted that in existing video transmission methods, when layered encoding is performed on multiple sub-images within a frame, the second layer of encoding is performed on each sub-image only after the first layer of code stream for each sub-image is obtained. After the second layer of code stream for each sub-image is obtained, the third layer of encoding is performed on each sub-image, and so on. However, with this encoding method, the receiving end must wait until all multiple layers of code streams for the frame are successfully received before displaying the frame, or after a specified transmission timeout, decode and display the frame based on the received code streams, resulting in significant transmission latency. In contrast, in the embodiments of the present application, when layered encoding is performed on multiple sub-images within a frame, the second sub-image is layered encoded only after the multi-layer code streams for the first sub-image are encoded. This allows the layered encoding of the second sub-image to proceed in parallel with the transmission of the multi-layer code streams for the first sub-image to the receiving end. The receiving end can display the first sub-image after receiving its multi-layer code stream without waiting for the code streams of other sub-images, significantly reducing the end-to-end latency when presenting the image to the user.

[0009] Optionally, this method can be applied to short-range wireless screen projection scenarios. Short-range wireless screen projection refers to a technology that encodes and compresses video data generated by a mobile terminal with strong processing capabilities and sends it wirelessly to a device with weak processing capabilities but good display quality for display.

[0010] For example, taking the sending end as a mobile phone and the receiving end as VR glasses, the mobile phone can project the rendered game screen onto the display interface of the VR glasses.

[0011] For another example, taking the transmitting end as a laptop and the receiving end as a TV, the laptop can project the TV screen onto the display interface of the TV.

[0012] It should be noted that the video data may include multiple frames of images arranged in a frame sequence, and the multiple frames of images include the first frame of image and the second frame of image.

[0013] Optionally, the sending end may obtain the video data in a variety of ways, which is not limited in this embodiment of the present application.

[0014] In a possible implementation, the sending end may receive the video data from the first device.

[0015] In another possible implementation, the sending end may generate the video data itself.

[0016] In another possible implementation, the sending end may obtain original video data, and pre-process the original video data to obtain processed video data.

[0017] Optionally, before the sending end performs layered encoding on the first sub-image to obtain a multi-layer code stream of the first sub-image, the sending end may divide the first frame image into the multiple sub-images.

[0018] It should be noted that the sub-image described in the embodiment of the present application can be understood as the granularity of video data encoded by the transmitter or decoded by the receiver, for example, the minimum granularity.

[0019] Optionally, the sub-image may be, but is not limited to, a slice, a tile, a block, a macroblock (MB) or a largest coding unit (LCU) of a frame image, which is not limited in the embodiments of the present application.

[0020] Optionally, the transmitting end may perform layered encoding on the first sub-image based on multiple video encoding protocols, which is not limited in the embodiment of the present application.

[0021] In a possible implementation, the sending end may perform layered encoding on the first sub-image based on the H.264 Scalable Video Coding (SVC) protocol and the H.265 Scalable Video Coding (SHVC) protocol.

[0022] In another possible implementation, the sending end may perform layered encoding on the first sub-image based on a future scalable video coding protocol, such as the H.266 scalable video coding protocol.

[0023] It should be noted that the multi-layer code streams of the first sub-image have different reference relationships. For example, the transmitting end may first encode the code stream of the base layer of the first sub-image, then encode the code stream of the enhancement layer 1 with reference to the base layer, and then encode the code stream of the enhancement layer 2 with reference to the code stream of the enhancement layer 1. In this case, the image quality (or bit rate) increases in sequence according to the encoding order of base layer - enhancement layer 1 - enhancement layer 2.

[0024] That is, in the process of layered coding, the image quality corresponding to the referenced code stream layer is lower than the image quality corresponding to the code stream layer that refers to the referenced layer.

[0025] It should also be noted that the image quality corresponding to the code stream layer described in the embodiment of the present application may include at least one of resolution, image quality or frame rate.

[0026] Optionally, before the receiving end displays the first sub-image, the receiving end may perform post-processing on the sub-image to obtain the processed first sub-image, and display the processed first sub-image.

[0027] For example, the post-processing may include at least one of color format conversion, color grading, retouch, or resampling.

[0028] It should be noted that the transmitting end encodes the second sub-image after completing multi-layer encoding on the first sub-image.

[0029] Optionally, the transmitting end and the receiving end may determine the encoding order of the second sub-image and the first sub-image and their positions in the first frame image in a variety of ways, which is not limited in the embodiment of the present application.

[0030] In a possible implementation, a method for dividing a frame image into multiple sub-images and a coding order for each sub-image in the frame image may be defined in a video coding protocol.

[0031] In another possible implementation, the transmitting end and the receiving end may pre-agree on a method for dividing a frame image into multiple sub-images and a coding order for each sub-image in the frame image.

[0032] In another possible implementation, the transmitting end may send instruction information to the receiving end, where the instruction information is used to indicate a method for dividing the frame image into multiple sub-images and an encoding order for each sub-image in the frame image. Accordingly, the receiving end may receive the instruction information from the transmitting end and determine the method for dividing the frame image into multiple sub-images and an encoding order for each sub-image in the frame image based on the instruction information.

[0033] It should be noted that the processing process of the second sub-image by the sending end (including encoding, sending, decoding and display) is similar to the processing process of the first sub-image. Please refer to the following introduction to the processing process of the first sub-image. To avoid repetition, it will not be repeated here.

[0034] It should be noted that in order to better reduce the delay of video transmission, the processing time of the encoding, transmission and decoding steps of the video data can be agreed upon and controlled in the embodiments of the present application.

[0035] Optionally, the sending end sending the multi-layer code stream of the first sub-image to the receiving end may include: the sending end sending the code stream of the first sub-image to the receiving end layer by layer within a first preset time interval; correspondingly, the receiving end may receive the code stream of the first sub-image from the sending end layer by layer within the first preset time interval.

[0036] That is, the transmitting end sends the code stream of the first sub-image to the receiving end layer by layer starting from the start time of the first preset time interval, and stops sending at the end time of the first preset time interval. Correspondingly, the receiving end receives the code stream of the first sub-image from the transmitting end layer by layer starting from the start time of the first preset time interval, and stops receiving at the end time of the first preset time interval.

[0037] In a possible implementation, the transmitting end may send the code stream of the first sub-image to the receiving end layer by layer according to the coding order of the multi-layer code stream of the first sub-image.

[0038] During the layered coding process, the image quality of the codestream obtained by encoding the first sub-image is lower than that of the codestream obtained by encoding the second sub-image. However, due to the lower image quality of the codestream obtained by encoding the first sub-image, the transmission delay of the codestream obtained by encoding the first sub-image layer by layer in the coding order helps to ensure that the user is provided with an acceptable display image of high image quality in the event of unstable channel conditions, thereby improving robustness to channel variations.

[0039] It should be noted that, since the multiple layers of code streams of the first sub-image have a reference relationship during encoding, and the transmitting end sends the code stream of the first sub-image layer by layer according to the encoding order of the multiple layers of code streams of the first sub-image, accordingly, the receiving end needs to decode the code stream of the first sub-image layer by layer according to the encoding order of the multiple layers of code streams of the first sub-image.

[0040] Optionally, the receiving end may determine the encoding order of the multi-layer code stream of the first sub-image in a variety of ways, which is not limited in this embodiment of the present application.

[0041] In a possible implementation, if the transmission order of the code stream is not affected by factors such as channel conditions, the receiving end may use the order in which the multi-layer code stream of the first sub-image is received as the encoding order. In other words, the receiving order is the encoding order.

[0042] In another possible implementation, if the transmission order of the code stream is affected by factors such as channel conditions, the transmitting end may carry indication information for indicating the encoding order of each layer of the code stream in each layer of the code stream. Accordingly, when the receiving end receives each layer of the code stream of the first sub-image, it can determine the encoding order of each layer of the code stream based on the indication information carried by each layer of the code stream.

[0043] Optionally, the sending end may send the code stream of the first sub-image to the receiving end in a variety of ways according to the coding order of the multi-layer code stream of the first sub-image, which is not limited in this embodiment of the present application.

[0044] In one possible implementation, the transmitting end may directly send each layer of the code stream of the first sub-image to the receiving end after encoding the layer of the code stream, and then continue to encode the next layer of the code stream of the first sub-image, and then send the next layer of the code stream to the receiving end, and so on.

[0045] In another possible implementation, after sequentially encoding the multi-layer code stream of the first sub-image, the transmitting end may send the code stream of the first sub-image to the receiving end layer by layer according to the encoding order.

[0046] Accordingly, the transmitting end may send the code stream of the second sub-image to the receiving end layer by layer within the second preset time interval, where the second preset time interval is adjacent to the first preset time interval and is after the first preset time interval; accordingly, the receiving end may receive the code stream of the second sub-image from the transmitting end layer by layer within the second preset time interval.

[0047] To achieve the aforementioned goal of lowering transmission latency for video transmission by agreeing on and controlling the processing time for the encoding, transmission, and decoding steps of video data, an embodiment of the present application provides a transmission pipeline structure in which each step executed by the transmitter and receiver is agreed upon / limited to be completed within a fixed and identical processing time. That is, if the actual processing time of a step exceeds the allocated, fixed processing time for that step, execution of that step is stopped, and the next step is executed within the allocated, fixed processing time for the next step. In this way, multiple sub-images are processed sequentially through the transmission pipeline structure. While a sub-image is executed within the allocated, fixed processing time for a certain step, the next sub-image of that sub-image executes the previous step within the allocated, fixed processing time for the previous step. This approach can strictly control the end-to-end transmission latency of video data while reducing the processing latency between sub-images. Furthermore, the transmission pipeline structure provided by the embodiment of the present application ensures that the receiving end can display a highly continuous and acceptable video image by sacrificing the image quality of sub-images transmitted within one or more preset time intervals. That is to say, after transmission through an unstable channel, the receiving end can still be provided with acceptable or even high-quality display image quality. In other words, when the Wi-Fi bandwidth suddenly changes, the receiving end can still be provided with a video image that is as continuous as possible and has acceptable image quality.

[0048] For example, if the first frame of video data includes sub-image 1, sub-image 2, and sub-image 3, T1 to T5 represent five adjacent time intervals arranged in chronological order, and the lengths of T1 to T5 are the same and equal to the preset length.

[0049] Sub-image 1: Encoding is completed in T1, transmission is completed in T2, and decoding is completed in T3;

[0050] Sub-image 2: Encoded in T2, transmitted in T3, and decoded in T4;

[0051] Sub-image 3: Encoding is completed in T3, transmission is completed in T4, and decoding is completed in T5.

[0052] Optionally, in the embodiment of the present application, the processing time for acquiring and displaying video data can also be agreed upon and controlled.

[0053] For example, taking the first frame of video data including sub-image 1, sub-image 2, and sub-image 3 as an example, T1 to T7 represent seven adjacent time intervals arranged in chronological order, and the lengths of T1 to T7 are the same and are all equal to the preset length.

[0054] Sub-image 1: Acquisition is completed in T1, encoding is completed in T2, transmission is completed in T3, decoding is completed in T4, and display is completed in T5;

[0055] Sub-image 2: Acquisition is completed in T2, encoding is completed in T3, transmission is completed in T4, decoding is completed in T5, and display is completed in T6;

[0056] Sub-image 3: Acquisition is completed in T3, encoding is completed in T4, transmission is completed in T5, decoding is completed in T6, and display is completed in T7.

[0057] Since the processing time of the acquisition, encoding, decoding and display steps can basically be completed within the preset time interval through existing hardware design or software implementation, the end-to-end transmission delay can be strictly controlled as long as the processing time of the transmission step is controlled.

[0058] Several solutions for completing the transmission step within the preset time interval proposed in the embodiments of the present application are described in detail below.

[0059] Optionally, after the transmitting end performs layered encoding on the first sub-image, that is, at the end of the first preset time interval, the multi-layer code stream of the first sub-image has the following three transmission results.

[0060] Transmission result 1: The multi-layer code streams of the first sub-image are all transmitted successfully.

[0061] Transmission result two: at least one layer of the multi-layer code stream of the first sub-image is successfully transmitted.

[0062] Transmission result three: The transmission of the multi-layer code stream of the first sub-image fails.

[0063] In the case of transmission result 1 described above, if the multi-layer code stream of the first sub-image is successfully transmitted within the first preset time interval, the receiving end can decode the multi-layer code stream of the successfully transmitted first sub-image to obtain the first sub-image, and refresh and display it on the display interface. Similarly, each time the receiving end receives the multi-layer code stream of a sub-image, it can decode the multi-layer code stream of the sub-image to obtain the sub-image, and refresh and display the sub-image on the display interface. This can reduce the end-to-end latency of presenting the image to the user while ensuring the image quality and integrity of the image displayed on the receiving end.

[0064] It should be noted that the above-mentioned transmission results 2 and 3 may be due to unstable channel bandwidth or low channel bandwidth, resulting in part or all of the code stream of the first sub-image failing to be successfully transmitted within the first preset time interval, which may result in poor image quality or continuity displayed at the receiving end.

[0065] In the second transmission result, the transmitting end successfully transmits at least one layer of the multi-layer codestream of the first sub-image within the first preset time interval, i.e., transmission of all codestreams other than the at least one layer of the multi-layer codestream of the first sub-image fails. Accordingly, the receiving end successfully receives at least one layer of the multi-layer codestream of the first sub-image within the first preset time interval, i.e., reception of all codestreams other than the at least one layer of the multi-layer codestream of the first sub-image fails.

[0066] Optionally, the transmitting end may process the target code stream of the first sub-image within the first preset time interval in a variety of ways, where the target code stream includes the code stream that fails to be sent in the multi-layer code stream of the first sub-image, that is, the target code stream includes the code stream in the multi-layer code stream of the first sub-image except for the at least one layer of code stream.

[0067] In one possible implementation, the transmitting end may directly discard the target codestream and transmit the multi-layer codestream of the second sub-image within the second preset time interval. Accordingly, the receiving end may decode only the at least one layer of the successfully received codestream of the first sub-image to obtain the first sub-image without further waiting for the target codestream of the first sub-image, and may continue to receive the multi-layer codestream of the second sub-image from the transmitting end within the second preset time interval.

[0068] In this way, the transmission delay can be reduced while ensuring the image quality and smoothness of the image displayed on the receiving end as much as possible, and the transmission delay can be strictly controlled and managed.

[0069] In another possible implementation, the sending end may send the target code stream of the first sub-image within the remaining time of another preset time interval. The other preset time interval may be a preset time interval for sending other sub-images of the first frame image. The other sub-images are sub-images encoded or sent after the first sub-image, and there is still time remaining in the other preset time interval when the multi-layer code streams of the other sub-images are successfully sent.

[0070] In this way, it is ensured that the multi-layer code stream of the first sub-image can be sent to the receiving end, thereby providing a reference for the receiving end to subsequently decode other sub-images.

[0071] In one possible implementation, if there is still time remaining in the second preset interval when the multiple layers of code streams of the second sub-image are all sent successfully, the sending end can send the target code stream of the first sub-image to the receiving end layer by layer within the remaining time of the second preset time interval.

[0072] It should be noted that if, within the remaining time of the second preset time interval, some code streams in the target code stream of the first sub-image still fail to be sent, the sending end may continue to send the partial code streams within the remaining time of the next preset time interval, and so on, until all the multiple code streams of the first sub-image are successfully sent.

[0073] For example, if the first frame of video data includes sub-image 1, sub-image 2, and sub-image 3, T1 to T8 represent eight adjacent time intervals arranged in chronological order, and the lengths of T1 to T8 are the same and equal to the preset length.

[0074] Sub-image 1: Acquisition is completed in T1, encoding is completed in T2, transmission of at least one layer of the multi-layer code stream of sub-image 1 is completed in T3, decoding is completed in T4, and display is completed in T5.

[0075] Sub-image 2: Acquisition is completed within T2, encoding is completed within T3, and transmission is completed within T4-1. Since there is a remaining time T4-2 in T4, part or all of the remaining code stream of sub-image 1 can be transmitted within T4-2, decoding is completed within T5, and display is completed within T6.

[0076] Sub-image 3: Acquisition is completed in T3, encoding is completed in T4, transmission is completed in T5, decoding is completed in T6, and display is completed in T7.

[0077] In the third transmission scenario, if the transmitter fails to transmit the multi-layer codestream for the first sub-image within the first preset time interval, and then continues to transmit the multi-layer codestream for the second sub-image within the next preset time interval, the area where the first sub-image is located may not be displayed when the receiver displays the image, thereby affecting the user experience. Therefore, in this case, the multi-layer codestream for the first sub-image should be retransmitted as soon as possible, and then the multi-layer codestream for the second sub-image should be transmitted again. This ensures the integrity and smoothness of the image displayed by the receiver while minimizing transmission latency.

[0078] Optionally, the transmitting end and the receiving end may retransmit the multi-layer code stream of the first sub-image in a variety of ways, which is not limited in this embodiment of the present application.

[0079] In one possible implementation, the transmitting end may re-send the code stream of the first sub-image to the receiving end layer by layer within a second preset time interval, where the second preset time interval is adjacent to and after the first preset time interval; accordingly, the receiving end re-receives the code stream of the first sub-image from the transmitting end layer by layer within the second preset time interval.

[0080] Since the retransmission of the multi-layer code stream of the first sub-image occupies the second preset time interval originally used to transmit the multi-layer code stream of the second sub-image, the transmitting end can retransmit the multi-layer code stream of the first sub-image within the second preset time interval, and send the code stream of the second sub-image to the receiving end layer by layer within the third preset time interval, wherein the third preset time interval is adjacent to the second preset time interval and is after the second preset time interval; accordingly, the receiving end receives the code stream of the second sub-image from the transmitting end layer by layer within the third preset time interval.

[0081] For example, the first frame of video data includes sub-image 1, sub-image 2, and sub-image 3. T1 to T8 represent eight adjacent time intervals arranged in chronological order. The lengths of T1 to T8 are the same and are equal to the preset length.

[0082] Sub-image 1: Acquisition is completed within T1, encoding is completed within T2, initial transmission is performed within T3, if transmission fails within T3, retransmission is performed within T4, decoding is completed within T5, and display is completed within T6.

[0083] Sub-image 2: Acquisition is completed in T2, encoding is completed in T3, transmission is completed in T5, decoding is completed in T6, and display is completed in T7.

[0084] Sub-image 3: Acquisition is completed in T3, encoding is completed in T4, transmission is completed in T6, decoding is completed in T7, and display is completed in T8.

[0085] In another possible implementation, the sending end may also resend the multi-layer code stream of the first sub-image within the remaining time of other preset time intervals. The other preset time intervals may be a preset time interval for sending other sub-images of the first frame image. The other sub-images are sub-images encoded or sent after the first sub-image, and there is still time remaining in the other preset time intervals when the multi-layer code streams of the other sub-images are all sent successfully.

[0086] For example, if there is still time remaining in the second preset interval when the multi-layer code stream of the second sub-image is successfully sent, the sending end can resend the multi-layer code stream of the first sub-image layer by layer to the receiving end within the remaining time of the second preset time interval.

[0087] Optionally, before the transmitting end performs layered encoding on the first sub-image, the method may further include: the transmitting end receiving feedback information from the receiving end, the feedback information including at least one of a first quantity, a second quantity, or a first bandwidth, the first quantity indicating the number of code stream layers of a third sub-image received by the receiving end, the second quantity indicating the number of code stream layers of the third sub-image successfully decoded by the receiving end, and the first bandwidth indicating the bandwidth for transmitting the code stream of the third sub-image, the code stream of the third sub-image being obtained by performing layered encoding on the third sub-image; wherein, the plurality of sub-images include the third sub-image, and the receiving end has completed decoding of the code stream of the third sub-image, or the third sub-image belongs to a second frame image, and the second frame image is located before the first frame image in the frame sequence of the video.

[0088] It should be noted that the above only schematically lists the contents included in the feedback information. Optionally, the feedback information may also include other information that can feedback the status of the channel, which is not limited in the embodiments of the present application.

[0089] In a possible implementation manner, the receiving end may send the feedback information to the sending end within a preset time interval for displaying the first sub-image.

[0090] In another possible implementation, the transmission pipeline structure may further include a preset time interval for sending feedback information. The preset time interval for sending feedback information may be located between the preset time interval for decoding and the preset time interval for display.

[0091] Correspondingly, the transmitting end may perform layered coding on the first sub-image according to the feedback information to obtain the multi-layer code stream of the first sub-image.

[0092] Specifically, the transmitting end can determine the encoding parameters of the first sub-image based on the feedback information, where the encoding parameters include at least one of the starting Qp, number of layers, encoding mode, and reference frame of the first sub-image, and the encoding mode may include intra-frame encoding or inter-frame encoding; and based on the encoding parameters, perform layered encoding on the first sub-image to obtain the multi-layer code stream of the first sub-image.

[0093] In one possible implementation, the transmitting end may obtain statistical information based on the feedback information, where the statistical information may include at least one of the average coding bit rate of each layer of code stream within a first time length (within a rolling time window), the average channel bandwidth within the first time length, and the instantaneous coding bit rate or instantaneous channel bandwidth of each layer within the first time length; and determine the coding parameters of the first sub-image based on the statistical information.

[0094] For example: when the ratio of the average channel bandwidth to the average coding rate within the first time duration is high, or the ratio of the instantaneous channel bandwidth to the instantaneous coding rate is high, coding parameters for increasing image quality can be used; when the ratio of the average channel bandwidth to the average coding rate within the first time duration is low, or the ratio of the instantaneous channel bandwidth to the instantaneous coding rate is low, or when continuous code stream loss occurs, coding parameters for reducing the code rate can be used.

[0095] Optionally, the coding parameters for increasing image quality may include: increasing the starting Qp, reducing the number of layers, using inter-frame coding and intra-frame coding, etc.; the coding parameters for reducing the bit rate may include: reducing the starting Qp, increasing the number of layers, using only inter-frame coding, etc.

[0096] For another example, when the bandwidth is relatively stable during the first time period, the number of layers may be reduced; when the bandwidth fluctuates or changes greatly during the first time period, the number of layers may be increased.

[0097] It should be noted that the fewer the number of sub-image layers, the more complex the encoding method of each layer can be, and vice versa, the simpler the encoding method of each layer can be. For example, when the number of layers is small, inter-frame coding and intra-frame coding can be used, and when the number of layers is large, only inter-frame coding can be used.

[0098] Optionally, the encoding parameter may further include an encoding tool, wherein the encoding tool may include at least one of a deblocking filter, the number of alternative modes of intra-frame encoding, or sample adaptive offset (SAO).

[0099] It should be noted that when the number of layers is small, the coding tools used in each layer can be more complex, and conversely, the coding tools used in each layer can be simpler. For example, when the number of layers is small, a deblocking filter, a larger number of intra-frame coding alternative modes, and Sample Adaptive Offset (SAO) are used, while when the number of layers is large, only a smaller number of intra-frame coding alternative modes are used.

[0100] In existing video transmission applications, there is no requirement to provide feedback on the transmission results of each frame image / sub-image. Instead, possible image errors are mainly recovered by periodically using intra-frame coded frames. Since intra-frame coded frames use themselves as a reference, the use of intra-frame coded frames will result in a larger data volume and thus a higher bit rate.

[0101] However, in the embodiment of the present application, due to the channel conditions, such as bandwidth, there may be real-time fluctuations and changes, and the channel conditions directly affect the transmission delay of the image and the image quality of the displayed image. In the embodiment of the present application, the receiving end provides timely feedback on the transmission results of each sub-image, so that the transmitting end can dynamically adjust the encoding strategy according to the feedback information in a timely manner, thereby achieving higher video compression efficiency, reducing the requirements for bandwidth and transmission power consumption, and ultimately improving the displayed image quality, while avoiding image transmission errors. At the same time, the embodiment of the present application can reduce the use of intra-frame coding frames by using feedback signals for feedback, thereby achieving the effect of reducing the bit rate.

[0102] In a second aspect, an embodiment of the present application also provides a video transmission method, which includes: obtaining a first frame image of video data, the first frame image including multiple sub-images, the multiple sub-images including a first sub-image and a second sub-image; performing layered encoding on the first sub-image to obtain a multi-layer code stream of the first sub-image; sending the multi-layer code stream of the first sub-image to a receiving end; after completing the layered encoding of the first sub-image, performing layered encoding on the second sub-image to obtain a multi-layer code stream of the second sub-image; and sending the multi-layer code stream of the second sub-image to the receiving end.

[0103] It should be noted that the method can be applied to the video transmission system in the above-mentioned first aspect or any possible implementation of the first aspect, and executed by the sending end in the video transmission system.

[0104] It should also be noted that the method may include the steps performed by the sending end in the above-mentioned first aspect or any possible implementation of the first aspect.

[0105] Optionally, this method can be applied to short-range wireless screen projection scenarios. Short-range wireless screen projection refers to a technology that encodes and compresses video data generated by a mobile terminal with strong processing capabilities and sends it wirelessly to a device with weak processing capabilities but good display quality for display.

[0106] For example, taking the sending end as a mobile phone and the receiving end as VR glasses, the mobile phone can project the rendered game screen onto the display interface of the VR glasses.

[0107] For another example, taking the transmitting end as a laptop and the receiving end as a TV, the laptop can project the TV screen onto the display interface of the TV.

[0108] In a possible implementation, sending the multi-layer code stream of the first sub-image to the receiving end includes: sending the code stream of the first sub-image to the receiving end layer by layer within a first preset time interval.

[0109] In one possible implementation, the sending of the code stream of the first sub-image to the receiving end layer by layer within a first preset time interval includes: sending the code stream of the first sub-image to the receiving end layer by layer according to the encoding order of the multi-layer code stream of the first sub-image within the first preset time interval.

[0110] In one possible implementation, when the multi-layer code stream of the first sub-image fails to be sent within the first preset time interval, the method further includes: sending the code stream of the first sub-image to the receiving end layer by layer within a second preset time interval, and the second preset time interval is adjacent to the first preset time interval and after the first preset time interval.

[0111] In a possible implementation, at least one layer of the multi-layer code stream of the first sub-image is successfully sent within the first preset time interval, and the method further includes: discarding the code streams other than the at least one layer of the multi-layer code stream of the first sub-image.

[0112] In one possible implementation, sending the multi-layer code stream of the second sub-image to the receiving end includes: sending the code stream of the second sub-image to the receiving end layer by layer within a second preset time interval, and the second preset time interval is adjacent to the first preset time interval and after the first preset time interval.

[0113] In a possible implementation, if the multi-layer code streams of the first sub-image fail to be sent within the first preset time interval, and the multi-layer code streams of the second sub-image are successfully sent, and there is still time remaining in the second preset time interval, the method further includes:

[0114] During the remaining time of the second preset time interval, the code stream of the first sub-image is sent to the receiving end layer by layer.

[0115] In one possible implementation, at least one layer of the multi-layer codestream of the first sub-image is successfully sent within the first preset time interval, and when the multi-layer codestream of the second sub-image is successfully sent, there is still time remaining in the second preset time interval. The method further includes: sending the codestreams of the multi-layer codestream of the first sub-image except the at least one layer of the codestream to the receiving end layer by layer within the remaining time of the second preset time interval.

[0116] In one possible implementation, before performing layered coding on the first sub-image to obtain a multi-layer codestream for the first sub-image, the method further includes: receiving feedback information from the receiving end, the feedback information including at least one of a first quantity, a second quantity, or a first bandwidth, the first quantity indicating the number of codestream layers of a third sub-image received by the receiving end, the second quantity indicating the number of codestream layers of the third sub-image successfully decoded by the receiving end, and the first bandwidth indicating the bandwidth used for transmitting the codestream of the third sub-image, the codestream of the third sub-image being obtained by performing layered coding on the third sub-image; wherein the plurality of sub-images includes the third sub-image, and the receiving end has completed decoding of the codestream of the third sub-image, or the third sub-image belongs to a second frame image, and the second frame image is located before the first frame image in a frame sequence of the video; performing layered coding on the first sub-image to obtain the multi-layer codestream for the first sub-image includes: performing layered coding on the first sub-image according to the feedback information to obtain the multi-layer codestream for the first sub-image.

[0117] In one possible implementation, performing layered encoding on the first sub-image based on the feedback information to obtain the multi-layer code stream of the first sub-image includes: determining encoding parameters of the first sub-image based on the feedback information, the encoding parameters including at least one of a starting Qp, a number of layers, an encoding mode, or a reference frame of the first sub-image; and performing layered encoding on the first sub-image based on the encoding parameters to obtain the multi-layer code stream of the first sub-image.

[0118] In a possible implementation, the sub-image is a slice, a tile, a block, a macroblock (MB), or a largest coding block (LCU) of a frame image.

[0119] In a third aspect, an embodiment of the present application also provides another video transmission method, which includes: receiving a multi-layer code stream of a first sub-image of a first frame image from a transmitting end; decoding the multi-layer code stream of the first sub-image to obtain the first sub-image; displaying the first sub-image; receiving a multi-layer code stream of a second sub-image of the first frame image from the transmitting end; decoding the multi-layer code stream of the second sub-image to obtain the second sub-image; and displaying the second sub-image.

[0120] It should be noted that the method can be applied to the video transmission system in the above-mentioned first aspect or any possible implementation of the first aspect, and executed by the receiving end in the video transmission system.

[0121] It should also be noted that the method may include the steps performed by the receiving end in the above-mentioned first aspect or any possible implementation of the first aspect.

[0122] Optionally, this method can be applied to short-range wireless screen projection scenarios. Short-range wireless screen projection refers to a technology that encodes and compresses video data generated by a mobile terminal with strong processing capabilities and sends it wirelessly to a device with weak processing capabilities but good display quality for display.

[0123] For example, taking the sending end as a mobile phone and the receiving end as VR glasses, the mobile phone can project the rendered game screen onto the display interface of the VR glasses.

[0124] For another example, taking the transmitting end as a laptop and the receiving end as a TV, the laptop can project the TV screen onto the display interface of the TV.

[0125] In one possible implementation, decoding the multi-layer code stream of the first sub-image to obtain the first sub-image includes: decoding the code stream of the first sub-image layer by layer according to the encoding order of the multi-layer code stream of the first sub-image to obtain the first sub-image.

[0126] In a possible implementation, the receiving of the multi-layer code stream of the first sub-image of the first frame image from the transmitter includes: receiving the code stream of the first sub-image from the transmitter layer by layer within a first preset time interval.

[0127] In one possible implementation, when the multi-layer code stream of the first sub-image is not successfully received within the first preset time interval, the method further includes: receiving the code stream of the first sub-image from the transmitting end layer by layer within a second preset time interval, and the second preset time interval is adjacent to the first preset time interval and after the first preset time interval.

[0128] In one possible implementation, the receiving of the multi-layer code stream of the second sub-image of the first frame image from the transmitting end includes: receiving the code stream of the second sub-image from the transmitting end layer by layer within a second preset time interval, where the second preset time interval is adjacent to the first preset time interval and is after the first preset time interval.

[0129] In one possible implementation, if the multi-layer code stream of the first sub-image is not successfully received within the first preset time interval, and if the multi-layer code stream of the second sub-image is successfully received, there is still time remaining in the second preset time interval. The method further includes: receiving the code stream of the first sub-image from the transmitting end layer by layer within the remaining time of the second preset time interval.

[0130] In one possible implementation, when at least one layer of the multi-layer codestream of the first sub-image is successfully received within the first preset time interval and the multi-layer codestream of the second sub-image is successfully received and there is still time remaining in the second preset time interval, the method further includes: receiving, within the remaining time of the second preset time interval, a target codestream from the transmitting end layer by layer, the target codestream including the codestream of the multi-layer codestream of the first sub-image except for the at least one layer of the codestream.

[0131] In a possible implementation, the method further includes: sending feedback information to the transmitting end, the feedback information including at least one of a first quantity, a second quantity, or a first bandwidth, the first quantity indicating the number of received code stream layers of the first sub-image, the second quantity indicating the number of successfully decoded code stream layers of the first sub-image, and the first bandwidth indicating the bandwidth for transmitting the code stream of the first sub-image.

[0132] In a possible implementation, the sub-image is a slice, a tile, a block, a macroblock (MB), or a largest coding block (LCU) of a frame image.

[0133] In a fourth aspect, an embodiment of the present application also provides a video transmission device, which is used to execute the method described in the above-mentioned second aspect or any possible implementation of the second aspect, and the device includes a unit for executing the method described in the above-mentioned second aspect or any possible implementation of the second aspect.

[0134] In the fifth aspect, an embodiment of the present application also provides a video transmission device, which is used to execute the method described in the third aspect or any possible implementation of the third aspect, and the device includes a unit for executing the method described in the third aspect or any possible implementation of the third aspect.

[0135] In the sixth aspect, an embodiment of the present application also provides a video transmission device, which includes: a processor and a communication interface, the processor and the communication interface communicating with each other through an internal connection path, the processor being used to call and run instructions from the communication interface, and when the processor executes the instruction, it implements the method described in the above second aspect or any possible implementation method of the second aspect.

[0136] Optionally, the video transmission device may further include a memory, which is used to store the above instructions.

[0137] In a possible implementation, the video transmission device is a terminal.

[0138] In the seventh aspect, an embodiment of the present application also provides a video transmission device, which includes: a processor and a communication interface, the processor and the communication interface communicating with each other through an internal connection path, the processor being used to call and run instructions from the communication interface, and when the processor executes the instruction, it implements the method described in the above third aspect or any possible implementation method of the third aspect.

[0139] Optionally, the video transmission device may further include a memory, which is used to store the above instructions.

[0140] In a possible implementation, the video transmission device is a terminal.

[0141] In an eighth aspect, an embodiment of the present application further provides a terminal, which includes the video transmission device provided in the sixth aspect and / or the video transmission device provided in the seventh aspect.

[0142] In a ninth aspect, an embodiment of the present application further provides a computer-readable storage medium for storing a computer program, wherein the computer program includes instructions for implementing the method described in the second aspect or any possible implementation of the second aspect.

[0143] In the tenth aspect, an embodiment of the present application further provides a computer-readable storage medium for storing a computer program, which includes instructions for implementing the method described in the third aspect or any possible implementation of the third aspect.

[0144] In the eleventh aspect, an embodiment of the present application further provides a computer program product, which includes instructions. When the instructions are executed on a computer, the computer implements the method in the second aspect or any possible implementation of the second aspect.

[0145] In the twelfth aspect, an embodiment of the present application further provides a computer program product, which includes instructions. When the instructions are executed on a computer, the computer implements the method in the third aspect or any possible implementation of the third aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0146] Figure 1 A schematic block diagram of a video transmission system 100 according to an embodiment of the present application is provided;

[0147] Figure 2 A schematic diagram of existing scalable coding is provided;

[0148] Figure 3 A schematic flow chart of a video transmission method 200 according to an embodiment of the present application is provided;

[0149] Figure 4A schematic diagram of the transmission pipeline structure of an embodiment of the present application is provided;

[0150] Figure 5 A schematic diagram of another transmission pipeline structure according to an embodiment of the present application is provided;

[0151] Figure 6 A schematic diagram of another transmission pipeline structure according to an embodiment of the present application is provided;

[0152] Figure 7 A schematic diagram of another transmission pipeline structure according to an embodiment of the present application is provided;

[0153] Figure 8 A schematic block diagram of a video transmission device 300 according to an embodiment of the present application is provided;

[0154] Figure 9 A schematic block diagram of another video transmission device 400 according to an embodiment of the present application is provided;

[0155] Figure 10 A schematic block diagram of another video transmission system 500 according to an embodiment of the present application is provided;

[0156] Figure 11 A schematic block diagram of a chip 600 according to an embodiment of the present application is provided. DETAILED DESCRIPTION

[0157] The embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application.

[0158] Figure 1 FIG. 1 shows a schematic diagram of a video transmission system 100 provided in an embodiment of the present application. Figure 1 As shown, the system 100 includes a transmitting end 110 and a receiving end 120 .

[0159] The transmitting end 110 is configured to obtain video data, encode the video data to obtain encoded video data, and send the encoded video data to the receiving end 120 .

[0160] The receiving end 120 is configured to receive the encoded video data from the transmitting end 110 , decode the encoded video data to obtain a video image, and display the video image through a display interface.

[0161] It should be noted that in the field of video coding, the terms "picture," "frame," or "image" can be used synonymously. Video encoding is performed at the transmitting end and typically involves processing (e.g., through compression) the original video picture to reduce the amount of data required to represent it, thereby enabling more efficient storage and / or transmission. Video decoding is performed at the receiving end and typically involves reversing the processing performed by the encoder to reconstruct the video picture.

[0162] In a possible implementation manner, the transmitting end or the receiving end may include a terminal.

[0163] It should be noted that the terminal described in the embodiments of the present application may be a device that provides video / data connectivity to users, such as: a notebook or laptop computer, a mobile phone, a smart phone, a tablet or tablet computer, a wearable device, a camera, a desktop computer, a set-top box, a television, a camera, a vehicle-mounted device, a display device, a digital media player, a video game console, a video streaming device (such as a content service server or a content distribution server), a vehicle-mounted computer, a wireless communication device, etc., and may not use or use any type of operating system.

[0164] It should also be noted that when the terminal device is a wearable device, the wearable device can also be a general term for wearable devices that are intelligently designed and developed using wearable technology for daily wear, such as glasses, gloves, watches, clothing and shoes. A wearable device is a portable device that is worn directly on the body or integrated into the user's clothes or accessories. Wearable devices are not just hardware devices, but also achieve powerful functions through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include those that are full-featured, large in size, and can achieve complete or partial functions without relying on smartphones, such as smart watches or smart glasses, as well as those that only focus on a certain type of application function and need to be used in conjunction with other devices such as smartphones, such as various smart bracelets and smart jewelry for vital sign monitoring.

[0165] Optionally, the video data may be transmitted between the transmitting end and the receiving end in a variety of ways, which are not limited in this embodiment of the present application.

[0166] In a possible implementation, the video data may be transmitted between the transmitting end and the receiving end via a wired manner.

[0167] It should be noted that the above-mentioned wired manner can be realized through data line connection or internal bus connection.

[0168] In another possible implementation, the video data may be transmitted wirelessly between the transmitting end and the receiving end.

[0169] It should be noted that the above-mentioned wireless method can be implemented through a communication network, which can be a local area network, a wide area network transferred through a relay device, or a combination of a local area network and a wide area network. When the communication network is a local area network, illustratively, the communication network can be a WiFi hotspot network, a WiFi P2P network, a Bluetooth network, a ZigBee network, a near field communication (NFC) network, or a general short-range communication network that may be possible in the future. When the communication network is a wide area network, illustratively, the communication network can be a third-generation wireless telephone technology (3G) network, a fourth-generation mobile communication technology (4G) network, a fifth-generation mobile communication technology (5G) network, a future-evolved public land mobile network (PLMN) or the Internet, etc., and the embodiments of the present application are not limited to this.

[0170] In one possible implementation, the system 100 can be applied to short-range wireless screen projection scenarios. Short-range wireless screen projection refers to a technology that encodes and compresses video data generated by a mobile terminal with strong processing capabilities and sends it wirelessly to a device with weak processing capabilities but good display quality for display.

[0171] For example, taking the sending end as a mobile phone and the receiving end as VR glasses, the mobile phone can project the rendered game screen onto the display interface of the VR glasses.

[0172] For another example, taking the transmitting end as a laptop and the receiving end as a TV, the laptop can project the TV screen onto the display interface of the TV.

[0173] In these scenarios, due to the interaction of video data between the transmitter and receiver, transmission latency is a significant factor affecting system performance. Furthermore, because wireless channels are susceptible to multiple factors, such as the operating environment, noise, and interference signals, transmission bandwidth instability can lead to data loss and image quality issues. Therefore, interference immunity is another crucial factor affecting system performance.

[0174] In order to solve the above problems, the existing technology uses a scalable video coding protocol to encode and decode video data. Scalable video coding is intuitively reflected in Bitrate In general, layered coding and selective transmission of video data are the main means to achieve scalability.

[0175] The so-called layered coding is to divide the video in time, space and quality, and output a multi-layer code stream with a reference relationship. The multi-layer code stream can include a base layer and at least one enhancement layer. The data of the base layer can enable the decoder to decode the basic video content completely normally, but the video obtained based on the data of the base layer is not good. image When the channel is limited or the channel environment is complex, it can ensure that the decoding end can receive smooth video that can be watched. image When the channel environment is good or the channel resources are abundant, the enhancement layer data can be transmitted to improve the image quality. The image quality can be evaluated and measured in terms of resolution, image quality, frame rate, etc.

[0176] It should be noted that in the existing video transmission method, the transmitter performs layered encoding on a frame of image based on the H.264 scalable video coding (SVC) protocol and the H.265 scalable high-efficiency video coding (SHVC) protocol to obtain a multi-layer code stream with a reference relationship. Taking the transmitter performing multi-layer encoding on the first frame of video data as an example, Figure 2 FIG. 4 shows a flowchart of the existing layered coding using the SVC / SHVC protocol standard. Figure 2 As shown, the sending end processes the first frame image (i.e. Figure 2 When performing layered coding on frame 1 in the image, the base layer code stream of the first frame is first coded, the base layer code stream is then coded to obtain the code stream of enhancement layer 1, and the enhancement layer 2 code stream is then coded to obtain the code stream of enhancement layer 2 with reference to the code stream of enhancement layer 1. The reference relationship between the base layer, the enhancement layer 1, and the enhancement layer 2 is as follows: Figure 2 As shown by the arrow in the figure, the coded video data indicated by the tail of the arrow is coded with reference to the coded video data indicated by the head of the arrow. After obtaining the 3-layer code stream of the first frame image, the transmitting end sends the 3-layer code stream of the first frame image to the receiving end layer by layer, and continues to process the next frame image (i.e. Figure 2 Frame 2) in the layered encoding is performed.

[0177] In one possible implementation, in an existing video transmission method, when performing layered encoding on the first frame image, the transmitter may divide the first frame image into multiple sub-images, encode each of the multiple sub-images to obtain a sub-stream of the base layer corresponding to each sub-image, thereby obtaining a code stream of the base layer of the first frame image, the code stream of the base layer including the sub-stream of the base layer corresponding to each sub-image of the multiple sub-images, and then encode based on the sub-stream of the base layer corresponding to each sub-image to obtain a sub-stream of enhancement layer 1 corresponding to each sub-image, thereby obtaining a code stream of enhancement layer 1 of the first frame image, and so on, to obtain a code stream of enhancement layer 2 of the first frame image.

[0178] Accordingly, after successfully receiving the three-layer code stream of the first frame image, the receiving end decodes the three-layer code stream to obtain the first frame image and displays the first frame image. Therefore, the transmission delay is relatively large.

[0179] It should be noted that in layered coding, the image quality corresponding to the referenced codestream layer is lower than the image quality corresponding to the codestream layer that referenced it. In other words, in the coding order, the image quality corresponding to the multiple layers of the image codestream increases successively, that is, the image quality of the base layer is lower than that of enhancement layer 1, and the image quality of enhancement layer 1 is lower than that of enhancement layer 2.

[0180] It should also be noted that the image quality of the multi-layer code stream pairs encoded by the transmitter is different. Correspondingly, the image quality of the image that can be reconstructed by the receiver based on each layer of the code stream is different. Therefore, if the receiver wants to reconstruct an image with high image quality, it needs to combine more layers of code streams.

[0181] It should be noted that Figure 2 The video data schematically illustrates three frames of images, each of which is encoded to produce three layers of bitstreams. Alternatively, the video data may include another number of frames of images, and each of which may be encoded in layers to produce another number of multi-layer bitstreams. This embodiment of the present application is not limited thereto.

[0182] It should also be noted that Figure 2 The reference relationship of the encoded video data in the layered encoding process is only schematically shown. Optionally, the encoded video data may also have other reference relationships in the layered encoding process, which is not limited in the embodiment of the present application.

[0183] It should be noted that the sub-image described in the embodiment of the present application can be understood as the granularity of video data encoded by the transmitter or decoded by the receiver, for example, the minimum granularity.

[0184] Optionally, the sub-image may be, but is not limited to, a slice, a tile, a block, a macroblock (MB) or a largest coding unit (LCU) of a frame image, which is not limited in the embodiments of the present application.

[0185] For example: Figure 2 Taking the first frame image of the video data shown in as being divided into three sub-images as an example, the encoding order in which the transmitting end uses the existing SVC / SHVC protocol to perform layered encoding on the video data can be as shown in Table 1 below. After completing the layered encoding, the transmitting end can send the first frame image to the receiving end layer by layer according to the encoding order.

[0186] Table 1

[0187] Time Unit Existing SVC / SHVC encoding order 1 Frame (1) Sub-image (0) Codestream layer (0) 2 Frame (1) Sub-image (1) Codestream layer (0) 3 Frame (1) Sub-image (2) Codestream layer (0) 4 Frame (1) Sub-image (0) Codestream layer (1) 5 Frame (1) Sub-image (1) Codestream layer (1) 6 Frame (1) Sub-image (2) Codestream layer (1) 7 Frame (1) Sub-image (0) Codestream layer (2) 8 Frame (1) Sub-image (1) Codestream layer (2) 9 Frame (1) Sub-image (2) Codestream layer (2)

[0188] However, when using the above-mentioned video encoding method and sending method, if the receiving end wants to reconstruct the first frame image with higher image quality, it is necessary to decode the multi-layer code stream of the first frame image (that is, the 3-layer code stream of the above-mentioned first frame image). That is, it is necessary to wait until the multi-layer code stream of the frame image is successfully received or the transmission time specified in the communication protocol times out before the frame image can be displayed. Therefore, the transmission delay is relatively large.

[0189] An embodiment of the present application provides a video transmission method that can reduce the end-to-end transmission delay of video data, especially reduce the delay from obtaining video data from the sending end to displaying the video data to the user at the receiving end, and improve the robustness to channel changes.

[0190] Figure 3 FIG. 2 shows a schematic flow chart of a video transmission method 200 provided in an embodiment of the present application. The method 200 can be applied to Figure 1 The video transmission system 100 shown in the figure can be applied to the scenario of short-range wireless screen projection.

[0191] S210: A transmitting end obtains a first frame image of video data, where the first frame image includes a plurality of sub-images, and the plurality of sub-images include a first sub-image and a second sub-image.

[0192] It should be noted that the video data may include multiple frames of images arranged in a frame sequence, and the multiple frames of images include the first frame of image and the second frame of image.

[0193] Optionally, the sending end may obtain the video data in a variety of ways, which is not limited in this embodiment of the present application.

[0194] In a possible implementation, the sending end may receive the video data from the first device.

[0195] For example, the first device may be a camera device, and the transmitting end may receive the video data captured by the camera device.

[0196] For another example, the first device may be a video processing device, and the transmitting end may receive the video data processed by the video processing device.

[0197] For another example, the first device may be a video head end, and the transmitting end may receive the video data generated by the video head end.

[0198] In another possible implementation, the sending end may generate the video data itself.

[0199] For example, when the sending end includes a camera device, the sending end can obtain the video data captured by the camera device.

[0200] For another example, when the sending end is a video head end, the sending end may generate the video data.

[0201] For another example, when the sending end includes a video processing module, the sending end may obtain the video data processed by the video processing module.

[0202] In another possible implementation, the sending end may obtain original video data, and pre-process the original video data to obtain processed video data.

[0203] For example, the pre-processing may include at least one of retouch, color format conversion, color grading, or denoising.

[0204] S220: The transmitting end performs layered coding on the first sub-image to obtain a multi-layer code stream of the first sub-image.

[0205] Optionally, before S220 , the transmitting end may divide the first frame image into the multiple sub-images.

[0206] It should be noted that the sub-image described in the embodiment of the present application can be understood as the granularity of video data encoded by the transmitter or decoded by the receiver, for example, the minimum granularity.

[0207] Optionally, the sub-image may be, but is not limited to, a slice, tile, block, MB or LCU of a frame image, which is not limited in the embodiment of the present application.

[0208] It should be noted that, when using the video transmission method provided in the embodiment of the present application, the smaller the encoding granularity selected by the transmitter when performing layered encoding on the frame image, the smaller the corresponding encoding delay.

[0209] For example, when the transmitter performs layered encoding on a frame image at block, MB, or LCU as the encoding granularity, lower encoding delay can be achieved.

[0210] For another example, when the transmitter performs layered encoding on a frame image with slice or tile as the encoding granularity, it can be compatible with the existing SVC and SHVC standards.

[0211] Optionally, the transmitting end may perform layered encoding on the first sub-image based on multiple video encoding protocols, which is not limited in the embodiment of the present application.

[0212] In a possible implementation, the sending end may perform hierarchical encoding on the first sub-image based on the H.264 scalable video coding (SVC) protocol and the H.265 scalable high-efficiency video coding (SHVC) protocol.

[0213] In another possible implementation, the sending end may perform layered encoding on the first sub-image based on a future scalable video coding protocol, such as the H.266 scalable video coding protocol.

[0214] It should be noted that the multi-layer code streams of the first sub-image have different reference relationships. Figure 2 Taking the reference relationship shown in as an example, the transmitting end first encodes the base layer of the first sub-image, then encodes the enhancement layer 1 with reference to the base layer, and then encodes the enhancement layer 2 with reference to the enhancement layer 1.

[0215] It should also be noted that during layered coding, the image quality corresponding to the referenced bitstream layer is lower than the image quality corresponding to the bitstream layer that references the referenced layer. In other words, the image quality (or bitrate) corresponding to the multi-layer bitstream increases in sequence according to the coding order.

[0216] It should also be noted that the image quality corresponding to the code stream layer described in the embodiment of the present application may include at least one of resolution, image quality or frame rate.

[0217] For example, image quality includes resolution. The higher the resolution, the higher the image quality.

[0218] For another example, image quality includes resolution and image quality. When the resolution is the same, the better the image quality, the higher the image quality.

[0219] S230, the transmitting end sends the multi-layer code stream of the first sub-image to the receiving end; correspondingly, the receiving end receives the multi-layer code stream of the first sub-image from the transmitting end.

[0220] S240: The receiving end decodes the multi-layer code stream of the first sub-image to obtain the first sub-image.

[0221] S250: The receiving end displays the first sub-image.

[0222] Optionally, before S250 , the receiving end may perform post-processing on the first sub-image to obtain the processed first sub-image, and display the processed first sub-image.

[0223] For example, the post-processing may include at least one of color format conversion, color grading, retouch, or resampling.

[0224] S260: The transmitting end performs layered coding on the second sub-image to obtain a multi-layer code stream of the second sub-image.

[0225] It should be noted that S260 is executed after S220, that is, the second sub-image is encoded after the multi-layer encoding is completed for the first sub-image.

[0226] Optionally, the transmitting end and the receiving end may determine the encoding order of the second sub-image and the first sub-image and their positions in the first frame image in a variety of ways, which is not limited in the embodiment of the present application.

[0227] In a possible implementation, a method for dividing a frame image into multiple sub-images and a coding order for each sub-image in the frame image may be defined in a video coding protocol.

[0228] In another possible implementation, the transmitting end and the receiving end may pre-agree on a method for dividing a frame image into multiple sub-images and a coding order for each sub-image in the frame image.

[0229] In another possible implementation, the transmitting end may send instruction information to the receiving end, where the instruction information is used to indicate a method for dividing the frame image into multiple sub-images and an encoding order for each sub-image in the frame image. Accordingly, the receiving end may receive the instruction information from the transmitting end and determine the method for dividing the frame image into multiple sub-images and an encoding order for each sub-image in the frame image based on the instruction information.

[0230] For example, the transmitting end may carry the indication information in the control information of the sub-image code stream.

[0231] S270, the transmitting end transmits the multi-layer code stream of the second sub-image to the receiving end; correspondingly, the receiving end receives the multi-layer code stream of the second sub-image from the transmitting end.

[0232] S280: The receiving end decodes the multi-layer code stream of the second sub-image to obtain the second sub-image.

[0233] S290: The receiving end displays the second sub-image.

[0234] Optionally, before S290 , the receiving end may perform post-processing on the two sub-images to obtain the processed second sub-image, and display the processed second sub-image.

[0235] It should be noted that S220~S250 is the processing process of the first sub-image, and S260~S290 is the processing process of the second sub-image. Among them, the processing process of the second sub-image is similar to that of the first sub-image. You can refer to the following introduction to the processing process of the first sub-image. To avoid repetition, it will not be repeated here.

[0236] It should be noted that in order to better reduce the delay of video transmission, the processing time of the encoding, transmission and decoding steps of the video data can be agreed upon and controlled in the embodiments of the present application.

[0237] Optionally, S230 may include: the transmitting end may send the code stream of the first sub-image to the receiving end layer by layer within a first preset time interval; correspondingly, the receiving end may receive the code stream of the first sub-image from the transmitting end layer by layer within the first preset time interval.

[0238] That is, the transmitting end sends the code stream of the first sub-image to the receiving end layer by layer starting from the start time of the first preset time interval, and stops sending at the end time of the first preset time interval. Correspondingly, the receiving end receives the code stream of the first sub-image from the transmitting end layer by layer starting from the start time of the first preset time interval, and stops receiving at the end time of the first preset time interval.

[0239] In a possible implementation, the transmitting end may send the code stream of the first sub-image to the receiving end layer by layer according to the coding order of the multi-layer code stream of the first sub-image.

[0240] During the layered coding process, the image quality of the codestream obtained by encoding the first sub-image is lower than that of the codestream obtained by encoding the second sub-image. However, due to the lower image quality of the codestream obtained by encoding the first sub-image, the transmission delay of the codestream obtained by encoding the first sub-image layer by layer helps ensure that the user is provided with an acceptable display image of high image quality in unstable channel conditions, thereby improving robustness to channel variations.

[0241] It should be noted that, since the multi-layer code streams of the first sub-image have a reference relationship during encoding, and the transmitting end sends the code stream of the first sub-image layer by layer according to the encoding order of the multi-layer code stream of the first sub-image, therefore, in S240, the receiving end needs to decode the code stream of the first sub-image layer by layer according to the encoding order of the multi-layer code stream of the first sub-image.

[0242] Optionally, the receiving end may determine the encoding order of the multi-layer code stream of the first sub-image in a variety of ways, which is not limited in this embodiment of the present application.

[0243] In a possible implementation, if the transmission order of the code stream is not affected by factors such as channel conditions, the receiving end may use the order in which the multi-layer code stream of the first sub-image is received as the encoding order. In other words, the receiving order is the encoding order.

[0244] In another possible implementation, if the transmission order of the code stream is affected by factors such as channel conditions, the transmitting end may carry indication information for indicating the encoding order of each layer of the code stream in each layer of the code stream. Accordingly, when the receiving end receives each layer of the code stream of the first sub-image, it can determine the encoding order of each layer of the code stream based on the indication information carried by each layer of the code stream.

[0245] Optionally, the sending end may send the code stream of the first sub-image to the receiving end in a variety of ways according to the coding order of the multi-layer code stream of the first sub-image, which is not limited in this embodiment of the present application.

[0246] In one possible implementation, the transmitting end may directly send each layer of the code stream of the first sub-image to the receiving end after encoding the layer of the code stream, and then continue to encode the next layer of the code stream of the first sub-image, and then send the next layer of the code stream to the receiving end, and so on.

[0247] In another possible implementation, after sequentially encoding the multi-layer code stream of the first sub-image, the transmitting end may send the code stream of the first sub-image to the receiving end layer by layer according to the encoding order.

[0248] Accordingly, S270 may include: the transmitting end may send the code stream of the second sub-image to the receiving end layer by layer within the second preset time interval, and the second preset time interval is adjacent to the first preset time interval and after the first preset time interval; accordingly, the receiving end may receive the code stream of the second sub-image from the transmitting end layer by layer within the second preset time interval.

[0249] To achieve the aforementioned goal of lowering transmission latency for video transmission by agreeing on and controlling the processing time for the encoding, transmission, and decoding steps of video data, an embodiment of the present application provides a transmission pipeline structure in which each step executed by the transmitter and receiver is agreed upon / limited to be completed within a fixed and identical processing time. That is, if the actual processing time of a step exceeds the allocated, fixed processing time for that step, execution of that step is stopped, and the next step is executed within the allocated, fixed processing time for the next step. In this way, multiple sub-images are processed sequentially through the transmission pipeline structure. While a sub-image executes a step within the allocated, fixed processing time for that step, the next sub-image of that sub-image executes the previous step within the allocated, fixed processing time for the previous step. This approach allows for strict control of the end-to-end transmission latency of video data while reducing the processing latency between sub-images. Furthermore, the transmission pipeline structure provided by the embodiment of the present application ensures that the receiver can display a highly continuous and acceptable video image by sacrificing the image quality of sub-images transmitted within one or more preset time intervals. That is to say, after transmission through an unstable channel, the receiving end can still be provided with acceptable or even high-quality display image quality. In other words, when the Wi-Fi bandwidth suddenly changes, the receiving end can still be provided with a video image that is as continuous as possible and has acceptable image quality.

[0250] For example, the first frame of video data includes three sub-images (sub-image 1, sub-image 2, and sub-image 3). Figure 4 FIG. 1 shows the transmission pipeline structure provided by the embodiment of the present application. Figure 4 As shown, T1 to T5 represent five adjacent time intervals, and the lengths of T1 to T5 are the same and are equal to the preset length.

[0251] Sub-image 1: Encoding is completed in T1, transmission is completed in T2, and decoding is completed in T3;

[0252] Sub-image 2: Encoded in T2, transmitted in T3, and decoded in T4;

[0253] Sub-image 3: Encoding is completed in T3, transmission is completed in T4, and decoding is completed in T5.

[0254] It should be noted that the first frame image described in the embodiment of the present application can be Figure 4 The first sub-image in the first frame image can be Figure 4 The second sub-image can be Figure 4 In the sub-image 2, the first preset time interval can be Figure 4In T2, the second preset time interval can be Figure 4 T3 in.

[0255] It should also be noted that Figure 4 The transmission pipeline structure of the first frame image is schematically shown in FIG. The transmission pipeline structure of other frame images in the frame sequence of video data is the same as that of FIG. Figure 4 To avoid repetition, similar details will not be given here.

[0256] Optionally, in the embodiment of the present application, the processing time for acquiring and displaying video data can also be agreed upon and controlled.

[0257] For example, taking the first frame of video data including three sub-images (sub-image 1, sub-image 2 and sub-image 3) as an example, Figure 5 Another transmission pipeline structure provided by an embodiment of the present application is shown. Figure 5 The transmission pipeline structure shown is in Figure 4 The transmission pipeline structure shown in FIG5 adds the processing of sub-image acquisition and display steps. Figure 5 As shown, T1 to T7 represent 7 adjacent time intervals, and the lengths of T1 to T7 are the same and are equal to the preset length.

[0258] Sub-image 1: Acquisition is completed in T1, encoding is completed in T2, transmission is completed in T3, decoding is completed in T4, and display is completed in T5;

[0259] Sub-image 2: Acquisition is completed in T2, encoding is completed in T3, transmission is completed in T4, decoding is completed in T5, and display is completed in T6;

[0260] Sub-image 3: Acquisition is completed in T3, encoding is completed in T4, transmission is completed in T5, decoding is completed in T6, and display is completed in T7.

[0261] It should be noted that the first frame image described in the embodiment of the present application can be Figure 5 The first sub-image in the first frame image can be Figure 5 The second sub-image can be Figure 5 In the sub-image 2, the first preset time interval can be Figure 5 In T3, the second preset time interval can be Figure 5 T4 in.

[0262] It should be noted that the above Figure 4 or Figure 5 The length of the time interval T can be set in combination with the time required for the sending end and the receiving end to execute each step. For example, the preset length of each time interval can be the maximum time required for executing each step.

[0263] Since the processing time of the acquisition, encoding, decoding and display steps can basically be completed within the preset time interval through existing hardware design or software implementation, the end-to-end transmission delay can be strictly controlled as long as the processing time of the transmission step is controlled.

[0264] Several solutions for completing the transmission step within the preset time interval proposed in the embodiments of the present application are described in detail below.

[0265] Optionally, in S230, at the end of the first preset time interval, the multi-layer code stream of the first sub-image has the following three transmission results.

[0266] Transmission result 1: The multi-layer code streams of the first sub-image are all transmitted successfully.

[0267] Transmission result two: at least one layer of the multi-layer code stream of the first sub-image is successfully transmitted.

[0268] Transmission result three: The transmission of the multi-layer code stream of the first sub-image fails.

[0269] In the case of transmission result 1 described above, if the multi-layer code stream of the first sub-image is successfully transmitted within the first preset time interval, the receiving end can decode the multi-layer code stream of the successfully transmitted first sub-image to obtain the first sub-image, and refresh and display it on the display interface. Similarly, each time the receiving end receives the multi-layer code stream of a sub-image, it can decode the multi-layer code stream of the sub-image to obtain the sub-image, and refresh and display the sub-image on the display interface. This can reduce the end-to-end latency of presenting the image to the user while ensuring the image quality and integrity of the image displayed on the receiving end.

[0270] For example, taking the case where the image of frame (X) is divided into three sub-images, and each sub-image is hierarchically encoded into three layers, a comparison between the transmission order of the present application and the transmission order of the existing SVC / SHVC can be shown in Table 2 below. It should be understood that the transmission order shown in Table 2 can also be understood as the encoding order, and in some cases, can also be understood as the reception order.

[0271] Table 2

[0272] Time Unit Existing SVC / SHVC transmission order The order in which this application is sent 1 Frame(X)Sub-image(0)Stream layer(0) Frame(X)Sub-image(0)Stream layer(0) 2 Frame(X)Sub-image(1)Stream layer(0) Frame (X) Sub-image (0) Codestream layer (1) 3 Frame (X) Sub-image (2) Codestream layer (0) Frame(X)Sub-image(0)Stream layer(2) 4 Frame (X) Sub-image (0) Codestream layer (1) Frame(X)Sub-image(1)Stream layer(0) 5 Frame (X) Sub-image (1) Codestream layer (1) Frame (X) Sub-image (1) Codestream layer (1) 6 Frame (X) Sub-image (2) Codestream layer (1) Frame (X) Sub-image (1) Codestream layer (2) 7 Frame(X)Sub-image(0)Stream layer(2) Frame (X) Sub-image (2) Codestream layer (0) 8 Frame (X) Sub-image (1) Codestream layer (2) Frame (X) Sub-image (2) Codestream layer (1) 9 Frame(X)Sub-image(2)Stream layer(2) Frame(X)Sub-image(2)Stream layer(2)

[0273] As can be seen from Table 2, under good channel conditions, using the existing SVC / SHVC standard encoding and transmission order, the transmitter can complete the transmission of the three-layer code stream of the sub-image (0) of frame (X) at the end of the 7th time unit (i.e., time interval) (as shown in the bold part in the existing SVC / SHVC transmission order); accordingly, the receiver needs to wait until all other sub-images of the code stream layer (2) are successfully transmitted before it can decode and display the image of frame (X).

[0274] By adopting the sending sequence of the present application, when the channel conditions are good, the sending end can complete the sending of the three-layer code stream of the sub-image (0) of frame (X) at the end of the third time unit (as shown in the bold part in the sending sequence of the present application); accordingly, after receiving the three-layer code stream of the sub-image (0), the receiving end can decode and display the sub-image (0), thereby reducing the end-to-end transmission delay of presenting the image to the user.

[0275] It should be noted that the second and third transmission results may be due to unstable or low channel bandwidth, resulting in the failure to successfully transmit part or all of the code stream of the first sub-image within the first preset time interval.

[0276] In the second transmission result, the transmitting end successfully transmits at least one layer of the multi-layer codestream of the first sub-image within the first preset time interval, i.e., transmission of all codestreams other than the at least one layer of the multi-layer codestream of the first sub-image fails. Accordingly, the receiving end successfully receives at least one layer of the multi-layer codestream of the first sub-image within the first preset time interval, i.e., reception of all codestreams other than the at least one layer of the multi-layer codestream of the first sub-image fails.

[0277] Optionally, the transmitting end may process the target code stream of the first sub-image within the first preset time interval in a variety of ways, where the target code stream includes the code stream that fails to be sent in the multi-layer code stream of the first sub-image, that is, the target code stream includes the code stream in the multi-layer code stream of the first sub-image except for the at least one layer of code stream.

[0278] In one possible implementation, the transmitting end may directly discard the target codestream and transmit the multi-layer codestream of the second sub-image within the second preset time interval. Accordingly, the receiving end may decode only the at least one layer of the successfully received codestream of the first sub-image to obtain the first sub-image without further waiting for the target codestream of the first sub-image, and may continue to receive the multi-layer codestream of the second sub-image from the transmitting end within the second preset time interval.

[0279] In this way, the transmission delay can be reduced while ensuring the image quality and smoothness of the image displayed on the receiving end as much as possible, and the transmission delay can be strictly controlled and managed.

[0280] In another possible implementation, the sending end may send the target code stream of the first sub-image within the remaining time of another preset time interval. The other preset time interval may be a preset time interval for sending other sub-images of the first frame image. The other sub-images are sub-images encoded or sent after the first sub-image, and there is still time remaining in the other preset time interval when the multi-layer code streams of the other sub-images are successfully sent.

[0281] In this way, it is ensured that the multi-layer code stream of the first sub-image can be sent to the receiving end, thereby providing a reference for the receiving end to subsequently decode other sub-images.

[0282] In one possible implementation, if there is still time remaining in the second preset interval when the multiple layers of code streams of the second sub-image are all sent successfully, the sending end can send the target code stream of the first sub-image to the receiving end layer by layer within the remaining time of the second preset time interval.

[0283] It should be noted that if, within the remaining time of the second preset time interval, some code streams in the target code stream of the first sub-image still fail to be sent, the sending end may continue to send the partial code streams within the remaining time of the next preset time interval, and so on, until all the multiple code streams of the first sub-image are successfully sent.

[0284] For example: Figure 6 FIG. 1 shows another transmission pipeline structure provided by an embodiment of the present application. Figure 6 As shown, T1 to T8 represent eight adjacent time intervals, and the lengths of T1 to T8 are the same and are equal to the preset length.

[0285] Sub-image 1: Acquisition is completed in T1, encoding is completed in T2, transmission of at least one layer of the multi-layer code stream of sub-image 1 is completed in T3, decoding is completed in T4, and display is completed in T5.

[0286] Sub-image 2: Acquisition is completed within T2, encoding is completed within T3, and transmission is completed within T4-1. Since there is a remaining time T4-2 in T4, part or all of the remaining code stream of sub-image 1 can be transmitted within T4-2, decoding is completed within T5, and display is completed within T6.

[0287] Sub-image 3: Acquisition is completed in T3, encoding is completed in T4, transmission is completed in T5, decoding is completed in T6, and display is completed in T7.

[0288] It should be noted that the first frame image described in the embodiment of the present application can be Figure 6 The first sub-image in the first frame image can be Figure 6 The second sub-image can be Figure 6 In the sub-image 2, the first preset time interval can be Figure 6 In T3, the second preset time interval can be Figure 6 T4 in.

[0289] In the third transmission scenario, if the transmitter fails to transmit the multi-layer codestream for the first sub-image within the first preset time interval, and then continues to transmit the multi-layer codestream for the second sub-image within the next preset time interval, the area where the first sub-image is located may not be displayed when the receiver displays the image, thereby affecting the user experience. Therefore, in this case, the multi-layer codestream for the first sub-image should be retransmitted as soon as possible, and then the multi-layer codestream for the second sub-image should be transmitted again. This ensures the integrity and smoothness of the image displayed by the receiver while minimizing transmission latency.

[0290] Optionally, the transmitting end and the receiving end may retransmit the multi-layer code stream of the first sub-image in a variety of ways, which is not limited in this embodiment of the present application.

[0291] In one possible implementation, the transmitting end may re-send the code stream of the first sub-image to the receiving end layer by layer within a second preset time interval, where the second preset time interval is adjacent to and after the first preset time interval; accordingly, the receiving end re-receives the code stream of the first sub-image from the transmitting end layer by layer within the second preset time interval.

[0292] Since the retransmission of the multi-layer code stream of the first sub-image occupies the second preset time interval originally used to transmit the multi-layer code stream of the second sub-image, the transmitting end can retransmit the multi-layer code stream of the first sub-image within the second preset time interval, and send the code stream of the second sub-image to the receiving end layer by layer within the third preset time interval, wherein the third preset time interval is adjacent to the second preset time interval and is after the second preset time interval; accordingly, the receiving end receives the code stream of the second sub-image from the transmitting end layer by layer within the third preset time interval.

[0293] Figure 7 FIG. 1 shows another transmission pipeline structure provided by an embodiment of the present application. Figure 7 As shown, T1 to T8 represent eight adjacent time intervals, and the lengths of T1 to T8 are the same and are equal to the preset length.

[0294] Sub-image 1: Acquisition is completed within T1, encoding is completed within T2, initial transmission is performed within T3, if transmission fails within T3, retransmission is performed within T4, decoding is completed within T5, and display is completed within T6;

[0295] Sub-image 2: Acquisition is completed in T2, encoding is completed in T3, transmission is completed in T5, decoding is completed in T6, and display is completed in T7;

[0296] Sub-image 3: Acquisition is completed in T3, encoding is completed in T4, transmission is completed in T6, decoding is completed in T7, and display is completed in T8.

[0297] It should be noted that the first frame image described in the embodiment of the present application can be Figure 7 The first sub-image in the first frame image can be Figure 7 The second sub-image can be Figure 7 In the sub-image 2, the first preset time interval can be Figure 7 In T3, the second preset time interval can be Figure 7 The third preset time interval can be T4. Figure 7 T5 in.

[0298] In another possible implementation, the sending end may also resend the multi-layer code stream of the first sub-image within the remaining time of other preset time intervals. The other preset time intervals may be a preset time interval for sending other sub-images of the first frame image. The other sub-images are sub-images encoded or sent after the first sub-image, and there is still time remaining in the other preset time intervals when the multi-layer code streams of the other sub-images are all sent successfully.

[0299] For example, if there is still time remaining in the second preset interval when the multi-layer code stream of the second sub-image is successfully sent, the sending end can resend the multi-layer code stream of the first sub-image layer by layer to the receiving end within the remaining time of the second preset time interval.

[0300] Optionally, before S220, the method may further include: the transmitting end receiving feedback information from the receiving end, the feedback information including at least one of a first quantity, a second quantity, or a first bandwidth, the first quantity indicating the number of code stream layers of the third sub-image received by the receiving end, the second quantity indicating the number of code stream layers of the third sub-image successfully decoded by the receiving end, and the first bandwidth indicating the bandwidth for transmitting the code stream of the third sub-image, the code stream of the third sub-image being obtained by performing layered encoding on the third sub-image; wherein, the multiple sub-images include the third sub-image, and the receiving end has completed decoding of the code stream of the third sub-image, or the third sub-image belongs to a second frame image, and the second frame image is located before the first frame image in the frame sequence of the video.

[0301] It should be noted that the above only schematically lists the contents included in the feedback information. Optionally, the feedback information may also include other information that can feedback the status of the channel, which is not limited in the embodiments of the present application.

[0302] In a possible implementation manner, the receiving end may send the feedback information to the sending end within a preset time interval for displaying the first sub-image.

[0303] For example: Figure 6 As shown, sub-image 1: the display is completed within T5, and the feedback is completed within T5.

[0304] In another possible implementation, the transmission pipeline structure may further include a preset time interval for sending feedback information. The preset time interval for sending feedback information may be located between the preset time interval for decoding and the preset time interval for display.

[0305] For example: Figure 6 T8 may also be included between T4 and T5, and sub-image 1: decoding is completed in T4, feedback is completed in T8, and display is completed in T5.

[0306] Accordingly, S220 may include: the transmitting end performing layered coding on the first sub-image according to the feedback information to obtain the multi-layer code stream of the first sub-image.

[0307] Specifically, the transmitting end can determine the encoding parameters of the first sub-image based on the feedback information, where the encoding parameters include at least one of the starting Qp, number of layers, encoding mode, and reference frame of the first sub-image, and the encoding mode may include intra-frame encoding or inter-frame encoding; and based on the encoding parameters, perform layered encoding on the first sub-image to obtain the multi-layer code stream of the first sub-image.

[0308] In one possible implementation, the transmitting end may obtain statistical information based on the feedback information, where the statistical information may include at least one of the average coding bit rate of each layer of code stream within a first time length (within a rolling time window), the average channel bandwidth within the first time length, and the instantaneous coding bit rate or instantaneous channel bandwidth of each layer within the first time length; and determine the coding parameters of the first sub-image based on the statistical information.

[0309] For example: when the ratio of the average channel bandwidth to the average coding rate within the first time duration is high, or the ratio of the instantaneous channel bandwidth to the instantaneous coding rate is high, coding parameters for increasing image quality can be used; when the ratio of the average channel bandwidth to the average coding rate within the first time duration is low, or the ratio of the instantaneous channel bandwidth to the instantaneous coding rate is low, or when continuous code stream loss occurs, coding parameters for reducing the code rate can be used.

[0310] Optionally, the coding parameters for increasing image quality may include: increasing the starting Qp, reducing the number of layers, using inter-frame coding and intra-frame coding, etc.; the coding parameters for reducing the bit rate may include: reducing the starting Qp, increasing the number of layers, using only inter-frame coding, etc.

[0311] For another example, when the bandwidth is relatively stable during the first time period, the number of layers may be reduced; when the bandwidth fluctuates or changes greatly during the first time period, the number of layers may be increased.

[0312] It should be noted that the fewer the number of sub-image layers, the more complex the encoding method of each layer can be, and vice versa, the simpler the encoding method of each layer can be. For example, when the number of layers is small, inter-frame coding and intra-frame coding can be used, and when the number of layers is large, only inter-frame coding can be used.

[0313] Optionally, the encoding parameter may further include an encoding tool, wherein the encoding tool may include at least one of a deblocking filter, the number of alternative modes of intra-frame encoding, or sample adaptive offset (SAO).

[0314] It should be noted that when the number of layers is small, the coding tools used in each layer can be more complex, and conversely, the coding tools used in each layer can be simpler. For example, when the number of layers is small, a deblocking filter, a larger number of intra-frame coding alternative modes, and Sample Adaptive Offset (SAO) are used, while when the number of layers is large, only a smaller number of intra-frame coding alternative modes are used.

[0315] In existing video transmission applications, there is no requirement to provide feedback on the transmission results of each frame image / sub-image. Instead, possible image errors are mainly recovered by periodically using intra-frame coded frames. Since intra-frame coded frames use themselves as a reference, the use of intra-frame coded frames will result in a larger data volume and thus a higher bit rate.

[0316] However, in the embodiment of the present application, due to the channel conditions, such as bandwidth, there may be real-time fluctuations and changes, and the channel conditions directly affect the transmission delay of the image and the image quality of the displayed image. In the embodiment of the present application, the receiving end provides timely feedback on the transmission results of each sub-image, so that the transmitting end can dynamically adjust the encoding strategy according to the feedback information in a timely manner, thereby achieving higher video compression efficiency, reducing the requirements for bandwidth and transmission power consumption, and ultimately improving the displayed image quality, while avoiding image transmission errors. At the same time, the embodiment of the present application can reduce the use of intra-frame coding frames by using feedback signals for feedback, thereby achieving the effect of reducing the bit rate.

[0317] It should be noted that in the embodiment of the present application, only the transmission process of the first frame image of the video data is taken as an example to introduce the method 100. The transmission process of other frame images of the video data is similar to the transmission process of the first frame image. The transmission process of the first frame image can be referred to. To avoid repetition, it will not be repeated here.

[0318] Based on the same inventive concept as the above method, Figure 8 A video transmission device 300 according to an embodiment of the present application is provided. The device 300 is suitable for implementing the embodiments described in the present application. The device 300 can be the transmitting end or the receiving end described in the embodiment of the method 200, which is not limited in the present embodiment.

[0319] It is understandable that, in order to implement the above functions, the device 300 includes hardware and / or software modules that perform the corresponding functions. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of this application.

[0320] In this embodiment, the device 300 can be divided into functional modules according to the above-mentioned method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into a single processing module. The above-mentioned integrated modules can be implemented in the form of hardware. It should be noted that the module division in this embodiment is illustrative and is only a logical functional division. In actual implementation, other division methods may be used.

[0321] In the case of dividing each functional module into corresponding functional modules, Figure 8A schematic diagram of a possible composition of the transmitting end and the receiving end involved in the above embodiment is shown. Figure 8 As shown, the apparatus 300 may include a transceiver unit 310 and a processing unit 320. The processing unit 320 may control the transceiver unit 310 to implement the method described in the embodiment of the method 200 and / or other processes used for the technology described herein.

[0322] It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0323] The apparatus 300 provided in this embodiment is used to execute the above method 200, and thus can achieve the same effect as the above implementation method.

[0324] When integrated, the apparatus 300 may include a processing unit, a storage unit, and a communication unit. The processing unit may be used to control and manage the operations of the apparatus 300, for example, supporting the apparatus 300 in executing the steps performed by the aforementioned units. The storage unit may be used to support the apparatus 300 in executing and storing program code and data. The communication unit may be used to support communication between the apparatus 300 and other devices.

[0325] Among them, the processing unit can be a processor or a controller. It can implement or execute the various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a video codec, a digital signal processor (DSP), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), an image signal processor (ISP), a modem processor, an application processor (AP), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, a combination of discrete hardware components and microprocessors, etc. The storage unit can be a memory. The communication unit can specifically be a device that interacts with other electronic devices, such as a radio frequency circuit, a Bluetooth chip, a Wi-Fi chip, etc.

[0326] In a possible implementation, the device 300 involved in the embodiment of the present application may be a Figure 9The schematic block diagram of the video transmission device 400 is shown. The device 400 is suitable for implementing the embodiments described in this application.

[0327] In a possible implementation, the apparatus 400 may be deployed at a transmitting end or a receiving end.

[0328] For example, when the apparatus 400 is deployed at a transmitting end, it may be referred to as a video encoding apparatus 400 ; and when the apparatus 400 is deployed at a receiving end, it may be referred to as a video decoding apparatus 400 .

[0329] The apparatus 400 may include a processor 410, a memory 430, and a bus system 450. The processor and memory are connected via the bus system; the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory. The memory of the video transmission apparatus stores program code, and the processor can call the program code stored in the memory to execute the video transmission method described herein. To avoid repetition, this description will not be repeated here.

[0330] In the embodiment of the present application, the processor 410 may be a central processing unit (CPU), or may be another general-purpose processor, a video codec, a digital signal processor (DSP), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), an image signal processor (ISP), a modem processor, an application processor (AP), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0331] The memory 430 may include a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device may also be used as the memory 430. The memory 430 may include code and data 431 accessed by the processor 410 using the bus 450. The memory 430 may further include an operating system 433 and an application 435, which includes at least one program that allows the processor 410 to execute the video transmission method described in the embodiment of the present application. For example, the application 435 may include applications 1 to N, which further include executing the video transmission method described in the present application.

[0332] In addition to the data bus, the bus system 450 may also include a power bus, a control bus, a status signal bus, etc. However, for the sake of clarity, various buses are labeled as the bus system 450 in the figure.

[0333] Optionally, the apparatus 400 may further include one or more output devices, such as a display 470. In one example, the display 470 may be a touch-sensitive display that combines a display with a touch-sensitive unit operable to sense touch input. The display 470 may be connected to the processor 410 via the bus 450.

[0334] Based on the above method 200, Figure 10 A schematic block diagram of another video transmission system 500 according to an embodiment of the present application is shown. Figure 10 As shown, the video transmission system 500 may include a transmitter 510 and a receiver 520. The transmitter 510 generates encoded video data, and thus, the transmitter 510 may be referred to as a video encoding device. The receiver 520 may decode the encoded video data generated by the transmitter 510, and thus, the receiver 520 may be referred to as a video decoding device.

[0335] Optionally, various implementations of the transmitter 510, the receiver 520, or both may include one or more processors and a memory coupled to the one or more processors. The memory may include, but is not limited to, read-only memory (ROM), random access memory (RAM), electrically erasable programmable read-only memory (EEPROM), flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures accessible by a computer, as described herein.

[0336] Alternatively, the transmitter 510 and the receiver 520 may comprise a variety of devices, including a desktop computer, a mobile computing device, a notebook (e.g., laptop) computer, a tablet computer, a set-top box, a telephone handset such as a so-called "smart" phone, a television, a camera, a display device, a digital media player, a video game console, an in-vehicle computer, a wireless communication device, or the like.

[0337] Although Figure 10 The transmitter 510 and the receiver 520 are shown as separate devices, but device embodiments may also include both the transmitter 510 and the receiver 520 or the functionality of both, i.e., the transmitter 510 or the corresponding functionality and the receiver 520 or the corresponding functionality. In such embodiments, the transmitter 510 or the corresponding functionality and the receiver 520 or the corresponding functionality may be implemented using the same hardware and / or software, or using separate hardware and / or software, or any combination thereof.

[0338] Transmitter 510 and receiver 520 may be communicatively coupled via link 530, and receiver 520 may receive encoded video data from transmitter 510 via link 530. Link 530 may include one or more media or devices capable of moving the encoded video data from transmitter 510 to receiver 520. In one example, link 530 may include one or more communication media that enable transmitter 510 to transmit the encoded video data directly to receiver 520 in real time. In this example, transmitter 510 may modulate the encoded video data according to a communication standard (e.g., a wireless communication protocol) and transmit the modulated video data to receiver 520. The one or more communication media may include wireless and / or wired communication media, such as radio frequency (RF) spectrum or one or more physical transmission lines. The one or more communication media may form part of a packet-based network, such as a local area network, a wide area network, or a global network (e.g., the Internet). The one or more communication media may include routers, switches, base stations, or other devices that facilitate communication from the sender 510 to the receiver 520 .

[0339] The transmitter 510 includes an encoder 511. Optionally, the transmitter 510 may also include an image source 512, an image preprocessor 513, and a communication interface 514. In a specific implementation, the encoder 511, image source 512, image preprocessor 513, and communication interface 514 may be hardware components in the transmitter 510 or software programs in the transmitter 510. They are described as follows:

[0340] Image source 512 may include or may be any type of image capture device, for example, for capturing real-world images, and / or any type of images or comments (for screen content encoding, some text on the screen is also considered part of the image or picture to be encoded), such as a computer graphics processor for generating computer-animated images, or any type of device for acquiring and / or providing real-world images, computer-animated images (e.g., screen content, virtual reality (VR) images), and / or any combination thereof (e.g., augmented reality (AR) images). Image source 512 may be a camera for capturing images or a memory for storing images. Image source 512 may also include any type of (internal or external) interface for storing previously captured or generated images and / or acquiring or receiving images. When image source 512 is a camera, image source 512 may be, for example, a local camera or an integrated camera integrated into the source device; when image source 512 is a memory, image source 512 may be, for example, a local memory or an integrated memory integrated into the source device. When the image source 512 includes an interface, the interface may be, for example, an external interface for receiving images from an external video source, such as an external image capture device, such as a camera, an external storage device, or an external image generation device, such as an external computer graphics processor, a computer, or a server. The interface may be any type of interface, such as a wired or wireless interface, or an optical interface, according to any proprietary or standardized interface protocol.

[0341] It should be noted that, in the embodiment of the present application, the image transmitted from the image source 512 to the image preprocessor 513 may also be referred to as original image data 11.

[0342] The image preprocessor 513 is configured to receive the original image data 11 and perform preprocessing on the original image data 11 to obtain a preprocessed image 12 or preprocessed image data 12. For example, the preprocessing performed by the image preprocessor 513 may include retouch, color format conversion (e.g., from RGB format to YUV format), color grading, or denoising.

[0343] In a possible implementation, the image source 512 and / or the image preprocessor 513 is configured to execute S210 and related processes in the above-mentioned method embodiment of the present application.

[0344] The encoder 511 (or video encoder 511 ) is configured to receive pre-processed picture data 12 and encode the picture data 12 to provide encoded picture data 13 .

[0345] In a possible implementation, the encoder 511 is used to execute S220 and S260 and related processes in the above-mentioned method embodiment of the present application.

[0346] Communication interface 514 may be configured to receive encoded picture data 13 and transmit the encoded picture data 13 to a receiving end 520 or any other device (e.g., a memory) via link 530 for storage or direct reconstruction. The other device may be any device configured for decoding or storage. Communication interface 514 may, for example, be configured to encapsulate the encoded picture data 13 into a suitable format, such as a data packet, for transmission over link 530.

[0347] The receiving end 520 includes a decoder 521. Optionally, the receiving end 520 may also include a communication interface 522, an image post-processor 523, and a display device 524. They are described as follows:

[0348] Communication interface 522 may be used to receive encoded picture data 13 from transmitter 510 or any other source, such as a storage device, such as an encoded picture data storage device. Communication interface 522 may be used to transmit or receive encoded picture data 13 via link 530 between transmitter 510 and receiver 520, such as a wireless connection, or via any type of network. Communication interface 522 may be used, for example, to decapsulate data packets transmitted by communication interface 514 to obtain encoded picture data 13.

[0349] Both communication interface 522 and communication interface 514 can be configured as unidirectional communication interfaces or bidirectional communication interfaces and can be used to, for example, send and receive messages to establish connections, confirm and exchange any other information related to communication links and / or data transmission, such as encoded picture data transmission.

[0350] In a possible implementation, the communication interface 514 and the communication interface 522 are used to execute S230 and S270 and related processes in the above-mentioned method embodiment of the present application.

[0351] The decoder 521 (or referred to as the video decoder 521 ) is configured to receive the encoded picture data 13 and provide decoded picture data 14 or a decoded picture 14 .

[0352] In a possible implementation, the decoder 521 is used to execute S240 and S280 and related processes in the above-mentioned method embodiment of the present application.

[0353] The image post-processor 523 is configured to perform post-processing on the decoded image data 14 (also referred to as reconstructed image data) to obtain post-processed image data 15. For example, the post-processing performed by the image post-processor 523 may include color format conversion (e.g., from YUV format to RGB format), color grading, retouch or resampling, or any other processing. The image post-processor 523 may also be configured to transmit the post-processed image data 33 to a display device 524.

[0354] A display device 524 is provided for receiving the post-processed image data 15 to display the image to, for example, a user or viewer. The display device 524 may be or include any type of display for presenting the reconstructed image, such as an integrated or external display or monitor. For example, the display may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro-LED display, liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display.

[0355] In a possible implementation, the display device 524 is used to execute S250 and S290 and related processes in the above-mentioned method embodiment of the present application.

[0356] Although, Figure 10 The transmitter 510 and the receiver 520 are shown as separate devices, but device embodiments may also include both the transmitter 510 and the receiver 520 or the functionality of both, i.e., the transmitter 510 or the corresponding functionality and the receiver 520 or the corresponding functionality. In such embodiments, the transmitter 510 or the corresponding functionality and the receiver 520 or the corresponding functionality may be implemented using the same hardware and / or software, or using separate hardware and / or software, or any combination thereof.

[0357] It is obvious to those skilled in the art based on the description that the functionality or Figure 10The presence and (accurate) division of functionality of the transmitter 510 and / or receiver 520 shown may vary depending on the actual device and application. The transmitter 510 and the receiver 520 may include any of the following devices, including any type of handheld or stationary device, such as a notebook or laptop computer, a mobile phone, a smart phone, a tablet or tablet computer, a video camera, a desktop computer, a set-top box, a television, a camera, a car device, a display device, a digital media player, a video game console, a video streaming device (such as a content service server or a content distribution server), and may not use or use any type of operating system.

[0358] The encoder 511 and the decoder 521 can be implemented as any of a variety of suitable circuits, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, or any combination thereof. If the technology is partially implemented in software, the device may store the software instructions in a suitable non-transitory computer-readable storage medium and use one or more processors to execute the instructions in hardware to perform the technology of the present application. Any of the foregoing (including hardware, software, a combination of hardware and software, etc.) can be considered one or more processors.

[0359] An embodiment of the present application further provides a computer storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the video transmission method in the above-mentioned embodiment.

[0360] An embodiment of the present application further provides a computer program product. When the computer program product is run on a computer, the computer is caused to execute the above-mentioned related steps to implement the video transmission method in the above-mentioned embodiment.

[0361] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory to enable the chip to execute the video transmission method in the above-mentioned method embodiments.

[0362] Figure 11 FIG. 6 is a schematic diagram showing the structure of a chip 600. The chip 600 includes one or more processors 610 and an interface circuit 620. Optionally, the chip 600 may further include a bus 630.

[0363] The processor 610 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in the processor 610 or by software instructions. The above-mentioned processor 610 can be a general-purpose processor, GPU, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. It can implement or execute the various methods and steps disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor.

[0364] The interface circuit 620 can be used to send or receive data, instructions or information. The processor 610 can use the data, instructions or other information received by the interface circuit 620 to process, and can send the processing completion information through the interface circuit 620.

[0365] Optionally, the chip also includes a memory, which may include a read-only memory and a random access memory, and provides operating instructions and data to the processor. Part of the memory may also include a non-volatile random access memory (NVRAM).

[0366] Optionally, the memory stores an executable software module or a data structure, and the processor can perform corresponding operations by calling an operation instruction stored in the memory (the operation instruction may be stored in an operating system).

[0367] Optionally, the chip can be used in an access device or access control device involved in the embodiments of the present application. Optionally, the interface circuit 620 can be used to output the execution result of the processor 610. For the video transmission method provided in one or more embodiments of the present application, reference can be made to the aforementioned embodiments and will not be repeated here.

[0368] It should be noted that the corresponding functions of the processor 610 and the interface circuit 620 can be implemented through hardware design, software design, or a combination of hardware and software, and there is no limitation here.

[0369] The video transmission device, video transmission system, computer storage medium, computer program product or chip provided in the embodiments of the present application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here.

[0370] Those skilled in the art will appreciate that the functions described in conjunction with the various illustrative logic blocks, modules, and algorithm steps disclosed herein can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions described in the various illustrative logic blocks, modules, and steps can be stored or transmitted as one or more instructions or codes on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media can include computer-readable storage media, which corresponds to tangible media, such as data storage media, or communication media including any media that facilitates the transfer of computer programs from one place to another (e.g., according to a communication protocol). In this manner, computer-readable media can generally correspond to (1) non-transitory tangible computer-readable storage media, or (2) communication media, such as signals or carrier waves. Data storage media can be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the techniques described in this application. A computer program product can include computer-readable media.

[0371] By way of example, and not limitation, such computer-readable storage media may include (RAM), (ROM), (EEPROM), (CD-ROM) or other optical disk storage, magnetic disk storage or other magnetic storage, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of medium. However, it should be understood that the computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but are actually directed to non-transitory tangible storage media. As used herein, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), and Blu-ray disc. Disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0372] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Thus, the term "processor," as used herein, may refer to any of the aforementioned structures or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described by the various illustrative logical blocks, modules, and steps described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into a combined codec. Furthermore, the techniques may be fully implemented in one or more circuits or logic elements.

[0373] The techniques of this application can be implemented in a variety of devices or apparatuses, including wireless handsets, integrated circuits (ICs), or a set of ICs (e.g., a chipset). Various components, modules, or units are described herein to emphasize functional aspects of devices for performing the disclosed techniques, but they do not necessarily require implementation by different hardware units. In fact, as described above, the various units may be combined in a codec hardware unit in conjunction with appropriate software and / or firmware, or provided by interoperating hardware units (including one or more processors as described above).

[0374] In the above embodiments, the description of each embodiment has different emphases. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0375] The above description is merely an exemplary embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A video transmission method, characterized in that: include: Acquire a first frame image of video data, where the first frame image includes a plurality of sub-images, and the plurality of sub-images include a first sub-image and a second sub-image; performing layered coding on the first sub-image to obtain a multi-layer code stream of the first sub-image; Sending the multi-layer code stream of the first sub-image to a receiving end within a first preset time interval; Within the first preset time interval, performing layered coding on the second sub-image to obtain a multi-layer code stream of the second sub-image; The multi-layer code stream of the second sub-image is sent to the receiving end within a second preset time interval, where the second preset time interval is adjacent to the first preset time interval and is after the first preset time interval.

2. The method according to claim 1, characterized in that The sending the multi-layer code stream of the first sub-image to the receiving end includes: The code stream of the first sub-image is sent to the receiving end layer by layer.

3. The method according to claim 2, characterized in that The sending the code stream of the first sub-image to the receiving end layer by layer includes: The code stream of the first sub-image is sent to the receiving end layer by layer according to the coding order of the multi-layer code stream of the first sub-image.

4. The method according to claim 2 or 3, characterized in that When the multi-layer code streams of the first sub-image fail to be sent within the first preset time interval, the method further includes: The code stream of the first sub-image is sent layer by layer to the receiving end within the second preset time interval, and the second preset time interval is adjacent to the first preset time interval and is after the first preset time interval.

5. The method according to claim 2 or 3, characterized in that At least one layer of the multi-layer code stream of the first sub-image is successfully sent within the first preset time interval, and the method further includes: The code streams other than the at least one layer of code stream in the multi-layer code stream of the first sub-image are discarded.

6. The method according to claim 2 or 3, characterized in that The sending the multi-layer code stream of the second sub-image to the receiving end includes: The code stream of the second sub-image is sent to the receiving end layer by layer.

7. The method according to claim 6, characterized in that If the multi-layer code streams of the first sub-image all fail to be sent within the first preset time interval, and the multi-layer code streams of the second sub-image all succeed in being sent, and there is still time remaining in the second preset time interval, the method further includes: During the remaining time of the second preset time interval, the code stream of the first sub-image is sent to the receiving end layer by layer.

8. The method according to claim 6, characterized in that At least one layer of the multi-layer code stream of the first sub-image is successfully sent within the first preset time interval, and when the multi-layer code stream of the second sub-image is successfully sent, there is still time remaining in the second preset interval, the method further includes: In the remaining time of the second preset time interval, code streams other than the at least one layer of code stream in the multi-layer code stream of the first sub-image are sent to the receiving end layer by layer.

9. The method according to any one of claims 1 to 8, characterized in that Before performing layered encoding on the first sub-image to obtain a multi-layer code stream of the first sub-image, the method further includes: receiving feedback information from the receiving end, the feedback information including at least one of a first quantity, a second quantity, or a first bandwidth, the first quantity indicating the number of codestream layers of the third sub-image received by the receiving end, the second quantity indicating the number of codestream layers of the third sub-image successfully decoded by the receiving end, and the first bandwidth indicating a bandwidth for transmitting a codestream of the third sub-image, the codestream of the third sub-image being obtained by performing layered coding on the third sub-image; The multiple sub-images include the third sub-image, and the receiving end has completed decoding of the code stream of the third sub-image, or the third sub-image belongs to a second frame image, and the second frame image is located before the first frame image in the frame sequence of the video; The performing layered encoding on the first sub-image to obtain a multi-layer code stream of the first sub-image includes: The first sub-image is layered-encoded according to the feedback information to obtain the multi-layer code stream of the first sub-image.

10. The method according to claim 9, characterized in that The step of performing layered coding on the first sub-image according to the feedback information to obtain the multi-layer code stream of the first sub-image includes: Determining, according to the feedback information, encoding parameters of the first sub-image, the encoding parameters including at least one of a starting Qp, a number of layers, an encoding mode, or a reference frame of the first sub-image; The first sub-image is layered-encoded according to the encoding parameters to obtain the multi-layer code stream of the first sub-image.

11. The method according to any one of claims 1 to 10, characterized in that The sub-image is a slice, tile, block, macroblock or maximum coding block of a frame image.

12. A video transmission method, characterized in that: include: receiving, within a first preset time interval, a multi-layer code stream of a first sub-image of a first frame image from a transmitting end; Decoding the multi-layer code stream of the first sub-image to obtain the first sub-image; displaying the first sub-image; receiving, within a second preset time interval, a multi-layer code stream of a second sub-image of the first frame from the transmitting end, the second preset time interval being adjacent to the first preset time interval and after the first preset time interval, the multi-layer code stream of the second sub-image being obtained by performing layered coding on the second sub-image within the first preset time interval; decoding the multi-layer code stream of the second sub-image to obtain the second sub-image; The second sub-image is displayed.

13. The method according to claim 12, characterized in that The decoding of the multi-layer code stream of the first sub-image to obtain the first sub-image includes: The code stream of the first sub-image is decoded layer by layer according to the coding order of the multi-layer code stream of the first sub-image to obtain the first sub-image.

14. The method according to claim 12 or 13, characterized in that The receiving of a multi-layer code stream of a first sub-image of a first frame image from a transmitting end includes: The code stream of the first sub-image is received layer by layer from the transmitting end.

15. The method according to claim 14, characterized in that When the multi-layer code stream of the first sub-image is not successfully received within the first preset time interval, the method further includes: In the second preset time interval, the code stream of the first sub-image is received layer by layer from the transmitting end.

16. The method according to claim 14, characterized in that The receiving, from the transmitting end, a multi-layer code stream of a second sub-image of the first frame image, comprises: The code stream of the second sub-image is received layer by layer from the transmitting end.

17. The method according to claim 16, characterized in that If the multi-layer code stream of the first sub-image is not successfully received within the first preset time interval, and the multi-layer code stream of the second sub-image is successfully received, and there is still time remaining in the second preset time interval, the method further includes: During the remaining time of the second preset time interval, the code stream of the first sub-image is received layer by layer from the transmitting end.

18. The method according to claim 16, characterized in that When at least one layer of the multi-layer code stream of the first sub-image is successfully received within the first preset time interval, and the multi-layer code stream of the second sub-image is successfully received, and there is still time remaining in the second preset time interval, the method further includes: During the remaining time of the second preset time interval, a target code stream is received layer by layer from the transmitting end, where the target code stream includes code streams other than the at least one layer of code stream in the multi-layer code stream of the first sub-image.

19. The method according to any one of claims 12 to 18, characterized in that The method further comprises: Sending feedback information to the transmitting end, where the feedback information includes at least one of a first quantity, a second quantity, or a first bandwidth, where the first quantity indicates the number of codestream layers of the received first sub-image, the second quantity indicates the number of codestream layers of the successfully decoded first sub-image, and the first bandwidth indicates a bandwidth for transmitting the codestream of the first sub-image.

20. The method according to any one of claims 12 to 19, characterized in that The sub-image is a slice, tile, block, macroblock or maximum coding block of a frame image.

21. A video transmission device, characterized in that: The apparatus includes: a processor and a transceiver coupled to the processor; The processor is configured to obtain a first frame image of video data, the first frame image including a plurality of sub-images, the plurality of sub-images including a first sub-image and a second sub-image; perform layered encoding on the first sub-image to obtain a multi-layer code stream of the first sub-image; The transceiver is configured to send the multi-layer code stream of the first sub-image to a receiving end within a first preset time interval; The processor is further configured to perform layered coding on the second sub-image within the first preset time interval to obtain a multi-layer code stream of the second sub-image; The transceiver is further configured to send the multi-layer code stream of the second sub-image to the receiving end within a second preset time interval, where the second preset time interval is adjacent to the first preset time interval and is after the first preset time interval.

22. The device according to claim 21, characterized in that The transceiver is specifically used for: In the first preset time interval, the code stream of the first sub-image is sent to the receiving end layer by layer.

23. The device according to claim 22, characterized in that The transceiver is specifically used for: Within the first preset time interval, the code stream of the first sub-image is sent to the receiving end layer by layer according to the coding order of the multi-layer code stream of the first sub-image.

24. The device according to claim 22 or 23, characterized in that When the multi-layer code streams of the first sub-image fail to be sent within the first preset time interval, the transceiver is further configured to: In the second preset time interval, the code stream of the first sub-image is sent to the receiving end layer by layer.

25. The device according to claim 22 or 23, characterized in that At least one layer of the multi-layer code stream of the first sub-image is successfully sent within the first preset time interval, and the processor is further configured to: The code streams other than the at least one layer of code stream in the multi-layer code stream of the first sub-image are discarded.

26. The device according to claim 22 or 23, characterized in that The transceiver is specifically used for: In the second preset time interval, the code stream of the second sub-image is sent to the receiving end layer by layer.

27. The device according to claim 26, characterized in that If the multi-layer code streams of the first sub-image all fail to be sent within the first preset time interval, and the multi-layer code streams of the second sub-image all succeed in being sent, and there is still time remaining in the second preset time interval, the transceiver is further configured to: During the remaining time of the second preset time interval, the multi-layer code stream of the first sub-image is sent to the receiving end layer by layer.

28. The device according to claim 26, characterized in that At least one layer of the multi-layer code stream of the first sub-image is successfully sent within the first preset time interval, and when the multi-layer code stream of the second sub-image is successfully sent, and there is still time remaining in the second preset time interval, the transceiver is further configured to: In the remaining time of the second preset time interval, code streams other than the at least one layer of code stream in the multi-layer code stream of the first sub-image are sent to the receiving end layer by layer.

29. The device according to any one of claims 21 to 28, characterized in that The transceiver is further configured to: Before performing layered coding on the first sub-image to obtain a multi-layer codestream of the first sub-image, feedback information is received from the receiving end, the feedback information including at least one of a first quantity, a second quantity, or a first bandwidth, the first quantity indicating the number of codestream layers of the third sub-image received by the receiving end, the second quantity indicating the number of codestream layers of the third sub-image successfully decoded by the receiving end, and the first bandwidth indicating a bandwidth for transmitting the codestream of the third sub-image, the codestream of the third sub-image being obtained by performing layered coding on the third sub-image; The multiple sub-images include the third sub-image, and the receiving end has completed decoding of the code stream of the third sub-image, or the third sub-image belongs to a second frame image, and the second frame image is located before the first frame image in the frame sequence of the video; The processor is further configured to: The first sub-image is layered-encoded according to the feedback information to obtain the multi-layer code stream of the first sub-image.

30. The device according to claim 29, characterized in that The processor is specifically configured to: Determining, according to the feedback information, encoding parameters of the first sub-image, the encoding parameters including at least one of a starting Qp, a number of layers, an encoding mode, or a reference frame of the first sub-image; The first sub-image is layered-encoded according to the encoding parameters to obtain the multi-layer code stream of the first sub-image.

31. The device according to any one of claims 21 to 30, characterized in that The sub-image is a slice, tile, block, macroblock or maximum coding block of a frame image.

32. A video transmission device, characterized in that: The device includes: a processor and a transceiver; The transceiver is used to receive a multi-layer code stream of a first sub-image of a first frame image from a transmitting end within a first preset time interval; The processor is configured to decode the multi-layer code stream of the first sub-image to obtain the first sub-image; The processor is further configured to control the first sub-image to be displayed on a display; The transceiver is further configured to receive, from the transmitting end, a multi-layer code stream of a second sub-image of the first frame of image within a second preset time interval, the second preset time interval being adjacent to the first preset time interval and after the first preset time interval, the multi-layer code stream of the second sub-image being obtained by performing layered coding on the second sub-image within the first preset time interval; The processor is further configured to decode the multi-layer code stream of the second sub-image to obtain the second sub-image; The processor is further configured to control the second sub-image to be displayed on the display.

33. The device according to claim 32, characterized in that The processor is specifically configured to: The code stream of the first sub-image is decoded layer by layer according to the coding order of the multi-layer code stream of the first sub-image to obtain the first sub-image.

34. The device according to claim 32 or 33, characterized in that The transceiver is specifically used for: In the first preset time interval, the code stream of the first sub-image is received layer by layer from the sending end.

35. The device according to claim 34, characterized in that When the multi-layer code stream of the first sub-image is not successfully received within the first preset time interval, the transceiver is further configured to: In the second preset time interval, the code stream of the first sub-image is received layer by layer from the transmitting end.

36. The device according to claim 34, characterized in that The transceiver is specifically used for: In the second preset time interval, the code stream of the second sub-image is received layer by layer from the transmitting end.

37. The device according to claim 36, characterized in that The multi-layer code stream of the first sub-image is not successfully received within the first preset time interval, and the multi-layer code stream of the second sub-image is successfully received, and there is still time remaining in the second preset time interval, the transceiver is further configured to: During the remaining time of the second preset time interval, the code stream of the first sub-image is received layer by layer from the transmitting end.

38. The device according to claim 36, characterized in that When at least one layer of the multi-layer code stream of the first sub-image is successfully received within the first preset time interval, and the multi-layer code stream of the second sub-image is successfully received, and there is still time remaining in the second preset time interval, the transceiver is further configured to: During the remaining time of the second preset time interval, a target code stream is received layer by layer from the transmitting end, where the target code stream includes code streams other than the at least one layer of code stream in the multi-layer code stream of the first sub-image.

39. The device according to any one of claims 32 to 38, characterized in that The transceiver is further configured to: Sending feedback information to the transmitting end, where the feedback information includes at least one of a first quantity, a second quantity, or a first bandwidth, where the first quantity indicates the number of codestream layers of the received first sub-image, the second quantity indicates the number of codestream layers of the successfully decoded first sub-image, and the first bandwidth indicates a bandwidth for transmitting the codestream of the first sub-image.

40. The device according to any one of claims 32 to 39, characterized in that The sub-image is a slice, tile, block, macroblock or maximum coding block of a frame image.

41. A computer-readable storage medium for storing a computer program, characterized in that: The computer program comprises instructions for implementing the method of any one of claims 1 to 11 or any one of claims 12 to 20 above.

42. A computer program product comprising instructions, characterized in that: When the instructions are executed on a computer or a processor, the computer or the processor is caused to implement the method according to any one of claims 1 to 11 or any one of claims 12 to 20.

Images