Distributed video coding using reliability data

Abstract

The apparatus is configured to generate prediction data for the picture, wherein the prediction data for the picture comprises a prediction of a block of the picture based at least in part on one or more previously reconstructed pictures of the video data; generating encoded video data based on prediction data of the picture, where the encoded video data includes a transform block, the transform block including transform coefficients; scaling a bit of the transform coefficient of the transform block based on a reliability value of a bit position; generating error-corrected encoded video data using the error correction data to perform an error correction operation on scaled bits of the transform coefficients of the transform block; and reconstructing the picture based on the error corrected encoded video data.

Description

[0001] Related applications

[0002] This application claims priority to U.S. Patent Application No. 18 / 640,692, filed April 19, 2024, and U.S. Provisional Patent Application No. 63 / 497,979, filed April 24, 2023, the entire contents of each of which are incorporated herein by reference. U.S. Patent Application No. 18 / 640,692, filed April 19, 2024, claims the benefit of U.S. Provisional Patent Application No. 63 / 497,979, filed April 24, 2023. Technical Field

[0003] This disclosure relates to video encoding and decoding. Background Technology

[0004] The popularity of Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) technologies is growing rapidly and is expected to be widely used in applications beyond gaming, such as healthcare, education, social networking, and retail. VR, AR, and MR can be collectively referred to as Extended Reality (XR). Due to this increasing popularity, there is a growing demand for XR devices with high 3D graphics quality, higher video resolution, and low latency response, such as XR goggles. Summary of the Invention

[0005] This disclosure describes techniques for processing video data in a transmitting and receiving device. The transmitting device may be an XR device or other types of devices. The receiving device may be a user equipment (UE) device, such as a smartphone or tablet. The transmitting device may perform limited video coding processing on the video data to generate encoded video data. The transmitting device may apply channel coding to the encoded video data to generate error correction data. The transmitting device may transmit the error correction data and at least some of the encoded video data to the receiving device. The receiving device may estimate video data based on one or more previously reconstructed images. The receiving device may then encode the estimated video data. The receiving device may use one or more coding tools to encode the estimated video data that the transmitting device did not use when performing limited video coding processing on the video data. The receiving device may use the error correction data and the predicted video data to regenerate portions of the encoded video data that the transmitting device did not transmit. This processing avoids the need to transmit portions of the encoded video data.

[0006] In one example, the disclosure describes a method of decoding video data, including obtaining error correction data at a receiving device and from a sending device, wherein the error correction data provides error correction information and is generated based on encoded video data of one or more blocks of a picture of the video data; generating, at the receiving device, prediction data for the picture using one or more coding tools that are not used to generate the encoded video data of the one or more blocks, wherein the prediction data for the picture comprises a prediction of a block of the picture based at least in part on a block of one or more previously reconstructed pictures of the video data; generating, at the receiving device, encoded video data based on the prediction data for the picture; generating, at the receiving device, error corrected encoded video data using the error correction data to perform an error correction operation on the encoded video data; and performing, at the receiving device, a reconstruction operation to reconstruct the block of the picture based on the error corrected encoded video data, wherein the reconstruction operation is controlled by values of one or more parameters.

[0007] In another example, the disclosure describes a method of encoding video data, including obtaining, at a sending device, video data from a video source; generating, at the sending device, encoded video data for a first picture of the video data and encoded video data for a second picture of the video data based on a parameter set; performing, at the sending device, channel coding on the encoded video data for the first picture and the encoded video data for the second picture to generate error correction data for the first picture and error correction data for the second picture; and sending, at the sending device, the encoded video data for the first picture, the error correction data for the first picture, and the error correction data for the second picture.

[0008] In another example, the disclosure describes a method of encoding video data, including obtaining, at a sending device, video data from a video source; generating, at the sending device, transform blocks based on the video data; determining, at the sending device, which of the transform blocks are anchor transform blocks; computing, at the sending device, a correlation matrix for a set of transform blocks; generating, at the sending device, bit-reduced non-anchor transform matrices; and sending, at the sending device, the anchor transform blocks, the non-anchor transform blocks, and the correlation matrix to a receiving device.

[0009] In another example, the disclosure describes an apparatus, including a memory configured to store video data; a communication interface; and one or more processors implemented in circuitry and coupled to the memory, the one or more processors configured to perform the method of any of the claims.

[0010] In another example, the disclosure describes an apparatus for processing video data, comprising a memory configured to store video data; and a communication interface configured to obtain error correction data from a transmitting apparatus, wherein the error correction data provides error correction information about a picture of the video data; one or more processors implemented in circuitry and coupled to the memory, the one or more processors configured to generate prediction data for the picture, wherein the prediction data for the picture comprises a prediction for a block of the picture based at least in part on one or more previously reconstructed pictures of the video data; generate encoded video data based on the prediction data for the picture, wherein the encoded video data includes a transform block comprising transform coefficients; scale bits of the transform coefficients of the transform block based on reliability values of the bits; generate error corrected encoded video data using the error correction data to perform an error correction operation on the scaled bits of the transform coefficients of the transform block; and reconstruct the picture based on the error corrected encoded video data.

[0011] In another example, the disclosure describes an apparatus for processing video data, comprising a memory configured to store video data; and one or more processors implemented in circuitry and coupled to the memory, the one or more processors configured to obtain video data; obtain prediction quality feedback, wherein the prediction quality feedback is based on reliability of an estimated picture generated by a receiving apparatus; adapt one or more of video coding parameters or channel coding parameters based on the prediction quality feedback; perform video coding processing to generate encoded video data based on one or more pictures of the obtained video data, wherein the video coding processing is controlled by the video coding parameters; perform channel coding processing on the encoded video data to generate channel coded data, wherein the channel coding processing is controlled by the channel coding parameters; and a communication interface configured to transmit the channel coded data to the receiving apparatus.

[0012] In another example, the disclosure describes a method of processing video data, comprising: obtaining error correction data at a receiving device and from a sending device, wherein the error correction data provides error correction information about a picture of the video data; generating, at the receiving device, prediction data for the picture, wherein the prediction data for the picture comprises a prediction for a block of the picture based at least in part on one or more previously reconstructed pictures of the video data; generating, at the receiving device, encoded video data based on the prediction data for the picture, wherein the encoded video data includes a transform block comprising transform coefficients; scaling, at the receiving device, bits of the transform coefficients of the transform block based on reliability values of the bits; generating, at the receiving device, error corrected encoded video data using the error correction data to perform an error correction operation on the scaled bits of the transform coefficients of the transform block; and reconstructing, at the receiving device, a picture based on the error corrected encoded video data.

[0013] In another example, the disclosure describes a method of processing video data, comprising: obtaining video data; obtaining prediction quality feedback, wherein the prediction quality feedback is based on reliability of an estimated picture generated by a receiving device; adapting one or more of video coding parameters or channel coding parameters based on the prediction quality feedback; performing video coding processing to generate encoded video data based on one or more pictures of the obtained video data, wherein the video coding processing is controlled by the video coding parameters; performing channel coding processing on the encoded video data to generate channel coded data, wherein the channel coding processing is controlled by the channel coding parameters; and sending the channel coded data to the receiving device.

[0014] In another example, the disclosure describes an apparatus comprising: a memory configured to store video data; and one or more processors implemented in circuitry and coupled to the memory, the one or more processors configured to: obtain a first set of multiview pictures of the video data, wherein the first set of multiview pictures includes a first picture and a second picture, the first picture from a first view point and the second picture from a second view point; send first encoded video data to a receiving apparatus, wherein the first encoded video data is based on the first set of multiview pictures; receive a multiview coding hint from the receiving apparatus; obtain a second set of multiview pictures of the video data, wherein the second set of multiview pictures includes a third picture and a fourth picture, the third picture from the first view point and the fourth picture from the second view point; perform a multiview coding process on the second set of multiview pictures based on the multiview coding hint received from the receiving apparatus to generate second encoded video data, wherein the multiview coding process reduces inter-view redundancy between the third picture and the fourth picture; and send the second encoded video data to the receiving apparatus.

[0015] In another example, the disclosure describes an apparatus comprising: a memory configured to store video data; and one or more processors implemented in circuitry and coupled to the memory, the one or more processors configured to: obtain first encoded video data from a sending apparatus, wherein the first encoded video data is based on a first set of multiview pictures of the video data, the first set of multiview pictures including a first picture and a second picture, the first picture from a first view point and the second picture from a second view point; determine a multiview coding hint based on the first encoded video data; send the multiview coding hint to the sending apparatus; obtain second encoded video data from the sending apparatus, wherein the second encoded video data is based on a second set of multiview pictures including a third picture and a fourth picture, the second encoded video data encoded using a multiview coding process that reduces inter-view redundancy between the third picture and the fourth picture based on the multiview coding hint.

[0016] In another example, this disclosure describes a method of processing video data that includes obtaining a first set of multiview pictures of video data, wherein the first set of multiview pictures includes a first picture and a second picture, the first picture being from a first view point and the second picture being from a second view point; sending first encoded video data to a receiving device, wherein the first encoded video data is based on the first set of multiview pictures; receiving a multiview coding hint from the receiving device; obtaining a second set of multiview pictures of the video data, wherein the second set of multiview pictures includes a third picture and a fourth picture, the third picture being from the first view point and the fourth picture being from the second view point; performing a multiview coding process on the second set of multiview pictures based on the multiview coding hint received from the receiving device to generate second encoded video data, wherein the multiview coding process reduces inter-view redundancy between the third picture and the fourth picture; and sending the second encoded video data to the receiving device.

[0017] In another example, this disclosure describes a method of processing video data that includes obtaining a first set of multiview pictures of video data, wherein the first set of multiview pictures includes a first picture and a second picture, the first picture being from a first view point and the second picture being from a second view point; sending first encoded video data to a receiving device, wherein the first encoded video data is based on the first set of multiview pictures; receiving a multiview coding hint from the receiving device; obtaining a second set of multiview pictures of the video data, wherein the second set of multiview pictures includes a third picture and a fourth picture, the third picture being from the first view point and the fourth picture being from the second view point; performing a multiview coding process on the second set of multiview pictures based on the multiview coding hint received from the receiving device to generate second encoded video data, wherein the multiview coding process reduces inter-view redundancy between the third picture and the fourth picture; and sending the second encoded video data to the receiving device.

[0018] In another example, this disclosure describes an apparatus comprising: means for obtaining a first set of multiview pictures of video data, wherein the first set of multiview pictures includes a first picture from a first view and a second picture from a second view; means for sending first encoded video data to a receiving apparatus, wherein the first encoded video data is based on the first set of multiview pictures; means for receiving a multiview coding hint from the receiving apparatus; means for obtaining a second set of multiview pictures of the video data, wherein the second set of multiview pictures includes a third picture from the first view and a fourth picture from the second view; means for performing a multiview coding process on the second set of multiview pictures based on the multiview coding hint received from the receiving apparatus to generate second encoded video data, wherein the multiview coding process reduces inter-view redundancy between the third picture and the fourth picture; and means for sending the second encoded video data to the receiving apparatus.

[0019] In another example, this disclosure describes an apparatus comprising: means for obtaining first encoded video data from a sending apparatus, wherein the first encoded video data is based on a first set of multiview pictures of the video data, the first set of multiview pictures including a first picture from a first view and a second picture from a second view; means for determining a multiview coding hint based on the first encoded video data; means for sending the multiview coding hint to the sending apparatus; means for obtaining second encoded video data from the sending apparatus, wherein the second encoded video data is based on a second set of multiview pictures including a third picture and a fourth picture, the second encoded video data encoded using a multiview coding process that reduces inter-view redundancy between the third picture and the fourth picture based on the multiview coding hint.

[0020] In another example, this disclosure describes an apparatus comprising: a memory configured to store video data; and one or more processors implemented in circuitry and coupled to the memory, the one or more processors configured to: encode a first set of pictures of the video data to generate first encoded video data; send the first encoded video data to a receiving apparatus; receive, from the receiving apparatus, a skip mode indication indicating a skip mode determined based on the first set of pictures, the skip mode being a mode in which encoded video data is not transmitted; encode a second set of pictures of the video data to generate second encoded video data; apply the skip mode to the second encoded video data to generate skipped video data; and send the skipped video data to the receiving apparatus.

[0021] In another example, the disclosure describes an apparatus comprising: a memory configured to store video data; and one or more processors implemented in circuitry and coupled to the memory, the one or more processors configured to: receive, from a sending apparatus, first encoded video data; perform a decoding process to reconstruct a first set of pictures based on the first encoded video data; determine, based on the first set of pictures, an extraction mode that indicates a mode in which encoded video data is not transmitted; send, to the sending apparatus, an extraction mode indication that indicates the determined extraction mode; receive, from the sending apparatus, extracted video data, wherein the extracted video data comprises second encoded video data to which the extraction mode has been applied, wherein the second encoded video data is generated based on a second set of pictures of the video data; and perform the decoding process to reconstruct the second set of pictures based on the second encoded video data.

[0022] In another example, the disclosure describes a method comprising: encoding a first set of pictures of video data to generate first encoded video data; sending, to a receiving apparatus, the first encoded video data; receiving, from the receiving apparatus, an extraction mode indication that indicates an extraction mode determined based on the first set of pictures, the extraction mode being a mode in which encoded video data is not transmitted; encoding a second set of pictures of the video data to generate second encoded video data; applying the extraction mode to the second encoded video data to generate extracted video data; and sending, to the receiving apparatus, the extracted video data.

[0023] In another example, the disclosure describes a method comprising: receiving, from a sending apparatus, first encoded video data; applying a decoding process to reconstruct a first set of pictures based on the first encoded video data; determining, based on the first set of pictures, an extraction mode that indicates a mode in which encoded video data is not transmitted; sending, to the sending apparatus, an extraction mode indication that indicates the determined extraction mode; receiving, from the sending apparatus, extracted video data, wherein the extracted video data comprises second encoded video data to which the extraction mode has been applied, wherein the second encoded video data is generated based on a second set of pictures of the video data; and performing the decoding process to reconstruct the second set of pictures based on the second encoded video data.

[0024] In another example, the disclosure describes an apparatus comprising: means for encoding a first set of pictures of video data to generate first encoded video data; means for sending the first encoded video data to a receiving apparatus; means for receiving, from the receiving apparatus, a decimation mode indication indicating a decimation mode determined based on the first set of pictures, the decimation mode being a mode in which encoded video data is not transmitted; means for encoding a second set of pictures of the video data to generate second encoded video data; means for applying the decimation mode to the second encoded video data to generate decimated video data; and means for sending the decimated video data to the receiving apparatus.

[0025] In another example, the disclosure describes an apparatus comprising: means for receiving first encoded video data from a sending apparatus; means for performing a decoding process to reconstruct a first set of pictures based on the first encoded video data; means for determining, based on the first set of pictures, a decimation mode indicating a mode in which encoded video data is not transmitted; means for sending, to the sending apparatus, a decimation mode indication indicating the determined decimation mode; means for receiving, from the sending apparatus, decimated video data, wherein the decimated video data comprises second encoded video data to which the decimation mode has been applied, wherein the second encoded video data is generated based on a second set of pictures of the video data; and means for performing a decoding process to reconstruct the second set of pictures based on the second encoded video data.

[0026] In another example, the disclosure describes an apparatus comprising: a memory configured to store video data; and one or more processors implemented in circuitry and coupled to the memory, the one or more processors configured to: encode a first picture of the video data to generate first encoded video data; send the first encoded video data to a receiving apparatus; receive, from the receiving apparatus, encoding selection data for a second picture of the video data, wherein: the encoding selection data for the second picture indicates an encoding selection used to encode an estimate of the second picture, and the second picture is subsequent to the first picture in decoding order; encode the second picture based on the encoding selection data for the second picture to generate second encoded video data; and send the second encoded video data to the receiving apparatus.

[0027] In another example, this disclosure describes an apparatus comprising: a memory configured to store video data; and one or more processors implemented in circuitry and coupled to the memory, the one or more processors configured to: receive first encoded video data from a sending apparatus; reconstruct a first picture of the video data based on the first encoded video data; estimate a second picture of the video data based on the first picture, the second picture being a picture that occurs after the first picture in decoding order; generate coding selection data for the second picture, wherein the coding selection data for the second picture indicates a coding selection for encoding the second picture; send the coding selection data for the second picture to the sending apparatus; receive second encoded video data from the sending apparatus; and reconstruct the second picture based on the second encoded video data.

[0028] In another example, this disclosure describes a method of processing video data comprising: encoding a first picture of video data to generate first encoded video data; sending the first encoded video data to a receiving apparatus; receiving, from a receiving apparatus, coding selection data for a second picture of video data, wherein: the coding selection data for the second picture indicates a coding selection for encoding an estimate of the second picture, and the second picture follows the first picture in decoding order; encoding the second picture based on the coding selection data for the second picture to generate second encoded video data; and sending the second encoded video data to the receiving apparatus.

[0029] In another example, this disclosure describes a method of processing video data comprising: receiving first encoded video data from a sending apparatus; reconstructing a first picture of the video data based on the first encoded video data; estimating a second picture of the video data based on the first picture, the second picture being a picture that occurs after the first picture in decoding order; generating coding selection data for the second picture, wherein the coding selection data for the second picture indicates a coding selection for encoding the second picture; sending the coding selection data for the second picture to the sending apparatus; receiving second encoded video data from the sending apparatus; and reconstructing the second picture based on the second encoded video data.

[0030] In another example, this disclosure describes an apparatus comprising: means for encoding a first picture of video data to generate first encoded video data; means for sending the first encoded video data to a receiving apparatus; means for receiving, from the receiving apparatus, encoding selection data for a second picture of the video data, wherein: the encoding selection data for the second picture indicates an encoding selection for encoding an estimate of the second picture, and the second picture is subsequent to the first picture in decoding order; means for encoding the second picture based on the encoding selection data for the second picture to generate second encoded video data; and means for sending the second encoded video data to the receiving apparatus.

[0031] In another example, this disclosure describes an apparatus comprising: means for receiving first encoded video data from a sending apparatus; means for reconstructing a first picture of the video data based on the first encoded video data; means for estimating a second picture of the video data based on the first picture, the second picture being a picture that occurs subsequent to the first picture in decoding order; means for generating encoding selection data for the second picture, wherein the encoding selection data for the second picture indicates an encoding selection for encoding the second picture; means for sending the encoding selection data for the second picture to the sending apparatus; means for receiving second encoded video data from the sending apparatus; and means for reconstructing the second picture based on the second encoded video data.

[0032] The details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 is a block diagram illustrating an example system in accordance with the techniques of this disclosure.

[0034] Figure 2 is a block diagram illustrating example components of a sending apparatus and a receiving apparatus in accordance with the techniques of this disclosure.

[0035] Figure 3A is a conceptual diagram illustrating an example channel encoding process in accordance with the techniques of this disclosure.

[0036] Figure 3B is a block diagram illustrating an example channel decoding process in accordance with the techniques of this disclosure.

[0037] Figure 4 is a flow diagram illustrating an example operation of a sending apparatus in accordance with the techniques of this disclosure.

[0038] Figure 5 is a flowchart illustrating example operations of a receiving device according to the techniques of this disclosure.

[0039] Figure 6 is a conceptual diagram illustrating example decimation patterns according to the techniques of this disclosure.

[0040] Figure 7 is a flowchart illustrating example operations of a transmitting device for hybrid decimation of transform blocks according to the techniques of this disclosure.

[0041] Figure 8 is a flowchart illustrating example operations of a receiving device for hybrid decimation of transform blocks according to the techniques of this disclosure.

[0042] Figure 9 is a conceptual diagram illustrating example decimation patterns adaptively selected by a receiving device according to one or more techniques of this disclosure.

[0043] Figure 10 is a block diagram illustrating example components of a transmitting device and a receiving device according to the techniques of this disclosure.

[0044] Figure 11 is a chart illustrating example error probabilities and corresponding log-likelihood ratio (LLR) absolute values according to one or more techniques of this disclosure.

[0045] Figure 12 is a flowchart illustrating example operations of a transmitting device using scaled bits according to the techniques of this disclosure.

[0046] Figure 13 is a flowchart illustrating example operations of a receiving device using scaled bits according to the techniques of this disclosure.

[0047] Figure 14 is a flowchart illustrating example data exchanges between a transmitting device and a receiving device related to multi-view processing according to one or more techniques of this disclosure.

[0048] Figure 15 is a flowchart illustrating example operations of a transmitting device for multi-view processing according to the techniques of this disclosure.

[0049] Figure 16 is a flowchart illustrating example operations of a receiving device for multi-view processing according to the techniques of this disclosure.

[0050] Figure 17 is a block diagram of example components of a transmitting device and a receiving device performing decimation of encoded video data according to the techniques of this disclosure.

[0051] Figure 18is a conceptual diagram illustrating example information exchange including an extraction mode indication in accordance with the techniques of this disclosure.

[0052] Figure 19 is a flow diagram illustrating example operations of a transmitting device that receives an extraction mode indication in accordance with the techniques of this disclosure.

[0053] Figure 20 is a flow diagram illustrating example operations of a receiving device that transmits an extraction mode indication in accordance with the techniques of this disclosure.

[0054] Figure 21 is a block diagram illustrating example components of a transmitting device and a receiving device that transmits encoding selection data to the transmitting device in accordance with the techniques of this disclosure.

[0055] Figure 22 is a communication diagram illustrating example data exchange between a transmitting device and a receiving device including transmission and reception of encoding selection data in accordance with the techniques of this disclosure.

[0056] Figure 23 is a flow diagram illustrating example operations of a transmitting device that receives encoding selection data in accordance with the techniques of this disclosure.

[0057] Figure 24 is a flow diagram illustrating example operations of a receiving device that transmits encoding selection data in accordance with the techniques of this disclosure.

[0058] Figure 25 is a conceptual diagram illustrating example tiers of encoded video data in accordance with the techniques of this disclosure.

[0059] Figure 26 is a block diagram illustrating alternative example components of a transmitting device in accordance with one or more techniques of this disclosure.

[0060] Figure 27 is a block diagram illustrating example alternative components of a receiving device in accordance with one or more techniques of this disclosure. DETAILED DESCRIPTION

[0061] While modern video coding processes can significantly reduce the amount of data needed to represent video data, such video coding processes are often resource intensive and can involve many memory operations. As a result, modern video coding processes can require complex processors, fast memory, and consume a considerable amount of energy. However, for some contemporary and future-planned wireless communication systems, such as 5G and 6G wireless communication systems, wireless transmission bandwidth can be less constrained, particularly when communicating over short distances, such as between devices on a human body.

[0062] This disclosure describes techniques that can use error correction to reduce the complexity of video encoding at a transmitting device, the error correction performed as part of channel decoding using error correction data. The transmitting device can perform a limited video encoding process that generates encoded video data. The limited video encoding process generally uses relatively less resource-intensive encoding tools, such as intra prediction. Because the video encoding process uses less complex encoding tools, the generated encoded video data can be larger than video data encoded using more complex and resource-intensive encoding tools. The error correction data is based on the encoded video data. The transmitting device can transmit the error correction data to a receiving device. The transmitting device can not need to transmit all of the encoded video data for one or more pictures to the receiving device.

[0063] The receiving device can estimate a picture of video data based on one or more previously reconstructed pictures. In some examples, to estimate the picture, the receiving device can extrapolate the content of a block from a previously reconstructed picture. The receiving device can then perform a full video encoding process on the estimated picture to generate estimated encoded video data for the picture. When performing the full video encoding process, the receiving device can use more complex encoding tools than the limited video encoding process performed by the transmitting device, such as inter prediction. The receiving device can perform a channel decoding process that generates error corrected encoded video data based on the estimated encoded video data for the picture and the error correction data for the picture. In some cases, the channel decoding process can generate the error corrected encoded video data based on the error correction data for the picture and a combination of the estimated encoded video data for the picture transmitted by the transmitting device and the encoded video data for the picture. The receiving device can reconstruct the picture based on the error corrected encoded video data. In this way, the receiving device is able to reconstruct each picture of the video data even though the transmitting device did not transmit all of the encoded video data for the picture.

[0064] As described further in this disclosure, various techniques can be applied that specify which transform blocks of lightly encoded video data are not signaled or have a reduced bit depth, such as the application of a decimation mode. Further, in some examples of the disclosure, a reliability value can be determined for a bit position, a reliability value can be used to scale bits of transform coefficients of a transform block, and the scaled values can be used for channel encoding and channel decoding.

[0065] As further described in the disclosure, a receiving device can determine a decimation pattern based on the first set of pictures. The decimation pattern is a pattern in which encoded video data is not transmitted. The receiving device can send a decimation pattern indication to a delivering device indicating the determined decimation pattern. The delivering device can receive the decimation pattern indication from the receiving device and apply the indicated decimation pattern to the encoded video data to generate decimated video data. The delivering device can send the decimated video data to the receiving device. In this way, the techniques of the disclosure can further reduce resource consumption at the delivering device while still avoiding transmitting excess data. This can further improve encoding efficiency.

[0066] Figure 1 is a block diagram illustrating an example system 100 in accordance with the techniques of this disclosure. In Figure 1 In the example of FIG. 1, system 100 includes a sending device 102, a receiving device 104, and a base station 106. Sending device 102 can be a device configured to perform an extended reality (XR) device (e.g., an XR headset), a mobile device, a wearable device, a sensor device, an Internet of Things (IoT) device, an intermediate networking device, or another type of device. In some examples, sending device 102 can be included in a robot or a vehicle. Receiving device 104 can be a computing device, such as a mobile device (e.g., a mobile phone or a tablet computer), a personal computer, a vehicle-based computing device, a wireless base station, a wearable computing device, an intermediate networking device, a specialized device, an Internet of Things (IoT) device, or another type of device. In some examples, receiving device 104 is a device that a user of sending device 102 can have in addition to sending device 102.

[0067] Sending device 102 and receiving device 104 can communicate with base station 106. In some examples, sending device 102 and receiving device 104 can communicate with base station 106 using a fifth generation (5G) wireless communication protocol, a sixth generation (6G) wireless communication protocol, a WiFi protocol, a Bluetooth protocol, or another type of wireless communication protocol. Base station 106 can transmit data from network 115 to sending device 102 and receiving device 104 via wireless downlink channels 108A, 108B (collectively, “wireless downlink channels 108”). Base station 106 can receive data from sending device 102 and receiving device 104 via wireless uplink channels 110A, 110B (collectively, “wireless uplink channels 110”) for transmission to other devices connected to network 115. Sending device 102 and receiving device 104 can communicate directly with each other via a wireless sidelink channel 112. In other examples, sending device 102 and receiving device 104 can communicate via other types of channels.

[0068] In Figure 1In the example of FIG. 1, the sending device 102 includes one or more processors 114, memory 116, a communication interface 118, a video source 120, and a display system 122. The receiving device 104 includes one or more processors 130, memory 132, and a communication interface 134. The processors 114 and 130 can include circuitry configured to perform various information processing tasks, including executing computer-readable instructions. The processors 114 and 130 can include microprocessors, digital signal processors, and other types of circuitry. The memory 116 and 132 can be configured to store data, such as computer-readable instructions, video data, and other types of data. The communication interfaces 118 and 134 can be configured to transmit and receive data, e.g., via the wireless downlink channel 108, the wireless uplink channel 110, and the wireless sidelink channel 112. In other examples, the sending device 102 and the receiving device 104 can communicate via other types of channels. The processor 114 can be coupled to the memory 116, e.g., via one or more communication channels. Similarly, the processor 130 can be coupled to the memory 132, e.g., via one or more communication channels.

[0069] In general, the video source 120 represents a source of video data, e.g., raw, uncoded video data. The video source 120 can include one or more video capture devices, such as video cameras, video archives containing previously captured raw video, and / or video feed interfaces to receive video from a video content provider. As another alternative, the video source 120 can generate data based on computer graphics as the source video, or a combination of live video, archived video, and computer-generated video.

[0070] In examples in which the sending device 102 is an XR device that presents MR and AR images to a user, it can be necessary to analyze the video data from the video source 120 so that the display system 122 of the sending device 102 can display virtual elements at the correct locations. Processing the video data in this way can require a large amount of computational resources. In other words, a powerful processor and a large amount of energy can be used when processing the video data. Because the sending device 102 can be designed to be worn on the head of a user, it can be important to minimize the weight and power consumption of the sending device 102 while supporting high-quality, low-latency video.

[0071] Further, in some examples, the sending device 102 is an XR headset, and the sending device 102 can be configured to process pictures of the video data to generate virtual element data. The receiving device 104 can be configured to transmit (and the sending device 102 configured to receive) the virtual element data. The transmission device 102 can include a display system 122 configured to display one or more virtual elements in an XR scene based on the virtual element data.

[0072] Accordingly, it can be desirable to offload processing of video data to a device other than the transmitting device 102 (e.g., the receiving device 104). The receiving device 104 can have more resources, permanently or temporarily, than the transmitting device 102. For example, the receiving device 104 can be equipped with a larger battery and a relatively powerful processor. However, for the receiving device 104 to process the video data, the transmitting device 102 can need to transmit the video data to the receiving device 104 via the wireless sidelink channel 112. Because a very large number of bits can be needed to represent unencoded high-quality video data, it can take a large amount of time and energy for the transmitting device 102 to transmit the unencoded high-quality video data to the receiving device 104. The time needed for the transmission can defeat the goal of providing low-latency video to a user. The energy needed for the transmission can defeat the goal of minimizing power consumption. Encoding the video data using a video coding specification (e.g., H.264 / Advanced Video Coding (AVC), H.265 / High Efficiency Video Coding (HEVC), or H.266 / Versatile Video Coding (VVC)) can significantly reduce the amount of data needed to represent the video data. However, the encoding process itself can introduce its own latency and power consumption needs.

[0073] This disclosure describes techniques that can solve these problems. According to the techniques of this disclosure, the transmitting device 102 and the receiving device 104 can use distributed video coding (DVC) processing. The DVC processing reduces the encoding workload performed by the transmitting device 102 and shifts some of the encoding workload to the receiving device 104. The receiving device 104 can have more resources (e.g., computing power, access to power, etc.) than the transmitting device 102 and can therefore be better equipped to perform the encoding work. In some examples, the DVC processing can be used to load balance computing tasks among devices. For example, the system can determine that it can be generally more efficient for the receiving device 104 to perform a particular video-related computing task than the transmitting device 102.

[0074] In addition to video encoding processing, the sending device 102 can perform channel encoding processing to prepare the encoded video data for transmission to the receiving device 104. The channel encoding processing can generate error correction data for data sequences within the encoded video data. Generally, the receiving device 104 uses the error correction data to correct errors introduced to the encoded video data during transmission. However, according to the techniques of this disclosure, the sending device 102 can transmit error correction data for some encoded video data but not the encoded video data to which the error correction data corresponds. The receiving device 104 can estimate one or more subsequent pictures. The receiving device 104 can perform video encoding processing on the subsequent pictures to generate estimated encoded video data. The receiving device can use the estimated encoded video data and the received error correction data to generate error corrected encoded video data. The receiving device 104 can then decode the error corrected encoded video data to reconstruct the video data that the sending device 102 did not transmit.

[0075] Thus, in some examples, the receiving device 104 can obtain first encoded video data and first error correction data from the sending device 102. The first encoded video data can represent one or more blocks of a first picture of video data. The first error correction data can provide error correction information about the blocks of the first picture. The receiving device 104 can perform an error correction operation on the first encoded video data using the first error correction data to generate first error corrected encoded video data. In addition, the receiving device 104 can perform a first reconstruction operation that reconstructs the blocks of the first picture based on the first encoded video data. The first reconstruction operation can be controlled by values of one or more parameters.

[0076] Further, the receiving device 104 can obtain second error correction data from the sending device 102. The second error correction data can provide error correction information about one or more blocks of a second picture of video data. The receiving device 104 can generate prediction data for the second picture. The prediction data for the second picture can include a prediction of the blocks of the second picture of video data based at least in part on blocks of one or more previously reconstructed pictures (e.g., the first picture). The receiving device 104 can generate the prediction data using one or more encoding tools that were not used to generate the encoded video data for the second picture. The receiving device 104 can generate second encoded video data based on the prediction of the blocks of the second picture. The receiving device 104 can perform an error correction operation on the second encoded video data using the second error correction data to generate second error corrected encoded video data. The receiving device 104 can perform a second reconstruction operation that reconstructs the blocks of the second picture based on the second error corrected encoded video data. The second reconstruction operation is controlled by values of the parameters.

[0077] Further, according to one or more techniques of this disclosure, the receiving device 104 can receive an extraction pattern indication from the receiving device 104. The receiving device 104 can determine an extraction pattern indication determined based on a previously reconstructed picture. The extraction pattern indication can indicate a pattern in which encoded video data is not transmitted. For example, the extraction pattern can indicate a pattern in which transmission of encoded video data for a complete picture is skipped. In some examples, the extraction pattern indicates a pattern in which transmission of encoded video data for a specified region within a picture is skipped. In some examples in which the video data is multi-view video data, the extraction pattern can indicate a pattern in which transmission of encoded video data for a picture from a specified view is skipped.

[0078] The transmitting device 102 can perform a video encoding process on a picture of video data. The video encoding process can compress the picture less than a “heavy” or more complex compression operation, such as compression operations described in the H.264, H.265, and H.266 video encoding standards. In addition to the video encoding process, the transmitting device 102 can also perform a channel encoding process to prepare the encoded video data for transmission to the receiving device 104. The channel encoding process can generate error correction data for a sequence of data within the encoded video data. Generally, the receiving device 104 uses the error correction data to correct errors introduced to the encoded video data during transmission. However, the receiving device 104 can also use the error correction data to recover information that was intentionally not transmitted to the receiving device. Accordingly, the transmitting device 102 can apply an extraction pattern to the second encoded video data to generate extracted video data. The transmitting device 102 can transmit the error correction data (which is generated based on the encoded video data that is not extracted) and the extracted video data to the receiving device 104.

[0079] The receiving device 104 can obtain the first encoded video data and the first error correction data from the transmitting device 102. The first encoded data can represent one or more blocks of a first picture of video data. The first error correction data can provide error correction information about the blocks of the first picture. The receiving device 104 can perform an error correction operation on the first encoded video data using the first error correction data to generate first error corrected encoded video data. Additionally, the receiving device 104 can perform a first reconstruction operation that reconstructs the blocks of the first picture based on the first encoded video data. The first reconstruction operation can be controlled by values of one or more parameters.

[0080] Furthermore, the receiving device 104 can obtain first error correction data and first encoded video data from the transmitting device 102. The receiving device 104 can apply error correction processing to modify the first encoded video data based on the first error correction data to generate first error-corrected encoded video data. The receiving device 104 can also apply decoding processing to reconstruct a first set of images based on the first error-corrected encoded video data. The receiving device 104 can determine a decimation mode based on the first set of images, indicating a mode in which encoded video data is not transmitted. The receiving device 104 can transmit a decimation mode indication to the transmitting device 102 to indicate the determined decimation mode. The receiving device 104 can receive second error correction data and decimated video data from the transmitting device 102. The decimated video data may include second encoded video data with the applied decimation mode. The second encoded video data is generated based on a second set of images of video data. The receiving device 104 can apply error correction processing to modify the second encoded video data based on the second error correction data to generate second error-corrected encoded video data. The receiving device 104 can apply decoding processing to reconstruct the second set of images based on the second error-corrected encoded video data.

[0081] Figure 2 This is a block diagram illustrating example components of a transmitting and receiving apparatus according to the technology of this disclosure. System 200 includes a transmitting apparatus 102 and a receiving apparatus 104. The transmitting apparatus 102 is configured to transmit encoded video data to the receiving apparatus 104. Figure 2 In one example, the transmitting device 102 includes a video encoder 210, a channel encoder 212, and a punching unit 214. The receiving device 104 includes a de-punching unit 220, a channel decoder 222, a video decoder 224, an image estimation unit 226, and a video encoder 228. In other examples, the transmitting device 102 and the receiving device 104 may contain more, fewer, or different units. The processor 114 of the transmitting device 102 ( Figure 1 The receiver 104's processor 130 can implement a video encoder 210, a channel encoder 212, and a punching unit 214. The processor 130 of the receiver 104 can implement a de-punching unit 220, a channel decoder 222, a video decoder 224, an image estimation unit 226, and a video encoder 228. Communication interface 118 ( Figure 1 () can represent the transmitting device 102 in transmitting and receiving data. Communication interface 134 ( Figure 1 () can represent receiving device 104 transmitting and receiving data.

[0082] The video encoder 210 of the transmitting device 102 can transmit data from a video source (e.g., video source 120). Figure 1) receives video data. The video data can include, for example, raw, unencoded video pictures from video source 120. In some examples, a memory (e.g., memory 116 (of FIG. 1) of transmitting device 102) can store the video data. Video encoder 210 can perform video encoding processing on the video data to generate encoded video data. The video encoding processing can be “limited” in that the video encoding processing can be relatively fast and consume fewer resources than more robust video compression processing, such as H.264 / AVC, H.265 / HEVC, or H.266 / VVC. The video encoding processing can not reduce the number of bits representing the video data to the same extent as more robust or complete video encoding processing. Figure 1

[0083] Video encoder 210 can perform the limited video encoding processing in one of a variety of ways. For example, in some examples, video encoder 210 can perform prediction processing, such as intra prediction processing, on each picture of the video data to generate prediction data. Video encoder 210 can generate residual data based on the prediction data. For example, video encoder 210 can subtract samples of the prediction data from corresponding samples of the original picture to determine samples of the residual data. A sample can be a value indicative of a color value (e.g., a Y, Cb, or Cr value in a YCbCr color space or a red, green, or blue value in an RGB color space).

[0084] Video encoder 210 can apply a transform (e.g., a discrete cosine transform (DCT)) to the residual data to generate a transform block including transform coefficients. Additionally, video encoder 210 can quantize the transform coefficients. Video encoder 210 can apply entropy encoding (e.g., context adaptive binary arithmetic coding (CABAC) encoding or exponential Golomb-Rice encoding) to syntax elements representing the quantized transform coefficients. The encoded video data can include the entropy encoded syntax elements. In some examples, video encoder 210 applies the transform and / or quantization directly to the video data without first using intra prediction. In some examples in which video encoder 210 does not apply entropy encoding, the encoded video data includes syntax elements representing quantized transform coefficients, unquantized transform coefficients, or residual data.

[0085] In examples in which video encoder 210 does not use inter-picture prediction, fewer memory read requests can be needed as compared to more robust video compression processing that can need to read data about previously encoded pictures from memory. Such memory read requests can be relatively time and energy intensive.

[0086] ​In some examples where the video data is multi-view video data, the video encoder 210 may perform multi-view video coding to generate prediction data. For example, the video encoder 210 may use inter-view prediction to generate prediction data for blocks (e.g., macroblocks, coding units, etc.) of non-anchored images. In some cases, inter-view prediction may involve determining a disparity vector for a block, which indicates the lateral displacement between the block and a corresponding block in one or more reference views.

[0087] The channel encoder 212 of the transmitting device 102 can apply channel coding processing to the encoded video data. The channel coding processing prepares the encoded video data for transmission over a wireless communication channel (e.g., channel 230). Channel 230 may be a wireless sidelink channel 112 (…). Figure 1 The channel-encoded video data may contain error correction data. The channel encoder 212 can generate error correction data in various ways. For example, the channel encoder 212 can generate error correction data as convolutional codes or turbo codes. The error correction data helps the receiving device 104 determine whether the received encoded video data has changed during transmission via channel 230, and helps the receiving device 104 correct such changes. A more detailed discussion of channel encoding and channel decoding is provided below with reference to Figure 3.

[0088] In addition, Figure 2 In the example, the puncturing unit 214 of the transmitting device 202 can apply bit puncturing processing to the error correction data to generate bit-punctured error correction data. Bit puncturing processing can reduce the number of bits in the error correction data. For example, the puncturing unit 214 can perform an operation to remove bits from the error correction data according to a puncturing pattern.

[0089] The transmitting device 102 can transmit data, such as encoded video data and error correction data (e.g., bit-punctured error correction data), to the receiving device 104 via channel 230. Channel 230 can introduce noise into the transmitted data. In some examples, channel 230 is a multipath channel, and the data transmitted in channel 230 can be time-varying. The receiving device 104 can receive the noise-modified data. The receiving device 104 can store the noise-modified data at least temporarily in a memory such as memory 132. Figure 1 ) in the memory.

[0090] The de-puncturing unit 220 can perform a de-puncturing operation on the received bit-punctured error correction data to reconstruct the error correction data. The de-puncturing operation can replace the puncture symbols with neutral values ​​according to the puncturing mode indication. The de-puncturing operation can generate erase bits that indicate the presence of neutral symbols in the error correction data.

[0091] Channel decoder 222 can apply a channel decoding process to generate error- corrected encoded video data based on error correction data and encoded video data (e.g., encoded video data received from a transmitting device and / or encoded video data generated by receiving device 104). For example, channel decoder 222 can change values of bit-encoded video data according to any of a variety of error correction schemes such as low-density parity check (LDPC) encoding or forward error correction (FEC).

[0092] Video decoder 224 can perform a video decoding process to reconstruct a picture based on the error-corrected encoded video data. For example, video decoder 224 can apply an entropy decoding process to bits of the error-corrected encoded video data to obtain quantized transform coefficients. Video decoder 224 can apply an inverse quantization operation to the quantized transform coefficients, apply an inverse transform to the inverse quantized transform coefficients to generate residual data, generate prediction data to generate prediction data, and use the prediction data and the residual data to reconstruct a picture of video data. Video decoder 224 can generate prediction data in the same manner as video encoder 210.

[0093] Picture estimation unit 226 can generate an estimate of a next picture of video data. For example, picture estimation unit 226 can extrapolate the next picture from two or more previously reconstructed pictures. For example, in this example, picture estimation unit 226 can partition a first previously reconstructed picture into blocks. For each block of the first previously reconstructed picture, picture estimation unit 226 can determine one or more corresponding blocks of blocks in one or more additional previously reconstructed pictures. A corresponding block of a block can be a best available match to the block. Picture estimation unit 226 can generate a prediction for a block based on the one or more corresponding blocks of the block. Picture estimation unit 226 can generate predictions using uni-prediction or bi-prediction. Thus, by generating a prediction for each block of the next picture, picture estimation unit 226 can generate an estimate of the next picture. In some examples, picture estimation unit 226 generates the next picture by applying global motion to a previously reconstructed picture.

[0094] In some examples, picture estimation unit 226 can re-encode a current picture that video decoder 224 has decoded. A next picture of the video data can be a picture that follows the picture that video decoder 224 has just decoded in decoding order. In this example, picture estimation unit 226 can perform intra prediction or inter prediction for a block of the current picture. When performing inter prediction for a block, picture estimation unit 226 can determine one or more motion vectors for the block. For example, picture estimation unit 226 can determine that a particular block of the current picture has a motion vector of magnitude m relative to a reference block in a reference picture that has a picture order count (POC) distance of pi from the current picture. In this example, the current picture and the next picture can have a POC distance of p2. Picture estimation unit 226 can determine a scaling factor s as p2 / pl. Picture estimation unit 226 can then scale the motion vector of the particular block by s (e.g., ). Picture estimation unit 226 can determine a location in the next picture indicated by the scaled motion vector, and set the sample at the determined location as the sample of the particular block of the current picture. Picture estimation unit 226 can repeat this process for each inter predicted block of the current picture.

[0095] In some examples, picture estimation unit 226 can apply one or more filters to the prediction data. For example, picture estimation unit 226 can apply one or more deblocking filters, smoothing filters, adaptive loop filters, or other types of filters to the prediction data.

[0096] Video encoder 228 can perform the same finite video encoding process as video encoder 210 on the video data generated by picture estimation unit 226. For example, video encoder 228 can perform intra prediction to generate prediction data. Video encoder 228 can use the prediction data and a corresponding block of the video data generated by picture estimation unit 226 to generate residual data. Video encoder 228 can apply a transform (e.g., a DCT transform, a DST transform, etc.) to the residual data to generate transform coefficients. Video encoder 228 can apply quantization to the transform coefficients. In addition, video encoder 228 can apply entropy encoding to syntax elements representing the transform coefficients.

[0097] As briefly mentioned above, channel decoder 222 can apply a channel decoding process to the channel encoded video data. Figure 3A and Figure 3B More information is provided regarding the channel encoding process performed by channel encoder 212 and the channel decoding process performed by channel decoder 222.

[0098] In particular, Figure 3Ais a block diagram illustrating an example channel encoding process according to the techniques of this disclosure. For each picture of encoded video data, channel encoder 212 of transmitting device 102 can apply a systematic encoding operation (e.g., a low density parity check (LDPC) encoding operation) to systematic bits of the picture to generate error correction data for the picture. The systematic bits of the picture can include encoded video data of the picture generated by video encoder 210.

[0099] In Figure 3A the example, the error correction data is labeled as “error correction bits.” For picture n, channel encoder 212 can generate error correction data 300A based on systematic bits 302A. Similarly, for picture n+1, channel encoder 212 can generate error correction data 300B based on systematic bits 302B. Channel encoder 212 can classify pictures of the video data as anchor video pictures and non-anchor pictures. Channel encoder 212 can classify the pictures such that anchor pictures occur periodically between pictures. In some examples, channel encoder 212 can classify a picture as an anchor picture if channel decoder 222 (e.g., from receiving device 104) receives an indication that there is an error in the picture. For each of the anchor pictures, transmitting device 102 can transmit the encoded anchor picture and the error correction data for the anchor picture. However, for the non-anchor pictures, transmitting device 102 can transmit only the error correction data for the non-anchor pictures.

[0100] For example, in Figure 3A the example, picture n can be an anchor picture and picture n+1 is a non-anchor picture. Accordingly, transmitting device 102 can transmit systematic bits 302A for picture n, error correction data 300A for picture n, and error correction data 300B for picture n+1, but not systematic bits 302B for picture n+1.

[0101] Figure 3B is a block diagram illustrating an example channel decoding process according to the techniques of this disclosure. As mentioned above, channel decoder 222 can perform a channel decoding process on encoded video data to reconstruct the encoded video data. When processing an anchor picture (e.g., picture n), channel decoder 222 can obtain systematic bits of the anchor picture (labeled SI Figure 3B in n ) and error correction data of the anchor picture (labeled y Figure 3B in nThe system bits of the anchor image can represent the encoded video data of the anchor image. The channel decoder 222 can use the error correction data of the anchor image to detect and / or correct errors in the system bits of the anchor image. The video decoder 224 can reconstruct the anchor image using the obtained error-corrected encoded video data for the anchor image. The receiving device 104 can store the reconstructed image (including the reconstructed anchor image and the reconstructed non-anchor image) in the decoded image buffer 350.

[0102] When processing non-anchored images, the channel decoder 222 can obtain the system bits (denoted as SI) representing the encoded video data of the non-anchored image. n+1 The video encoder 228 of the receiving device 104 can generate encoded video data of the unanchored image based on the video data generated by the image estimation unit 226. The channel decoder 222 can obtain error correction data for the unanchored image from the de-punching unit 220 (in... Figure 3B The middle is represented as y n+1 Then, channel decoder 222 can perform the same channel decoding processing that channel decoder 222 applies when processing anchored images. Therefore, channel decoder 222 can use error correction data from non-anchored images to detect and / or correct "errors" in the system bits of non-anchored images. However, the "errors" in the system bits of non-anchored images are not due to noise in channel 230 (as is the case with errors in the system bits of anchored images). Instead, the "errors" in the system bits of non-anchored images may be due to differences between the predicted version of the non-anchored image and the original version of the non-anchored image. Therefore, channel decoder 222 can use the error correction data transmitted for non-anchored images as a mechanism for "correcting" prediction errors.

[0103] The video encoder 210 of the transmitting device 102, the video decoder 224 of the receiving device 104, and the video encoder 228 of the receiving device 104 can perform video encoding and video decoding processes based on the values ​​of one or more sets of parameters. In other words, the values ​​of the parameters can control various aspects of the video encoding process performed by the video encoder 210, video encoder 228, and video decoder 224. In some examples, the parameters may include one or more of the following:

[0104] Parameters indicating the color space (e.g., red-green-blue, Y-Cb-Cr, etc.).

[0105] Pixel extraction parameters

[0106] Parameters indicating DCT size

[0107] transmitted DCT coefficients

[0108] a parameter indicating a number of bits for each DCT coefficient

[0109] a quantization parameter (e.g., a parameter indicating a quantization scheme, such as linear, Max-Lloyd, etc.)

[0110] Each of video encoder 210, video decoder 224, and video encoder 228 can need to use the same parameter values. Accordingly, transmitting device 102 can transmit values of parameters to receiving device 104, in accordance with one or more techniques of this disclosure. Receiving device 104 can receive the transmitted parameter values. Video decoder 224 and video encoder 228 can use the values of the parameters in video decoding and video encoding processing.

[0111] In some examples, the values of the transmitted parameters are static or semi-static. For example, in examples in which the values of the transmitted parameters are static, transmitting device 102 can transmit the values of the parameters to receiving device 104 once, and receiving device 104 can operate with the values of the parameters for an indefinite period of time. In examples in which the values of the transmitted parameters are semi-static, transmitting device 102 can occasionally update the parameter values and retransmit the updated parameter values to receiving device 104.

[0112] Transmitting device 102 can transmit the values of the parameters in one of various ways. For example, in some examples, transmitting device 102 can use an uplink control information (UCI) / medium access control-control element (MAC-CE) message, a radio resource control (RRC) message, or another type of message to transmit the values of the parameters to receiving device 104.

[0113] In examples in which transmitting device 102 transmits the values of the parameters to receiving device 104, video encoder 210 can segment each of the color components (e.g., R, G, and B components; Y, Cb, Cr components) of a picture of video data into blocks of uniform size (M x M). Examples of such blocks can include macroblocks (MBs) largest coding units (LCUs). Video encoder 210 of transmitting device 102 can compute a transform (e.g., 2D-DCT) on each block, resulting in M x M DCT coefficients. Video encoder 210 can quantize the DCT coefficients, resulting in quantized DCT coefficients. Video encoder 210 can encode the quantized DCT coefficients into a bitstream. 2Video encoder 210 can assign an order to the transform coefficients of the block. For example, video encoder 210 can order the transform coefficients of the block according to a zigzag scan order that starts with the most important transform coefficient (e.g., lowest frequency) and ends with the least important transform coefficient (e.g., highest frequency). Video encoder 210 can select the first Nc transform coefficients, where Nc is a parameter value that indicates a number of transform coefficients to transmit. Video encoder 210 can discard the unselected transform coefficients.

[0114] Additionally, video encoder 210 can quantize the selected transform coefficients. For example, in a case where the selected transform coefficients have indices i ranging from 0 to N c - 1, the parameters can include bit width parameters (e.g., Bi, i = 0, 1,..., N c - 1) corresponding to different index values. For each of the selected transform coefficients di, video encoder 210 can quantize the selected transform coefficients di using the following equation.

[0115]

[0116] In the above equation, ci is a quantized version of the transform coefficient di, a is a scaling constant, Bi is a bit width parameter for index i, and round is a function that rounds to the nearest integer. Thus, in an example where B0 is 8, B1 is 4, and B2 is 4, the quantized transform coefficients can be, for example, c0= 00100011, c1= 0110, c2= 1001, etc.

[0117] Video encoder 228 of receiving device 104 can generate prediction data for a picture, generate residual data based on the prediction data, and apply one or more transforms to the residual data to generate a transform block including transform coefficients. Video encoder 228 can need to use the same bit width parameters as video encoder 210 so that channel decoder 222 can correctly associate particular system bits with corresponding error correction data received from depuncturing unit 220.

[0118] In some examples of the disclosure, receiving device 104 can determine values for one or more of the parameters without transmitting device 102 transmitting the values for the parameters to receiving device 104. Examples in which receiving device 104 determines values for one or more of the parameters without transmitting device 104 transmitting the values to receiving device 104 can achieve a better compression-distortion tradeoff with lower control signaling overhead. For example, receiving device 104 can determine the number of DCT coefficients (N c c ​). For example, in this example, the receiving device 104 can determine the minimum value of N c that achieves the desired peak signal-to-noise ratio (P-SNR). In other words, the receiving device 104 can determine the minimum value of N c that achieves the desired P-SNR as:

[0119]

[0120] In the above equation, c i is a transform coefficient (e.g., a DCT coefficient) having index i, SNR d is the desired P-SNR, M 2 -1 is the maximum number of transform coefficients. In some examples, the receiving device 104 can evaluate the number of transform coefficients (N c ) per picture (or other segment) based on a robust prediction of the picture. Periodic resetting of the parameter values can be applied to avoid error propagation.

[0121] In another example, the receiving device 104 can determine the number of quantization bits based on a prediction of the picture, rather than receiving the number of quantization bits from the sending device 102. For example, in this example, the receiving device 104 can calculate the probability distribution of quantized coefficients and unquantized coefficients:

[0122]

[0123] In the above equation, p is the probability distribution of quantized coefficients, p i is the probability distribution of unquantized coefficients, c i is a quantized version of a transform coefficient d i .

[0124] The receiving device 104 can determine the number of quantization bits B i based on the entropy ratio of quantized coefficients to unquantized coefficients, as follows:

[0125]

[0126] The receiving device 104 can thus evaluate B i once per picture (or segment) based on a prediction of the picture generated by the picture estimation unit 226.

[0127] Figure 4 is a flowchart illustrating example operations of the sending device 102 in accordance with the techniques of this disclosure. In Figure 4In the example of FIG. 1, video encoder 210 of transmitting device 102 can obtain video data (400). For example, video encoder 210 can obtain the video data from video source 120. In addition, video encoder 210 can perform video encoding on the video data to generate encoded video data (402). For example, video encoder 210 can apply intra prediction to generate prediction data, generate residual data based on the prediction data and the original video data, and apply a transform (e.g., DCT) to blocks of the residual data to generate transform blocks. Video encoder 210 can quantize transform coefficients of the transform blocks. Furthermore, in some examples, video encoder 210 can apply entropy encoding to syntax elements representing the quantized transform coefficients. In some examples, video encoder 210 can implement a reconstruction loop that can apply entropy decoding, inverse quantization, and one or more inverse transforms to reconstruct the residual data. Video encoder 210 can apply the prediction data and the reconstructed residual data to reconstruct the video data. In some examples, video encoder 210 applies one or more filters to the reconstructed video data, such as a deblocking filter, an adaptive loop filter, a sample adaptive offset filter, etc. Video encoder 210 can use the reconstructed video data as reference data for intra prediction.

[0128] Video encoder 210 can perform the video encoding process based on values of one or more parameters. For example, video encoder 210 quantizes transform coefficients according to a particular quantization parameter, uses a particular color space, etc.

[0129] Channel encoder 212 of transmitting device 102 can perform channel encoding on the encoded video data to generate error correction data (404). Transmitting device 102 can transmit the encoded video data and the error correction data to receiving device 104 (406), e.g., via channel 230. In some examples, transmitting device 102 can selectively transmit portions of the encoded video data and not transmit other portions of the encoded video data. For example, transmitting device 102 transmits encoded video data for some pictures and does not transmit encoded video data for other pictures. In another example, transmitting device 102 can transmit encoded video data for some transform blocks of a picture and does not transmit encoded video data for other transform blocks of the picture. In some examples, transmitting device 102 can transmit a certain number of most significant bits of transform coefficients and does not transmit less significant bits of the transform coefficients.

[0130] In some examples, the sending device 102 can also send values of one or more parameters to the receiving device 104. The values of the parameters can control how the receiving device 104 reconstructs the video data. For example, a parameter can include a transform size parameter that indicates a size of a transform block in the encoded video data generated by the video encoder 210. In this example, the receiving device 104 can need to interpret the received encoded video data according to the same transform block size in order to properly reconstruct the video data. In other examples, a parameter can include a parameter that indicates a number of transform coefficients, a bit width parameter, and the like.

[0131] Accordingly, in Figure 4 In examples in which the sending device 102 can obtain video data from a video source, the sending device 102 can generate encoded video data for a first picture of the video data and encoded video data for a second picture of the video data based on a set of parameters. The sending device 102 can perform channel encoding on the encoded video data for the first picture and the encoded video data for the second picture to generate error correction data for the first picture and error correction data for the second picture. The sending device 102 can send the encoded video data for the first picture, the error correction data for the first picture, and the error correction data for the second picture. In some examples, the sending device 102 can send values of the parameters to the receiving device 104.

[0132] Figure 5 is a flowchart illustrating example operations of the receiving device 104 in accordance with the techniques of this disclosure. In Figure 5 In examples in which the receiving device 104 can obtain first encoded video data and first error correction data from the sending device 102 (500). The first encoded data represents one or more blocks of a first picture of video data. The first error correction data can provide error correction information about the blocks of the first picture.

[0133] The receiving device 104 can generate first error corrected encoded video data using the first error correction data to perform an error correction operation on the first encoded video data (502). For example, the channel decoder 222 of the receiving device 104 can use the first error correction data to perform low density parity check (LDPC) encoding on the first encoded video data. In other examples, the channel decoder 222 can use the error correction data in other error correction algorithms, such as forward error correction (FEC) or turbo encoding. Performing the error correction operation on the first encoded video data can remove errors introduced by noise in the channel 230.

[0134] The video decoder 224 of the receiving device 104 can perform a first reconstruction operation that reconstructs a block of the first picture based on the first error- corrected encoded video data (504). The first reconstruction operation is controlled by values of one or more parameters. For example, the video decoder 224 can perform an inverse transform on a transform block of the first error-corrected encoded video data to obtain residual data. In addition, in this example, the video decoder 224 can generate prediction data, e.g., using intra prediction. In this example, the video decoder 224 can use the prediction data and the residual data to reconstruct the block of the first picture.

[0135] In some examples, the video encoder 210 and the video encoder 228 can use a quantization parameter to quantize transform coefficients generated based on prediction data of a picture to generate encoded video data. When performing a reconstruction operation, the video decoder 224 can use the quantization parameter to inverse quantize transform coefficients of the error-corrected encoded video data. In some examples, the sending device 102 and / or the receiving device 104 can calculate the quantization parameter based on an entropy ratio of quantized transform coefficients and unquantized transform coefficients, e.g., as described above.

[0136] In some examples, the parameters include a transform size parameter. As part of generating encoded video data, the video encoder 210 and the video encoder 228 can apply a forward transform having a transform size indicated by the transform size parameter to sample domain data (e.g., predicted sample data or residual data) of a picture. As part of performing a reconstruction operation, the video decoder 224 can apply an inverse transform having a transform size indicated by the transform size parameter to transform coefficients of the error-corrected encoded video data.

[0137] In some examples, the parameters include a parameter that indicates a number of transform coefficients. As part of generating encoded video data, the video encoder 210 and the video encoder 228 can include a set of transform coefficients in the encoded video data, the set of transform coefficients including the indicated number of transform coefficients. When performing a reconstruction operation, the video decoder 224 can parse a set of transform coefficients from the error-corrected encoded video data, the set of transform coefficients including the indicated number of transform coefficients. In addition, in some examples, the receiving device 104 can receive the encoded video data and the error correction data from the sending device via a communication channel, the receiving device 104 can apply an optimization process that determines the number of transform coefficients based on a signal-to-noise ratio of data transmitted on the communication channel, e.g., as discussed above.

[0138] In some examples, the parameters include a bit-width parameter for the plurality of index values. Performing the reconstruction operation for each respective index value of the plurality of index values can include parsing a first set of bits from the error-corrected encoded video data. The first set of bits can indicate transform coefficients having the respective index value, and a number of bits in the first set of bits is equal to a bit-width indicated by the bit-width parameter for the respective index value. As part of generating the encoded video data, video encoder 210 and video encoder 228 can include a second set of bits in the encoded video data. The second set of bits can indicate transform coefficients having the respective index value, and a number of bits in the second set of bits is equal to a bit-width indicated by the bit-width parameter for the respective index value. Video decoder 224 can parse a third set of bits from the error-corrected encoded video data. The third set of bits can indicate transform coefficients having the respective index value, and a number of bits in the third set of bits is equal to a bit-width indicated by the bit-width parameter for the respective index value.

[0139] Other parameters can include one or more of a color space, a transform size, a quantization parameter, a number of transform coefficients in the first encoded video data, or a number of bits per transform coefficient in the first encoded video data.

[0140] Receiving device 104 can obtain second error correction data from transmitting device 102 (506). The second error correction data provides error correction information for one or more blocks of a second picture of the video data.

[0141] Additionally, picture estimation unit 226 of receiving device 104 can estimate the second picture based on one or more previously reconstructed pictures (e.g., the first picture) (508). The estimated second picture includes a prediction of blocks of the second picture of the video data based at least in part on blocks of the first picture. For example, picture estimation unit 226 can use perform inter-prediction, a combination of intra- and inter-prediction, or other video coding tools (e.g., as described elsewhere in this disclosure) to generate the prediction data.

[0142] The video encoder 228 of the receiving device 104 can generate second encoded video data based on the estimated second picture (510). For example, the video encoder 228 of the receiving device 104 can generate residual data based on the prediction data. For instance, the video encoder 228 can perform intra prediction to generate second prediction data based on the estimated second picture. The video encoder 228 can then generate residual data by subtracting the second prediction data from the prediction data generated by the picture estimation unit 226. The video encoder 228 can then generate transform blocks by applying one or more forward transforms to the residual data. The video encoder 228 can perform the same processing as the video encoder 210 of the sending device 102 and, thus, can need to use the same parameters as the video encoder 210.

[0143] The channel decoder 222 of the receiving device 104 can perform channel decoding processing to generate second error corrected encoded video data based on the second error correction data and the second encoded video data (512). The channel decoder 222 of the receiving device 104 can perform the same processing as the channel decoder 222 performed when generating the first error corrected encoded video data to generate the second error corrected encoded video data.

[0144] The video decoder 224 of the receiving device 104 can perform second video decoding processing to reconstruct blocks of the second picture based on the second error corrected encoded video data (514). The second reconstruction operation is controlled by values of parameters. The video decoder 224 can perform the second reconstruction operation in the same manner as the first reconstruction operation. In this way, the receiving device 104 can reconstruct video data for a picture (or block) without needing to receive all encoded video data for each of the picture (or block).

[0145] As described above, the puncturing unit 214 of the sending device 102 can perform a bit puncturing operation on the error correction data generated by the channel encoder 212. Bit puncturing involves selectively discarding some of the error correction data before the sending device 102 transmits the error correction data. The discarded bits are typically the least important for performing error correction. The depuncturing unit 220 of the receiving device 104 can perform an inverse bit puncturing operation (i.e., a bit depuncturing operation) that reverses the bit puncturing operation performed by the puncturing unit 214. The puncturing unit 214 can perform the bit puncturing operation according to a set of one or more puncturing parameters. In different examples, the puncturing parameters can be predefined, static, or semi-static.

[0146] According to one or more techniques of this disclosure, the transmitting device 102 can perform decimation processing, which can reduce memory bandwidth and enhance compression. For example, the video encoder 210 of the transmitting device 102 can segment images of video data into a grid of blocks (e.g., MB, LCU, etc.) and can generate a transform block for each of said blocks. The transmitting device 102 may need to store each transform block intended to be sent to the receiving device 104 into a memory (e.g., memory 116). The transmitting device 102 can then retrieve the stored transform blocks from the memory for channel coding and ultimately for transmission. These writes to and reads from the memory can increase time and energy requirements. These time and energy requirements may be directly related to the amount of data to be written and read. Therefore, reducing the amount of data to be written to and read from the memory may be advantageous.

[0147] Performing decimation processing can reduce the amount of data written to and read from memory in the transform block. Decimation processing can also reduce the amount of data transmitted from transmitting device 102 to receiving device 104. In some examples, video encoder 210 may generate a transform block for the current block while encoding the current block of the current image. Additionally, channel encoder 212 of transmitting device 102 can determine whether the current block is decimation-targeted based on the decimation mode. If the current block is decimation-targeted (i.e., the transform block is a "non-anchored transform block"), channel encoder 212 can reduce the number of bits in the non-anchored transform block before storing it in memory. If the current block is not decimation-targeted (i.e., the transform block is an "anchored block"), channel encoder 212 does not reduce the number of bits in the anchored block. Channel encoder 212 can perform decimation processing after generating error correction data. Therefore, the error correction data generated by channel encoder 212 for non-anchored transform blocks (and potentially transmitted to receiving device 104) can be based on the entire bit set of the transform block rather than the number of bits reduced.

[0148] Figure 6 This is a conceptual diagram illustrating an example extraction pattern 600 according to the technology disclosed herein. Figure 6 The example illustrates a grid of DCT blocks. A DCT block is a block of transform coefficients generated by applying a DCT transform to video data (such as residual or sample data). In other examples, a DCT block can be a transform block generated using other types of transforms. Figure 6 In extraction mode 600, the "X" marker indicates the DCT block targeted for extraction (i.e., the non-anchored transform block). Therefore, in Figure 6 In the example, decimation mode 600 decimates the DCT block twice in both the horizontal and vertical directions. In some examples that may be referred to herein as "full" decimation, the video encoder 210 can reduce the number of bits in the non-anchored transform block to zero.

[0149] Thus, in some examples, the decimation pattern defines a pattern of anchor transform blocks and non-anchor transform blocks in the picture. The receiving device 104 can receive the systematic bits for the anchor transform blocks and not receive the systematic bits for the non-anchor transform blocks. The systematic bits for the anchor transform blocks can represent the transform coefficients in the anchor transform blocks. The systematic bits for the non-anchor transform blocks can represent reduced bit-depth versions of the original transform coefficients in the non-anchor transform blocks. The error correction data can include error correction data for the anchor transform blocks and error correction data for the non-anchor transform blocks. The error correction data for the non-anchor transform blocks is based on the original transform coefficients in the non-anchor transform blocks. As part of generating error corrected encoded video data, the channel decoder 222 can use the error correction data for the anchor transform blocks to perform error correction on the systematic bits for the anchor transform blocks. The channel decoder 222 can use the error correction data for the non-anchor transform blocks to perform error correction on the portion of the encoded video data corresponding to the non-anchor transform blocks. In some examples, the receiving device 104 can determine the decimation pattern and send the decimation pattern to the sending device 102.

[0150] In some examples, the sending device 102 stores the encoded bits (encoded video data and error correction data) in a circular buffer. The sending device 102 uses two parameters to select which bits in the circular buffer to transmit. The first parameter is a starting position and the second parameter indicates a number of consecutive bits to transmit. The starting position can only have various values in order to support selective transmission and no transmission of systematic bits. The starting position can be selected to skip transmission of certain systematic bits (i.e., bits of encoded video data) without skipping transmission of error correction data. Thus, decimation of non-anchor transform blocks can be accomplished simply by manipulating the first parameter and the second parameter such that the sending device 102 does not transmit bits for non-anchor transform blocks.

[0151] In some examples, the channel encoder 212 applies a hybrid decimation method that does not reduce the number of bits in any target "non-anchor" transform block to zero, but rather reduces the number of bits in the transform coefficients in the non-anchor transform blocks. For example, in the example of Figure 6 In some examples, the channel encoder 212 applies a hybrid decimation method that does not reduce the number of bits in any target "non-anchor" transform block to zero, but rather reduces the number of bits in the transform coefficients in the non-anchor transform blocks. For example, in the example of

[0152] The channel decoder 222 of the receiving device 104 can receive the remaining reduced bits of the non-anchor transform block and the error correction data for the non-anchor transform block. As part of the channel decoding process, the channel decoder 222 can use the error correction data for the non-anchor transform block to perform an error correction process that recovers the removed bits of the non-anchor transform block. This error correction process can be the same error correction process that the channel decoder 222 uses to correct errors introduced by noise in the channel 230. In total, the channel encoder 212 generates error correction data because the error correction data will be needed to correct for inevitable noise in the channel 230, but these same error correction data are used to recover bits as if the noise in the channel 230 just happened to coincidentally corrupt the least significant bits of particular transform coefficients in particular transform blocks in particular pictures. As a result, the number of bits transmitted in the channel 230 can be effectively reduced.

[0153] In some examples, the channel encoder 212 can generate a correlation matrix based on the set of transform blocks of a picture before performing any decimation process on any non-anchor transform block of the set of transform blocks. The correlation matrix includes values that indicate a level of correlation between transform coefficients at corresponding positions within a transform block. For example, the correlation matrix can include a correlation value for the DC transform coefficients (i.e., the top-left transform coefficients) of the set of transform blocks. If the differences between the DC transform coefficients are relatively small, the correlation value for the DC transform coefficients can be relatively high. Conversely, if the differences between the DC transform coefficients are relatively large, the correlation value for the DC transform coefficients can be relatively small. Each correlation value can be a value between 0 and 1.

[0154] In some examples, the channel encoder 212 can use the following equation to calculate the correlation value for the DC transform coefficients:

[0155]

[0156] In Equation (5) above, / denotes the interval between transform blocks containing DC coefficients, and N denotes the number of transform blocks for which the calculation is performed. The function y is a transform coefficient. The line above y denotes the conjugate number. The interval can be 1 if each consecutive transform block is used, 2 if alternate transform blocks are used, and so on. The channel encoder 212 can calculate the correlation values for the corresponding AC transform coefficients (i.e., non-DC transform coefficients) in the same manner. In this disclosure, the corresponding transform coefficients occupy the same positions within a transform block. Thus, by calculating the autocorrelation values for each transform coefficient in a transform block, the channel encoder 212 can generate a correlation matrix for the transform block. The channel encoder 212 can repeat the process of generating a correlation matrix for each transform block because the channel encoder 212 will use transform coefficients from different transform blocks when calculating the correlation values.

[0157] The transmitting device 102 can transmit the correlation matrix to the receiving device 104 along with the encoded video data and error correction data. The channel decoder 222 of the receiving device 104 can use the correlation matrix as part of the process to recover the original bit width of the non-anchor transform block. For example, continuing the example of the DC transform coefficients, after applying the error correction process, the channel decoder 222 can obtain the values of the non-anchor DC transform coefficients (i.e., the DC transform coefficients in the non-anchor transform block).

[0158] The error correction process can use the correlation values to estimate the non-anchor transform coefficients. For example, if we consider all even transform blocks as anchor transform blocks and all non-even transform blocks as non-anchor transform blocks, the channel decoder 222 can estimate the values of the non-anchor transform coefficients in transform block n by:

[0159]

[0160] In equation (6) above, R yy [1] indicates the correlation value in the correlation matrix of the transform block with index 1 (i.e., the non-even non-anchor transform block), y[n+1] indicates the corresponding transform coefficient in the anchor transform block with index n+1, R yy [2] indicates the correlation value in the correlation matrix of the transform block with index 3, y[n+3] indicates the corresponding transform coefficient in the anchor transform block is index n+3, and so on. The number of transform blocks used in equation (6) can be configurable. In this way, the estimated value of the non-anchor transform coefficient can be considered as a weighted average of the corresponding transform coefficients in the anchor blocks weighted by the correlation values in the corresponding positions in the correlation matrix. In other words, the channel decoder 222 can interpolate the value of the non-anchor transform coefficient from the correlation matrix. The channel decoder 222 can output the calculated value of the transform coefficient to the video decoder 224. The channel decoder 222 can perform this process for other transform coefficients. Using the correlation matrix in this way can improve the quality of the reconstructed video data.

[0161] In some examples, the video encoder 210 can reduce the bit width of each transform coefficient in the target transform block by the same amount. In other examples, the video encoder 210 can reduce the bit width of different transform coefficients in the target transform block by different amounts. In some examples, the amount by which the video encoder 210 reduces the bit width of a transform coefficient is related to the distance of the transform coefficient from an anchor transform block. An anchor transform block is a transform block for which the decimation pattern does not apply.

[0162] Figure 7 is a flowchart illustrating example operations of the transmitting device 102 for hybrid decimation of transform blocks in accordance with the techniques of this disclosure. In Figure 7 In the example of FIG. 6, the transmitting device 102 can receive video data from a video source 120 (Figure 7 Video data is obtained (700). Video encoder 210 of transmitting device 102 can generate transform blocks based on the video data (702). For example, video encoder 210 can generate a prediction block by performing intra-prediction on a block of a picture of the video data. Video encoder 210 can use the prediction block to generate residual data. Video encoder 210 can generate a transform block by applying a transform (e.g., DCT, DST, or other transform) to the residual data. In other examples, video encoder 210 can generate a transform block by applying a transform directly to a block of the video data.

[0163] Channel encoder 212 can then determine which transform blocks are anchor transform blocks based on the extraction pattern (704). For example, in the example of extraction pattern 600 using Figure 6 video encoder 210 can determine that every other transform block in the horizontal and vertical directions are anchor transform blocks. In other examples, channel encoder 212 can use other extraction patterns to determine which transform blocks are anchor transform blocks. Channel encoder 212 can store the anchor transform blocks to a memory (e.g., memory 116 (706) of transmitting device 102. Figure 1

[0164] Channel encoder 212 can calculate a correlation matrix for a set of transform blocks (708). Each set of transform blocks includes one or more anchor transform blocks and one or more non-anchor transform blocks. For example, each set of transform blocks can correspond to a different row of transform blocks in Figure 6 In another example, each set of transform blocks can correspond to a group of 2 transform blocks by 2 transform blocks. The number of values in the correlation matrix for a set of transform blocks is the same as the number of transform coefficients in each transform block, respectively. Each value in the correlation matrix corresponds to a different position within a transform coefficient block. For example, the value in position (0, 0) of the correlation matrix corresponds to the transform coefficient at position (0, 0) of each transform coefficient block in the set of transform blocks, the value in position (0, 1) of the correlation matrix corresponds to the transform coefficient at position (0, 1) of each transform coefficient block in the set of transform blocks, and so on. The transform coefficient at position (0, 0) of a transform coefficient block can be referred to as a DC coefficient, and all other transform coefficients can be referred to as AC coefficients.

[0165] Additionally, channel encoder 212 can generate a bit-reduced non-anchor transform matrix (710). The non-anchor transform matrix is a transform matrix other than the anchor transform matrix. For example, referring to Figure 6 ​The transform matrix marked with X can be a non-anchor transform matrix. The transform coefficients in the bit-reduced non-anchor transform matrix can include fewer bits than in the original version of the non-anchor transform matrix. The sending device 102 can then send the anchor transform block, the non-anchor transform block, the bit-reduction value, the correlation matrix, and the error correction data (712).

[0166] The channel encoder 212 can reduce the bits in the non-anchor transform coefficients in one of various ways. For example, the channel encoder 212 can determine a bit-reduction value for each transform coefficient in the non-anchor transform coefficient block. In this example, to compute the bit-reduction value, the channel encoder 212 can compute an interpolated value for the transform coefficient according to the correlation matrix. The channel encoder 212 can use Equation (6) above to compute the interpolated value. The channel encoder 212 can then subtract the interpolated value from the original value of the transform coefficient to compute a first distortion value. The channel encoder 212 can then reduce the number of bits of the original value of the transform coefficient by one. The channel encoder 212 can subtract the interpolated value from the reduced-bit original value of the transform coefficient to compute a second distortion value. The channel encoder 212 can determine whether the second distortion value is acceptable based on the first distortion value and the second distortion value. For example, the channel encoder 212 can compute the average or maximum squared error from the interpolated value. The channel encoder 212 can determine that the second distortion value is acceptable by comparing the second distortion value to a predetermined threshold.

[0167] If the second distortion value is acceptable, the channel encoder 212 can reduce the number of bits of the original value of the transform coefficient value and repeat the process. If the second distortion value is not acceptable, the channel encoder 212 can increase the number of bits of the original value of the transform coefficient. The resulting number of bits of the original value of the transform coefficient that is reduced is the bit-reduction value.

[0168] In some examples, channel encoder 212 can determine the bit reduction value for the non-anchor transform coefficient block as a whole. In this example, to calculate the bit reduction value for the transform coefficient block, channel encoder 212 can calculate an interpolated value for each transform coefficient from the correlation matrix, e.g., as described above. Video encoder 210 can then subtract the interpolated value from the original value of the transform coefficient and use the resulting difference to calculate a first distortion value. For example, video encoder 210 can calculate the first distortion value as a mean squared error. Video encoder 210 can then reduce the number of bits for the original value of each transform coefficient by one. Video encoder 210 can subtract the interpolated value from the reduced bit original value of the transform coefficient and use the resulting value to calculate a second distortion value (e.g., using a mean squared error). Channel encoder 212 can determine whether the second distortion value is acceptable based on the first distortion value and the second distortion value. If the second distortion value is acceptable, video encoder 210 can again reduce the number of bits for the original value of the transform coefficient value and repeat the process. If the second distortion value is not acceptable, video encoder 210 can increase the number of bits for the original value of the transform coefficient. The resulting number of bits for the original value of the transform coefficient reduction is the bit reduction value.

[0169] Accordingly, in some examples, transmitting device 102 can obtain video data from a video source. Transmitting device 102 can generate transform blocks based on the video data. Transmitting device 102 can determine which of the transform blocks are anchor transform blocks. Transmitting device 102 can calculate a correlation matrix for the set of transform blocks. Additionally, transmitting device 102 can generate bit-reduced non-anchor transform matrices. Transmitting device 102 can transmit the anchor transform blocks, the non-anchor transform blocks, and the correlation matrix to a receiving device. In some examples, transmitting device 102 can receive an indication of a decimation pattern from receiving device 104.

[0170] Figure 8 is a flowchart illustrating example operations of receiving device 104 for hybrid decimation of transform blocks in accordance with the techniques of this disclosure. In Figure 8 In the example of FIG. 8, receiving device 104 can receive anchor transform blocks, non-anchor transform blocks, one or more bit reduction values, and a correlation matrix for the non-anchor blocks (800).

[0171] Further, in Figure 8In the example above, channel decoder 222 of receiving device 104 can compute an interpolated value for a current non-anchor transform coefficient (802). The current non-anchor transform coefficient is a transform coefficient of one of the non-anchor transform blocks. Channel decoder 222 can compute the interpolated value for the current non-anchor transform coefficient based on the correlation matrix of the non-anchor block. Channel decoder 222 can compute the interpolated value for the current non-anchor transform coefficient in one of various ways. For example, in some examples, channel decoder 222 can apply a machine learning model that takes as input one or more non-anchor transform coefficients (including the current non-anchor transform coefficient), one or more anchor transform coefficients, and the correlation matrix of the non-anchor transform coefficient block. In this example, the machine learning model can output the interpolated value for the current non-anchor transform coefficient. In this example, the machine learning model can be implemented as a neural network model, a support vector machine, a regression model, or another type of machine learning model.

[0172] In another example, the correlation matrix can include values that indicate a correlation between the current non-anchor transform coefficient and each corresponding anchor transform coefficient of the one or more anchor transform coefficient blocks. Video decoder 224 can compute the interpolated value for the current non-anchor transform coefficient as:

[0173]

[0174] In the equation above, t int is the current non-anchor transform coefficient, a i is an anchor transform coefficient, c i is a correlation value that indicates a correlation between the current non-anchor transform coefficient and a i , and n indicates a number of anchor transform coefficients from which the current non-anchor transform coefficient is derived. In this example, the values of c0to c n may add up to 1.

[0175] Additionally, video decoder 224 can compute a reconstructed value for the non-anchor transform coefficient (804). Video decoder 224 can compute the reconstructed value for the non-anchor transform coefficient based on the interpolated value for the current non-anchor transform coefficient and a transmitted value for the non-anchor transform coefficient. The transmitted value for the non-anchor transform coefficient is included in the received non-anchor transform block. In some examples, video decoder 224 computes the reconstructed value for the non-anchor transform coefficient as an average of the interpolated value for the current non-anchor transform coefficient and the transmitted value for the non-anchor transform coefficient.

[0176] The video decoder 224 can determine whether there are any remaining non-anchor transform coefficients in the non-anchor transform block (806). If there is one or more remaining non-anchor transform coefficients in the non-anchor transform block (the "Yes" branch of 806), the video decoder 224 can repeat steps 802-806 with another non-anchor transform coefficient. The video decoder 224 can continue to do so until there are no remaining non-anchor transform coefficients (the "No" branch of 806). In this way, the video decoder 224 can calculate a reconstructed value for each non-anchor transform coefficient.

[0177] In this way, the receiving device 104 can receive the system bits for the anchor transform block, the system bits for the non-anchor transform block, and the correlation matrix. The system bits for the anchor transform block can represent the transform coefficients in the anchor transform block. The system bits for the non-anchor transform block can represent a reduced bit-depth version of the original transform coefficients in the non-anchor transform block. As part of performing the reconstruction operation, the receiving device 104 can calculate, for each non-anchor transform coefficient in the non-anchor transform block, an interpolated value for the non-anchor transform coefficient based on the correlation matrix and the corresponding anchor transform coefficient. The receiving device 104 can calculate, at the receiving device, a reconstructed value for the non-anchor transform coefficient based on the interpolated value for the non-anchor transform coefficient in the error corrected encoded video data and the value for the non-anchor transform coefficient.

[0178] In some examples, the receiving device 104 can adaptively select a decimation pattern for reducing or eliminating bits of a particular transform block. In such examples, the receiving device 104 can deliver the selected decimation pattern back to the sending device 102. The sending device 102 can then use the selected decimation pattern in one or more pictures of the video data.

[0179] Figure 9 is a conceptual diagram illustrating an example decimation pattern 900 adaptively selected by the receiving device 104 according to one or more techniques of this disclosure. As with the decimation pattern 600 of Figure 6 In contrast to the decimation pattern 600, the non-anchor blocks in the decimation pattern 900 do not necessarily occur at regular intervals or intervals.

[0180] The receiving device 104 can determine the decimation pattern based on information about a previous picture of the video data. The previous picture can or can not have already been decimated. In some examples, the receiving device 104 can transmit a request to the sending device 102 for an un-decimated version of the picture. The sending device 102 can transmit the un-decimated version of the picture to the receiving device 104 in response to the request. After receiving the un-decimated version of the picture, the receiving device 104 can determine the decimation pattern based on the un-decimated version of the picture. For example, the receiving device 104 can perform a rate-distortion optimization process that evaluates a plurality of potential decimation patterns in order to identify which decimation pattern results in the best combination of bit rate and distortion.

[0181] In some examples, the sending device 102 and the receiving device 104 can continue to use the selected decimation pattern for a predetermined number of pictures, after which the sending device 102 and / or the receiving device 104 can adaptively select another decimation pattern. In some examples, the sending device 102 can transmit a message to the receiving device 104 requesting that the sending device 102 select another decimation pattern. In some examples, the receiving device 104 can determine that an event or condition has occurred that will make it advantageous to select another decimation pattern. For example, the receiving device 104 can determine that it is advantageous to select another decimation pattern when the receiving device 104 determines that a scene change has occurred, that motion in the video data has crossed one or more thresholds, or that other characteristics of the video data have changed.

[0182] The receiving device 104 can signal the selected decimation pattern to the sending device 102 in one of a variety of ways. For example, the receiving device 104 can signal the selected decimation pattern to the sending device 102 by indicating a difference from an existing decimation scheme, such as the decimation scheme currently in use. For example, in this example, the receiving device 104 can indicate the selected decimation scheme to the sending device 102 by specifying a change in downsampling or upsampling along a particular axis, specifying a change in a particular region of a picture, eliminating a particular transform block, enabling a particular transform block, and so on.

[0183] In some examples, there can be a predefined mapping of index values to predefined decimation patterns. In such examples, the receiving device 104 can select a decimation pattern from the predefined decimation patterns and signal an index value of the selected decimation pattern to the sending device 102.

[0184] Figure 10 is a block diagram illustrating example components of a sending device and a receiving device in accordance with the techniques of this disclosure. In Figure 10 the example, the sending device 102 can include the same components as shown in Figure 2 However, in the example of Figure 10 the receiving device 104 can additionally include a reliability unit 1002. Unless otherwise noted, the similarly named components of the sending device 102 and the receiving device 104 in Figure 2 and Figure 10 perform the same functions.

[0185] In general, higher significant bits (MSBs) of transform coefficients can be more easily accurately predicted than lower significant bits (LSBs) of transform coefficients. This is because MSBs translate to higher Euclidean distances in the video picture domain. Additionally, blocks of a video picture that are experiencing high motion can be more difficult to accurately predict than blocks that are in a low motion region.

[0186] According to one or more techniques of this disclosure, the transmitting device 102 and the receiving device 104 can implement a system in which bit-level reliability values are used. The use of bit-level reliability values can enable the transmitting device 102 and the receiving device 104 to properly weight a priori information. This can result in an improvement in system performance (e.g., a reduction in the amount of data transferred and / or an improvement in video quality). For example, if "soft" information is used, decoding performance can be improved. In other words, when the receiving device 104 is "informed" of the reliability of each bit (what is the a priori probability that the bit value is "0" or "1"), the receiving device 104 can utilize this information and can improve performance.

[0187] In the example of FIG. 2, the reliability unit 1002 of the receiving device 104 receives encoded video data from the video encoder 228 of the receiving device 104. The encoded video data can include transform coefficients of a transform block of the video data. Additionally, in some examples, the reliability unit 1002 receives prediction quality information from the picture estimation unit 226 of the receiving device 104. Figure 10

[0188] In the constant scaling process, for each bit position of the transform coefficients of the transform block, the prediction quality information includes a reliability value for the bit position. For example, the most significant bit of the transform coefficients has a first reliability value, the second most significant bit of the transform coefficients has a second reliability value, the third most significant bit of the transform coefficients has a third reliability value, and so on. The reliability value for a bit position is a measure of the likelihood that the bit at the bit position has an incorrect value. For example, a bit at a bit position can have an incorrect value when the bit value is predicted to be '0' but the actual value is '1' or the bit is predicted to be '1' but the actual value is '0'.

[0189] The picture estimation unit 226 can determine the reliability value for a bit position by collecting statistics regarding the error rate that occurs at the bit position. For example, the picture estimation unit 226 can determine a probability that the most significant bit of the transform coefficients contains an error, determine a probability that the second most significant bit of the transform coefficients contains an error, determine a probability that the third most significant bit of the transform coefficients contains an error, and so on. The picture estimation unit 226 can collect these statistics by counting the number of times that a predicted bit (i.e., a bit in an estimated picture) is incorrect in a plurality of test events. For example, the picture estimation unit 226 can estimate a picture, and the video encoder 228 can encode video data of the estimated picture. Subsequently, the video decoder 224 can decode error-corrected encoded video data of the picture. The picture estimation unit 226 can compare the bits in the transform coefficients of the encoded video data of the estimated picture and the error-corrected video data of the picture to determine whether the bits of the encoded video data of the estimated picture are incorrect. ​

[0190] The reliability unit 1002 can convert the probability values to LLR values. In some examples, the reliability unit 1002 can convert the probability values to LLR values using the following formula:

[0191]

[0192] In the above formula, M represents the LLR value, P error represents the error probability value, and In represents the natural logarithm function. In some examples, the LLR value is the reliability value.

[0193] Figure 11 A chart of example error probabilities and corresponding log-likelihood ratio (LLR) absolute values according to one or more techniques of this disclosure is illustrated. In the example of FIG. 11, 150 bits are used to represent each transform block. The plot 1100 plots the error probability for each bit position within a transform block. As can be seen in the plot 1100, bits at certain positions have a greater error probability. The plot 1102 shows the error probabilities converted to LLR absolute values. The LLR absolute values can be scaled. Figure 11

[0194] In some examples, the receiving device 104 uses a dynamic scaling process. In the dynamic scaling process, the picture estimation unit 226 dynamically determines the prediction quality information based on the video data. For example, some regions of a picture are more difficult to predict than more easily predictable regions (e.g., regions with reduced prediction accuracy). Examples of more difficult to predict picture regions can include regions with higher motion. For example, if the total magnitude of the motion vectors in a region crosses a threshold, the picture estimation unit 226 can determine that the region is a difficult to predict region. Accordingly, the picture estimation unit 226 can identify such regions and generate reliability values for the bits of the transform coefficients of a transform block based at least in part on whether the transform block is inside or outside of such regions. In some examples, the picture estimation unit 226 determines the reliability values for the bits of the transform coefficients based on general statistics of errors for the locations of the bits modified based on whether the transform block containing the transform coefficients is in a difficult to predict region.

[0195] ​The reliability unit 1002 can use the reliability values to scale bits of transform coefficients in encoded video data generated by the video encoder 228. For example, bits of encoded video data generated by the video encoder 228 can be considered “hard” bits and can have a value of exactly 0 or exactly 1. The reliability unit 1002 can use prediction quality information generated by the picture estimation unit 226 and encoded video data generated by the video encoder 228 to determine “soft” bits between 0 and 1. By way of example, if a bit of encoded video data has a value of 1, the reliability unit 1002 can generate a “soft” value for the bit by multiplying the LLR absolute value of the bit by -1 (i.e., -1). If a bit of encoded video data has a value of 0, the reliability unit 1002 can generate a “soft” value for the bit by multiplying the LLR absolute value of the bit by +1 (i.e., +1). Thus, the “soft” or scaled values of bits of transform coefficients can be M or -M.

[0196] Thus, in some examples, each bit can be transformed into a scaled value that has a more positive value if there is a greater confidence that the bit has a value of 0 and a more negative value if there is a greater confidence that the bit has a value of 1. The reliability unit 1002 provides the scaled values as a-priori information to the channel decoder 222.

[0197] The channel decoder 222 performs a channel decoding process using the scaled values. For example, the reliability unit 1002 and the channel encoder 212 can encode the encoded video data into a codeword (e.g., a low density parity check code). The bits of the codeword generated by the reliability unit 1002 can scale the bits of the codeword as described above. The bits of the codeword generated by the channel encoder 212 can change during delivery through the channel 230 such that the bits of the codeword can be received as values between -1 and 1. The channel decoder 222 can apply an LPDC decoding process to the codeword to correct errors in the codeword. The corrected values of the bits in the codeword are 0 or 1. The LPDC decoding process can then convert the codeword from encoded video data back to the original data. In other examples, other encoding schemes can be used. In this example, the error correction data received by the channel decoder 222 can include cyclic redundancy check (CRC) data that is not used in the LPDC decoding process. In this way, the channel decoder 222 can determine the value of each bit of the transform coefficients.

[0198] In some examples, the channel encoder 212 can use prediction quality feedback to order the bits of the transform coefficients before applying an unequal protection channel coding, such as a polar code or a ridge code. Unequal protection channel coding involves allocating coding redundancy according to the importance of the information bits. For example, the channel encoder 212 can use the reliability data to protect different bits according to the predictability of the receiving device 104.

[0199] In some examples, the reliability unit 1002 communicates prediction quality feedback to the transmitting device 102. The video encoder 210 can adjust one or more encoding parameters of the video encoding process applied by the video encoder 210 to the video data. For example, the transmitting device 102 can determine a compression rate of the limited video encoding process based on the predictability (reliability) of the receiving device 104. For example, if the predictability is low, the transmitting device 102 can reduce the quality of the encoded video by reducing the number of transform coefficients or the bit width of each transform coefficient, such that less information is delivered.

[0200] In some examples, the transmitting device 102 can adjust one or more channel encoding parameters used by the channel encoder 212 based on the prediction quality feedback. For example, the channel encoder 212 can use an unequal error protection code, where the protection depends on the prediction quality feedback. In some examples, the channel encoder 212 can select different LDPC graphs depending on the reliability. In some examples, the channel encoder 212 can change the encoding scheme used to generate error correction data to increase the error correction capability of bit locations or regions of pictures having lower reliability, or to decrease the error correction capability of bit locations or regions of pictures having higher reliability.

[0201] In some examples, the transmitting device 102 can update one or more bit puncturing parameters used by the puncturing unit 214 based on the prediction quality feedback. For example, the puncturing unit 214 can change the puncturing pattern to allow more error correction data to be transmitted for bit locations and / or picture regions having lower reliability. Thus, the transmitting device 102 can avoid puncturing bits having lower reliability. In some examples, the puncturing unit 214 can change the puncturing pattern to allow less error correction data to be transmitted for bit locations and / or picture regions having higher reliability.

[0202] In some examples, the prediction quality feedback communicated by the reliability unit 1002 to the transmitting device 102 applies to the entire picture. In some examples, the reliability unit 1002 can communicate prediction quality feedback to the transmitting device 102 on a per region basis. Each region can be a defined region within the picture. The reliability unit 1002 can communicate prediction quality feedback for some regions of the picture but not others.

[0203] In some examples, the reliability unit 1002 can periodically transmit prediction quality feedback to the sending device 102. For example, the reliability unit 1002 can transmit prediction quality feedback to the sending device 102 every N pictures, where N is an integer value. In some examples, the reliability unit 1002 transmits prediction quality feedback to the sending device 102 after a particular number of groups of pictures (GOPs) are completed. In other examples, the reliability unit 1002 can transmit prediction quality feedback on a non-periodic basis, such as in response to a particular condition or event.

[0204] In some examples, the prediction quality feedback can include prediction quality data based on one or more noise models, such as a Gaussian noise model or a Laplacian noise model. Noise model parameters can control the one or more noise models. The reliability unit 1002 can transmit the noise model parameters to the sending device 102. The use of noise models is an alternative to collecting error statistics. In this mode, the prediction errors (error statistics between the estimated pictures estimated by the picture estimation unit 226 and the actual pictures reconstructed by the video decoder 224) can be modeled using a few parameters that describe the error distribution function. The receiving device 104 can more easily deliver the parameters of the noise model to the sending device 102, as the parameters of the noise model can include less data than per-bit statistics. The sending device 102 can use the noise model in the same manner as the sending device 102 can use other types of prediction quality feedback.

[0205] In some examples, instead of the reliability unit 1002 receiving prediction quality information from the picture estimation unit 226 of the receiving device 104, the video encoder 210 can generate and transmit the prediction quality information to the reliability unit 1002. The video encoder 210 can determine the prediction quality information based on a priori information about the video encoding process. For example, the video encoder 210 can evaluate the reliability per bit based on compression parameters (e.g., MSBs are more reliable than LSBs, lower frequency transform coefficients are more reliable than higher frequency transform coefficients, etc.). The video encoder 210 can perform prediction processing to evaluate the statistics itself, with pre-defined statistics for different sets of light compression parameters. Additionally, the sending device 102 can use other sensors to evaluate instantaneous motion and adjust the reliability accordingly.

[0206] Figure 12 is a flowchart illustrating example operations of a sending device 102 using scaled bits in accordance with the techniques of this disclosure. In Figure 12In the example of FIG. 12, the transmitting device 102 can obtain video data from a video source 120 (1200), for example. In addition, the transmitting device 102 can obtain prediction quality feedback (1202). In some examples, the transmitting device 102 can obtain the prediction quality feedback from the receiving device 104. The prediction quality feedback includes bit reliability information. In some examples, the prediction quality feedback is expressed in terms of noise model parameters, such as parameters of a Gaussian noise model or a Laplacian noise model.

[0207] The transmitting device 102 can adapt one or more of video coding parameters, channel coding parameters, or bit puncturing parameters based on the prediction quality feedback (1204). The video encoder 210 of the transmitting device 102 can perform video coding processing to generate encoded video data (1206). The video coding processing can be controlled by the video coding parameters. For example, the video coding parameters can control the number of transform coefficients included in a transform block, the number of bits included in a transform coefficient, and so on. In some examples, the video coding parameters include a quantization parameter, and the video encoder 210 can adapt the quantization parameter based on the prediction quality feedback. For example, if the prediction quality feedback indicates low reliability, the quantization parameter can be decreased to decrease quantization levels. As part of performing the video coding processing, the video encoder 210 can quantize transform coefficients of transform blocks of one or more pictures using the quantization parameter.

[0208] The channel encoder 212 of the transmitting device 102 can perform channel coding processing on the scaled bits to generate channel encoded data (1208). The channel coding processing can be controlled by the channel coding parameters. For example, the channel coding parameters can include an LDPC graph that controls which LDPC graph the channel coding processing uses to generate a codeword, the channel coding parameters can control error correction capability, and so on. For example, the channel coding parameters include an LDPC graph, the channel encoder 212 can adapt the LPDC graph, and use the LDPC graph to generate a codeword for transmission to the receiving device.

[0209] In addition, the puncturing unit 214 of the transmitting device 102 can perform bit puncturing processing on the error correction data generated by the channel encoder 212 (1210). The bit puncturing processing can be controlled by the bit puncturing parameters. For example, the prediction quality feedback can indicate that certain portions of the encoded video data are less reliable. Accordingly, the transmitting device 102 can adjust the bit puncturing parameters to reduce bit puncturing on the error correction data for the less reliable portions of the encoded video data. The transmitting device 102 can transmit the channel encoded data and the bit punctured error correction data to the receiving device 104 (1212).

[0210] Figure 13 FIG. 13 is a flowchart illustrating example operations of a receiving device 104 that uses scaled bits in accordance with the techniques of this disclosure. In Figure 13In the example of FIG. 13, the receiving device 104 can obtain error correction data from the sending device (1300). The error correction data provides error correction information about the pictures of the video data.

[0211] The picture estimation unit 226 can generate prediction data for a picture (1302). The prediction data for a picture can include a prediction for a block of the picture based at least in part on one or more previously reconstructed pictures of the video data. For example, the picture estimation unit 226 can use inter prediction and / or intra prediction to generate the prediction for the block.

[0212] Further, the receiving device 104 can generate encoded video data based on the prediction data for the picture (1304). For example, the video encoder 228 of the receiving device 104 can perform a video encoding process that generates the encoded video data. The encoded video data includes transform blocks that include transform coefficients.

[0213] The receiving device 104 can scale bits of the transform coefficients of the transform block based on the reliability values for the bit positions (1306). In some examples, the receiving device 104 generates the reliability values. For example, the receiving device 104 can generate the reliability values based on statistics about the occurrence of errors in the bit positions. In some instances, the receiving device 104 can generate the reliability values based on reliability characteristics of individual regions of the pictures of the video data. In some examples, the receiving device 104 can generate the reliability values based on a noise model. Further, in some examples, the receiving device 104 can transmit the reliability values to the sending device 102. In other examples, the receiving device 104 can receive the reliability values from the sending device 102.

[0214] Additionally, the channel decoder 222 of the receiving device 104 can use the error correction data to generate error corrected encoded video data to perform error correction operations on the scaled bits of the transform coefficients of the transform block (1308).

[0215] The video decoder 224 of the channel decoder 222 can reconstruct the picture based on the error corrected encoded video data (1310).

[0216] This disclosure describes techniques that can reduce the complexity of video encoding at a sending device (e.g., an extended reality (XR) headset). The sending device can obtain multi-view video data. The multi-view video data can include pictures from two or more viewpoints. For example, the XR headset can include two cameras for stereoscopic views of a scene that a user is viewing. In this example, the multi-view video data can include pictures from each of the cameras.

[0217] Processing content of multi-view video data can consume considerable processing resources. For example, in the context of augmented reality (AR) or mixed reality (MR), considerable processing resources can be needed in order to determine where to position virtual elements and how the virtual elements should appear. Multi-view video data can help process the virtual elements. As an example, when a virtual element is to be positioned in a shadowed area of a scene, the same virtual element can need to be darker, and when the virtual element is positioned in a sunny area of a scene, the same virtual element can need to be brighter. As another example, a system can need to analyze the content of a scene to determine whether a virtual element will be occluded by a physical element in the scene, such as a rock or a tree. Multi-view video data can be used to determine the depth of objects in the scene. In order to keep an XR headset lightweight and preserve battery power at the XR headset, it can be desirable to minimize the processing of video data performed at the XR headset. Thus, processing video data at another device, such as a user's smartphone or other nearby device, can help reduce the need for processing resources at the XR headset.

[0218] While multi-view video data can be very useful in certain situations, simply transmitting unencoded multi-view video data can be impractical because the amount of data needed to transmit multiple concurrent streams of video data simultaneously can be very large. However, there is often considerable redundancy between pictures of different views of multi-view video data. As an example, what a person's left eye sees is often not as different from what a person's right eye sees. Thus, video compression techniques have been developed to reduce this redundancy in order to reduce the amount of data needed to transmit multi-view video data.

[0219] However, some techniques for multi-view video encoding themselves require considerable computational resources. For example, a video encoder can determine differences between a set of two or more concurrent pictures to determine a depth map of a scene. A depth map is an array of values that indicate the depth / distance of objects shown in a picture from a camera. In this example, one of the concurrent pictures can be an anchor picture and one or more of the concurrent pictures can be non-anchor pictures. The video encoder can use the depth map to compute disparity vectors for blocks in the non-anchor pictures. A disparity vector for a block indicates a lateral displacement between the block and a corresponding block in another concurrent picture, e.g., the anchor picture. Generally, blocks representing deeper objects have disparity vectors of lower magnitude than blocks representing nearer objects. The video encoder can use the disparity vectors for the blocks to determine a predicted block, generate residual data based on the predicted block, apply a transform to the residual data, quantize transform coefficients of the resulting transform block, and signal the quantized transform coefficients. In this example, generating the depth map can involve considerable computational resources.

[0220] In another example, the lighting levels can differ between concurrent pictures from different viewpoints. These differences in lighting can break the coding efficiency of multi-view video coding. Thus, lighting compensation can be applied to the non-anchor pictures to temporarily modify the lighting levels of the non-anchor pictures according to lighting compensation factors to make the non-anchor pictures more consistent with the lighting levels of the anchor pictures during video coding. The original lighting levels of the non-anchor pictures can be restored during video decoding. Determining the lighting compensation factors can consume computational resources.

[0221] The techniques of this disclosure can shift some processing associated with multi-view video coding from a sending device (e.g., an XR headset) to a receiving device (e.g., a mobile device). For example, a sending device can obtain a first set of multi-view pictures of video data. The first set of multi-view pictures includes a first picture and a second picture. The first picture is from a first viewpoint, and the second picture is from a second viewpoint. The sending device can send first encoded video data to a receiving device. The first encoded video data is based on the first set of multi-view pictures. The sending device can receive a multi-view coding hint from the receiving device. In addition, the sending device can obtain a second set of multi-view pictures of video data. The second set of multi-view pictures includes a third picture and a fourth picture. The third picture is from the first viewpoint, and the fourth picture is from the second viewpoint. The sending device can perform a multi-view coding process on the second set of multi-view pictures based on the multi-view coding hint received from the receiving device to generate second encoded video data. The multi-view coding process reduces inter-view redundancy between the third picture and the fourth picture. The sending device can then send the second encoded video data to the receiving device.

[0222] Similarly, a receiving device can obtain first encoded video data from a sending device. The first encoded video data is based on a first set of multi-view pictures of video data. The first set of multi-view pictures can include a first picture and a second picture. The first picture is from a first viewpoint, and the second picture is from a second viewpoint. The receiving device can determine a multi-view coding hint based on the first encoded video data. The receiving device can communicate the multi-view coding hint to the sending device. In addition, the receiving device can obtain second encoded video data from the sending device. The second encoded video data is based on a second set of multi-view pictures including a third picture and a fourth picture. The second encoded video data is encoded using a multi-view coding process that reduces inter-view redundancy between the third picture and the fourth picture based on the multi-view coding hint.

[0223] Because the receiving device determines the multi-view coding hint and communicates the multi-view coding hint to the sending device, the burden of determining the multi-view coding hint can be shifted from the sending device to the receiving device. This can reduce the demand on resources at the sending device.

[0224] Referring to Figure 2based on the multi-view encoding cues obtained from the receiving device 104. For example, the multi-view encoding cues can include a depth map. In this example, the video encoder 210 can use the depth map to estimate disparity vectors for blocks in a non-anchor view of the multi-view video data. The video encoder 210 can use the disparity vectors for a current block of a current picture to determine a prediction block for the current block based on samples of a concurrent reference picture. The concurrent reference picture has the same picture order count (POC) value as the current picture. The video encoder 210 can determine residual data for the current block based on original samples of the current block and the prediction block for the current block. The video encoder 210 can apply one or more transforms to the residual data to generate one or more transform blocks. The video encoder 210 can quantize transform coefficients in the transform blocks. The encoded video data generated by the video encoder 210 can be based on the quantized transform coefficients.

[0225] In some examples, the multi-view encoding cues can include one or more illumination compensation factors. When encoding a current picture of the multi-view video data, the video encoder 210 can modify each sample of the current picture based on the one or more illumination compensation factors. In some examples, different illumination compensation factors can be applied to different regions of the current picture. Modifying the samples of the current picture in this way can make the illumination level of the current picture more consistent with the illumination level of the concurrent reference picture. After modifying the samples of the current picture, the video encoder 210 can perform multi-view encoding processing, such as the multi-view encoding processing described in the previous paragraph, to encode blocks of the current picture.

[0226] Further, according to some examples of the disclosure, the video decoder 224 can perform multi-view decoding processing. For example, the video decoder 224 can use a disparity vector for a block of a current picture to generate a prediction block. The video decoder 224 can use the prediction block and residual data received from the channel decoder 222 to reconstruct samples of the block of the current picture. In some examples in which the video encoder 210 applied illumination compensation to the picture, the video decoder 224 can use illumination parameters to reverse the illumination compensation applied to the picture. In other examples, the video decoder 224 can apply other multi-view decoding operations.

[0227] Further, according to one or more techniques of the disclosure, the video decoder 224 can determine multi-view encoding cues based on the encoded video data received from the transmitting device 102. For example, the video decoder 224 can determine a depth map, illumination compensation parameters, and other information that can be used in multi-view encoding operations. The receiving device 104 can send the multi-view encoding cues back to the transmitting device 102 so that the transmitting device 102 can use the multi-view encoding cues to perform multi-view encoding processing on subsequent pictures.

[0228] Picture estimation unit 226 can generate an estimate of a next picture of the video data. In some examples, picture estimation unit 226 can estimate a picture based on one or more previously reconstructed reference pictures associated with different views. For example, picture estimation unit 226 can extrapolate content of a picture from pictures in the same time instance using information from pictures of previous time instances (e.g., disparity vectors or depth maps). In another example, picture estimation unit 226 can extrapolate a picture based on one or more pictures associated with the same view regardless of pictures associated with other views in substantially the same manner as discussed elsewhere in this disclosure regarding picture estimation unit 226 estimating pictures of single view video data.

[0229] Video encoder 228 can perform the same operations on the estimated next picture as video encoder 210. For example, video encoder 228 can perform intra prediction to predict a block, generate residual data using the predicted block and a corresponding predicted block of the picture generated by picture estimation unit 226. In some examples, video encoder 228 can perform the same multi-view encoding process as video encoder 210 using the multi-view encoding cues. Video encoder 228 can apply a transform (e.g., a DCT transform) to the residual data to generate transform coefficients. Video encoder 228 can apply quantization to the transform coefficients.

[0230] Figure 14 is a flowchart illustrating an example exchange of data between a transmitting device 102 and a receiving device 104 related to multi-view processing in accordance with one or more techniques of this disclosure. In Figure 14 In the example of FIG. 14, transmitting device 102 can obtain a first set of multi-view pictures (1400). Transmitting device 102 can transmit first encoded video data based on the first set of multi-view pictures to receiving device 104. In some examples, transmitting device 102 performs light compression on the first set of multi-view pictures to generate the first encoded video data. In other examples, the first encoded video data can include an unencoded version of the first set of multi-view pictures.

[0231] Receiving device 104 can perform multi-view view processing on the first set of multi-view pictures (1402). For example, receiving device 104 can decode the first set of multi-view pictures if necessary. In addition, receiving device 104 can determine multi-view encoding cues, e.g., as described elsewhere in this disclosure. Receiving device 104 can transmit the multi-view encoding cues to transmitting device 102.

[0232] Further, in Figure 14In the example, transmitting device 102 may obtain a second set of multi-view images (1404). Transmitting device 102 may perform multi-view encoding processing on the second set of multi-view images to generate second encoded video data (1406). The second encoded video data may include encoded anchored and secondary (non-anchored) images. Receiving device 104 may perform multi-view decoding on the second encoded video data to reconstruct the second set of multi-view images (1408). Receiving device 104 may also perform multi-view processing on the second set of multi-view images to determine updated multi-view encoding hints (1410). Receiving device 104 may transmit the updated multi-view encoding hints to transmitting device 102. Transmitting device 102 may use the updated multi-view encoding hints for multi-view encoding of subsequent sets of multi-view images.

[0233] Figure 15 This is a flowchart illustrating an example operation of a transmitting device 102 for multi-view processing according to the technology of this disclosure. Figure 5 In the example, transmitting device 102 can obtain a first set (1500) of multi-view images of video data. The first set of multi-view images includes a first image and a second image. The first image is from a first viewpoint, and the second image is from a second viewpoint. The communication interface 118 of transmitting device 102 ( Figure 1 The transmitting device 102 can transmit first encoded video data to the receiving device 104 (1502). The first encoded video data is a first set of multi-view images. The transmitting device 102 can receive multi-view encoding prompts (1504) from the receiving device 104. In some examples, the multi-view encoding prompts include one or more of the following: relative shifts between blocks of images from the first and second viewpoints, brightness corrections between images from the first and second viewpoints, inter-block shifts between anchor blocks and reconstructed blocks, or motion data for reference shifts. The transmitting device 102 can receive the multi-view encoding prompts in one of a variety of ways. For example, the transmitting device 102 can receive the multi-view encoding prompts via Uplink Control Information (UCI) / Media Access Control-Control Element (MAC-CE) messages, Radio Resource Control (RRC) messages, or another type of message.

[0234] Further, the sending device 102 can obtain a second set of multiview pictures of the video data (1506). The second set of multiview pictures includes a third picture and a fourth picture. The third picture is from the first viewpoint and the fourth picture is from the second viewpoint. The video encoder 210 can perform a multiview encoding process on the second set of multiview pictures based on the multiview encoding hints received from the receiving device to generate second encoded video data (1508). The multiview encoding process reduces inter-view redundancy between the third picture and the fourth picture. The sending device 102 can then send the second encoded video data to the receiving device 104 (1510).

[0235] The operations of 1504 can be performed multiple times for sets of subsequent multiview pictures. For example, after sending the second encoded video data to the receiving device, the sending device 102 can receive updated multiview encoding hints from the receiving device 104. The sending device 102 can obtain a third set of multiview pictures of the video data. The third set of multiview pictures can include a fifth picture and a sixth picture, the fifth picture being from the first viewpoint and the sixth picture being from the second viewpoint. The video encoder 210 of the sending device 102 can encode the third set of multiview pictures based on the updated multiview encoding hints received from the receiving device to generate third encoded video data. The sending device 102 can send the third encoded video data to the receiving device 104. Figure 15

[0236] Figure 16 is a flowchart illustrating example operations of the receiving device 104 for multiview processing in accordance with the techniques of this disclosure. In the example of Figure 16 , the receiving device 104 can obtain first encoded video data from the sending device 102 (1600). For example, the receiving device 104 can obtain the first encoded video data via the communication interface 134 (1602). Figure 1 The first encoded video data is based on a first set of multiview pictures of the video data. The first set of multiview pictures can include a first picture and a second picture. The first picture is from the first viewpoint and the second picture is from the second viewpoint. The receiving device 104 can determine multiview encoding hints based on the first encoded video data (1602).

[0237] The receiving device 104 can send the multiview encoding hints to the sending device 102 (1604). The receiving device 104 can send the multiview encoding hints in one of various ways. For example, the receiving device 104 can send the decimation pattern indication via an uplink control information (UCI) / medium access control-control element (MAC-CE) message, a radio resource control (RRC) message, or another type of message.

[0238] ​Additionally, the receiving device 104 can obtain second encoded video data from the sending device 102 (1606). The second encoded video data is based on a second set of multi-view pictures that includes a third picture and a fourth picture. The second encoded video data is encoded using a multi-view encoding process that reduces inter-view redundancy between the third picture and the fourth picture based on the multi-view encoding cues. A video decoder 224 of the receiving device 104 can decode the second encoded video data.

[0239] In some examples, the multi-view encoding cues include a depth map that indicates depths of objects represented in the first picture and the second picture. As part of determining the multi-view encoding cues, the receiving device 104 can determine the depth map based on the first picture and the second picture. In some examples, the multi-view encoding cues include one or more illumination compensation factors, and as part of determining the multi-view encoding cues, the receiving device 104 can determine the illumination compensation factors based on the first and second pictures.

[0240] Figure 16 The process of 1604 can be repeated multiple times. For example, the receiving device 104 can determine second multi-view encoding cues based on the second encoded video data. The receiving device 104 can send the second multi-view encoding cues to the sending device 102. Subsequently, the receiving device 104 can obtain third encoded video data from the sending device 102. The third encoded video data is based on a third set of multi-view pictures that includes a fifth picture and a sixth picture, and the third encoded video data is encoded using the multi-view encoding process that reduces inter-view redundancy between the fifth picture and the sixth picture based on the second multi-view encoding cues.

[0241] According to one or more techniques of this disclosure, the sending device 102 can receive a thinning pattern indication from the receiving device 104. The thinning pattern indication can indicate a thinning pattern. As described in more detail elsewhere in this disclosure, the receiving device 104 can determine the thinning pattern. The thinning pattern can be a pattern for which encoded video data is not transmitted.

[0242] The sending device 102 can receive the thinning pattern indication in one of various ways. For example, the sending device 102 can receive the thinning pattern indication via an uplink control information (UCI) / medium access control-control element (MAC-CE) message, a radio resource control (RRC) message, a sidelink control information (SCI), or another type of message.

[0243] The sending device 102 can apply the extraction pattern to the encoded video data generated by the video encoder 210, thereby generating extracted video data. For example, the extraction pattern can indicate a pattern of skipping transmission of encoded video data for complete pictures. Thus, in this example, the sending device 102 (e.g., a channel encoder 212 of the sending device 102) can transmit encoded video data for some pictures according to the indicated pattern, and not transmit encoded video data for other pictures. For example, the sending device 102 can skip transmission of encoded video data for every other picture. In another example, the sending device 102 can transmit encoded video data for one picture, and then not transmit encoded video data for the next two or more pictures.

[0244] In another example, the extraction pattern can indicate a pattern of skipping transmission of encoded video data for specified regions within a picture. The specified regions of a series of pictures can not change much, if at all, from picture to picture. For example, the background of a scene from a static viewpoint can not change significantly, while changes occur in a more limited region of interest. Because regions outside the region of interest change little, such regions can be more easily and accurately predicted. Thus, in accordance with the techniques of this disclosure, the receiving device 104 can identify regions outside the region of interest. Thus, the sending device 102 can transmit encoded video data for the region of interest, and not transmit encoded video data for other regions.

[0245] In another example, the video data is multi-view video data, and the extraction pattern can indicate a pattern of skipping transmission of encoded video data for pictures from specified views. For example, two views can have very similar content, such as views that primarily show a distant object. Thus, in this example, the sending device 102 can transmit encoded video data for one of the views and not transmit encoded video data for one or more other views, as indicated by the extraction pattern.

[0246] In another example, the extraction pattern can indicate a bit pattern to be omitted from syntax elements indicating transform coefficients. For example, the extraction pattern can indicate that a particular number of least significant bits are to be omitted from transform coefficients. In some examples, the extraction pattern can indicate that particular transform coefficients (e.g., high frequency transform coefficients) are to be omitted.

[0247] Figure 17 is a block diagram illustrating example components of a sending device and a receiving device that perform extraction of encoded video data in accordance with the techniques of this disclosure. In Figure 17In the example of FIG. 2, the sending device 102 includes a video encoder 210, a channel encoder 212, a puncturing unit 214, and a transmitter decimation unit 1700. The receiving device 104 includes a depuncturing unit 220, a channel decoder 222, a video decoder 224, a picture estimation unit 226, a video encoder 228, and a receiver decimation unit 1702. The video encoder 210, the channel encoder 212, the puncturing unit 214, the depuncturing unit 220, the channel decoder 222, the video decoder 224, the picture estimation unit 226, and the video encoder 228 can operate in the same manner as described elsewhere in this disclosure.

[0248] However, in Figure 17 In the example of FIG. 2, the transmitter decimation unit 1700 can apply a decimation pattern to the encoded video data after the channel encoder 212 generates error correction data for the encoded video data. The decimation pattern indicates a pattern of non-transmission of encoded video data. For example, the transmitter decimation unit 1700 can cause the sending device 102 to not send encoded video data for a particular picture, a region of a picture, a pattern of blocks within a picture, a particular view, etc. The receiver decimation unit 1702 can determine the decimation pattern indication based on the pictures reconstructed by the video decoder 224. The receiver decimation unit 1702 can send a decimation pattern indication to the sending device 102 indicating the decimation pattern. The transmitter decimation unit 1700 can apply the decimation pattern indicated by the decimation pattern indication.

[0249] Although the Figure 17 techniques of this disclosure related to communicating a decimation pattern indication from the receiving device 104 to the sending device 102 are not necessarily limited to this. For example, in some examples, the picture estimation unit 226 and the video encoder 228 can be omitted.

[0250] Figure 18 is a conceptual diagram illustrating an example information exchange including a decimation pattern indication according to the techniques of this disclosure. In Figure 18 In the example of FIG. 2, the sending device 102 can send encoded video data for a first set of pictures (e.g., pictures n-nl, pictures n-n n1+1 and pictures n) to the receiving device 104. The sending device 102 can also send error correction data for the first set of pictures.

[0251] The receiving device 104 can send and the sending device 102 can receive a decimation pattern indication indicating a decimation pattern determined based on the first set of encoded pictures. In Figure 18 In the example of FIG. 2, the decimation pattern decimates pictures according to a 1 :2 ratio. In other words, encoded video data for one picture out of every two pictures will be transmitted.

[0252] Accordingly, the transmitting device 102 can transmit encoded video data of the second set of pictures to the receiving device 104. Depending on the decimation pattern indicated by the received decimation pattern indication, the transmitting device 102 skips transmission of encoded video data for every other picture in the second set of pictures. As Figure 18 As shown in the example of FIG. 6B, the index values of the pictures in the second set of pictures (e.g., n+2, n+4, n+n2) increase by 2 instead of 1 as was the case for the first set of pictures.

[0253] Subsequently, the receiving device 104 can determine that a more appropriate decimation pattern would be a 1 : 1 decimation pattern (i.e., the decimation pattern in which the transmitting device 102 transmits encoded video data for every picture) based on the second set of pictures. Accordingly, in the example of FIG. 6B, the receiving device 104 can transmit a second decimation pattern indication indicating the second decimation pattern, and the transmitting device 102 can receive the second decimation pattern indication indicating the second decimation pattern. Subsequently, the transmitting device 102 can transmit encoded video data of a third set of pictures. Depending on the second decimation pattern, the transmitting device 102 does not skip transmission of encoded video data for any pictures in the third set of pictures. Accordingly, as shown in the example of FIG. 6C, the index values (e.g., n+n2+1, n+n2+2, etc.) increase by 1 instead of 2. Figure 18 As shown in the example of FIG. 6B, the index values of the pictures in the second set of pictures (e.g., n+2, n+4, n+n2) increase by 2 instead of 1 as was the case for the first set of pictures. Figure 18

[0254] Figure 19 is a flowchart illustrating example operations of a transmitting device 102 in accordance with the techniques of this disclosure, in which the transmitting device 102 receives a decimation pattern indication. In the example of FIG. 6A, the video encoder 210 of the transmitting device 102 can encode a first set of pictures of video data to generate first encoded video data (1900). The transmitting device 102 can transmit the first encoded video data to a receiving device 104 (1902). Figure 19

[0255] In addition, the transmitting device 102 can receive a decimation pattern indication from the receiving device 104 indicating a decimation pattern determined based on the first set of pictures (1904). The decimation pattern can be a pattern in which encoded video data is not transmitted. For example, in some instances, the decimation pattern indication is a pattern that skips transmission of encoded video data for a full picture. In other words, the transmitting device 102 can not transmit any encoded video data for a particular picture, and can transmit some or all of the encoded video data for other pictures. In some instances, the decimation pattern indication is a pattern that skips transmission of encoded video data for a particular region within a picture. For example, the decimation pattern can indicate that the transmitting device 102 will skip transmission of encoded video data associated with a particular picture block, e.g., as described in Figure 6 and 9 ​​As shown in the diagram. In some instances where the video data is multi-view video data, the decimation mode can indicate a mode for skipping the transmission of encoded video data from images of a particular view. In such examples, the view may be associated with a sensor on the same user device (e.g., the same XR headset), or with a view associated with a sensor on a different user device (e.g., different XR headsets worn by different users) or camera. In some instances, different decimation modes may exist for different regions that do not have images. For example, decimation may not be applied to the region of interest, and decimation modes of limiting blocks or transmission of lower significant bits or higher frequency transform coefficients may be applied relative to image regions outside the region of interest.

[0256] The video encoder 210 can encode a second set of images of video data to generate second encoded video data (1906). Additionally, the transmitter decimation unit 1700 can apply a decimation mode to the second encoded video data to generate decimated video data (1908). The transmitting device 102 can transmit the decimated video data to the receiving device (1910).

[0257] In some examples, the transmitter decimation unit 1700 can determine the decimation mode. Therefore, in Figure 19 In the context of [the above context], transmitting device 102 may encode a third set of images of video data to generate third encoded video data, determine a second extraction mode indicating that the encoded video data has not been transmitted, and apply the second extraction mode to the third encoded video data to generate second extracted video data. Transmitting device 102 may transmit the second extracted video data to receiving device 104. Transmitting device 102 may send a second extraction mode indication to receiving device. The second extraction mode indication is used to indicate that the second extraction mode is applied to the third encoded video data.

[0258] The transmitter decimation unit 1700 can determine decimation patterns in various ways. For example, the transmitter decimation unit 1700 can test various decimation patterns. When testing a decimation pattern, the transmitter decimation unit 1700 can apply the decimation pattern to an image and reconstruct the image from one or more previous original images of the image's error correction data and video data. The transmitter decimation unit 1700 can compare the reconstructed image with the original image to determine the distortion level. The transmitter decimation unit 1700 can compare the distortion levels associated with different decimation patterns to determine the decimation pattern.

[0259] In some examples, Figure 19The operations are performed within the DVC context. Therefore, the channel encoder 212 of the transmitting device 102 can generate first error correction data based on the first encoded video data. The transmitting device 102 can then transmit the first error correction data to the receiving device 104. The channel encoder 212 can generate second error correction data based on the second encoded video data. The transmitting device 102 can then transmit the second error correction data to the receiving device.

[0260] Figure 20 This is a flowchart illustrating an example operation of a receiving device 104 according to the technology of this disclosure, wherein the receiving device 104 transmits a decimation mode indication. Figure 20 In one example, receiving device 104 may receive first encoded video data (2000) from transmitting device 102. Video decoder 224 may perform decoding processing to reconstruct a first set of images based on the first error-corrected encoded video data (2002).

[0261] Additionally, the receiver extraction unit 1702 can determine an extraction mode (2004) based on the first image set, indicating a mode in which encoded video data has not been transmitted. In some instances, the extraction mode indicates a mode that skips the transmission of encoded video data for the entire image. In some instances, the extraction mode indicates a mode that skips the transmission of encoded video data for a specified area or block within an image, for example in... Figure 6 and Figure 9 In some instances, the video data is multi-view video data and the extraction mode indicates skipping the transmission of encoded video data from a specified view.

[0262] The receiver decimation unit 1702 can determine the decimation mode in one of a variety of ways. For example, the receiver decimation unit 1702 can apply one or more experimental decimation modes to error-corrected encoded video data generated by the channel decoder 222 for a first set of images to generate decimated encoded video data. The receiver decimation unit 1702 can then cause the channel decoder 222 to apply error correction processing to modify the decimated encoded video data based on first error correction data to generate experimental error-corrected video data. The receiver decimation unit 1702 can then cause the video decoder 224 to apply decoding processing to reconstruct the first set of images based on the experimental error-corrected video data. The receiver decimation unit 1702 can determine whether the decimation mode meets one or more criteria based on a comparison between the first set of images reconstructed based on the experimental error-corrected video data and the first set of images reconstructed based on the first error-corrected video data. For ease of explanation, the images reconstructed based on the experimental error-corrected video data may be referred to as "experimental images," and the images reconstructed based on the first error-corrected video data may be referred to as "baseline images." The receiver extraction unit 1702 can repeat this process with multiple trial extraction patterns until the receiver extraction unit 1702 identifies an extraction pattern that meets the criteria.

[0263] For example, receiver extraction unit 1702 can compare each test image with a corresponding baseline image to determine whether the test image meets a standard. For example, if the sum of the differences between the test image and the corresponding baseline image is less than a certain amount, receiver extraction unit 1702 can determine that the test image meets the standard. If at least a given number of test images exceed a threshold, receiver extraction unit 1702 can select an extraction mode associated with the test images.

[0264] In a more general example, receiver extraction unit 1702 can apply a function to the test image and the corresponding baseline image to generate a value. If the value is less than a threshold, receiver extraction unit 1702 can select an extraction mode associated with the test image.

[0265] Furthermore, in some instances, receiver decimation unit 1702 can cancel the use of a decimation mode (e.g., revert to a mode in which all encoded video data is transmitted) or switch to a less aggressive decimation mode upon the occurrence of specific conditions. For example, receiver decimation unit 1702 can cancel the use of a decimation mode in response to determining that a given number of baseline images fail to meet a criterion. For instance, if the sum of the differences between a test image and its corresponding baseline image is greater than a certain amount, receiver decimation unit 1702 can determine that the test image fails the criterion. If at least a given number of test images (failing the criterion) exceed a threshold, receiver decimation unit 1702 can cancel the use of the decimation mode or revert to a less aggressive decimation mode. In a more general example, receiver decimation unit 1702 can apply a function to the test image and its corresponding baseline image to generate a value. If this value is greater than a second threshold, receiver decimation unit 1702 can cancel the use of the decimation mode or revert to a less aggressive decimation mode.

[0266] The receiver decimation unit 1702 may send a decimation mode indication (2006) to the transmitting device 102, indicating the decimation mode determined therein. The receiver decimation unit 1702 may send the decimation mode indication using uplink control information (UCI) / media access control-control element (MAC-CE) messages, radio resource control (RRC) messages, sidelink control information (SCI) messages, or another type of message.

[0267] The receiving device 104 can receive extracted video data (2008) from the transmitting device 102. The extracted video data may include second encoded video data with an applied extraction mode. The second encoded video data is generated based on a second set of images of the video data.

[0268] Video decoder 224 can perform decoding processing to reconstruct a second set of pictures based on second error-corrected encoded video data (2010). Video decoder 224 can perform the same decoding processing as described elsewhere in this disclosure.

[0269] In some examples, receiving device 104 can receive and use a decimation mode indication from transmitting device 102. Therefore, in Figure 20In the example, receiving device 104 may receive a second decimation mode indication indicating that a second mode of encoded video data has not been transmitted. Receiving device 104 may receive third error correction data and second decimated video data from transmitting device 102. The second decimated video data may include third encoded video data to which the second decimation mode has been applied. Third encoded video data may be generated based on a third set of images of the video data. Channel decoder 222 may apply error correction processing to generate third error-corrected encoded video data based on the third encoded video data and the third error correction data. Video decoder 224 may apply decoding processing to reconstruct the third set of images based on the third error-corrected encoded video data.

[0270] In some examples, Figure 20 The processing can be performed in a DVC-based implementation. Therefore, the receiving device 104 can receive first error correction data from the transmitting device 102. The receiving device 104 applies error correction processing to modify the first encoded video data based on the first error correction data to generate first error-corrected encoded video data. The receiving device 104 can perform decoding processing to reconstruct a first set of images based on the first error-corrected encoded video data. Additionally, the receiving device 104 can receive second error correction data from the transmitting device 102. The receiving device 104 can apply error correction processing to generate second error-corrected encoded video data based on the second encoded video data, the predicted encoded video data generated by the video encoder 228, and the second error correction data. The receiving device 104 can perform decoding processing to reconstruct a second set of images based on the second error-corrected encoded video data.

[0271] During video encoding processing, a video encoder typically analyzes multiple encoding options and selects the optimal one. For example, a video encoder might analyze various ways to divide a maximum coding unit (LCU) or macroblock into coding units (CUs) and / or prediction units (PUs). In another example, a video encoder might analyze multiple intra-frame prediction modes to generate prediction blocks for a PU while performing intra-frame prediction. In yet another example, a video encoder might analyze multiple reference images and motion vectors to generate prediction blocks for a PU while performing inter-frame prediction. This analysis and selection can be resource-intensive. For example, to be efficient, a video encoder might need to process multiple options in parallel, which increases the hardware complexity and power requirements of the video encoder. Analysis and selection may also involve multiple requests to read data and write data to memory, further increasing power requirements.

[0272] According to one or more techniques of this disclosure, a large portion of the processing of analyzing and selecting encoding operations is transferred from a transmitting device (e.g., transmitting device 102) to a receiving device (e.g., receiving device 104). For example, the transmitting device may encode a first image of video data to generate first encoded video data. The transmitting device may transmit the first encoded video data to the receiving device. The receiving device may receive the first encoded video data from the transmitting device and reconstruct the first image based on the first encoded video data. Additionally, the receiving device may estimate a second image of the video data based on the first image. The second image may be an image that appears after the first image in decoding order. The receiving device may generate encoding selection data for the estimated second image. The encoding selection data indicates encoding selections for encoding the estimated second image. The receiving device may transmit the encoding selection data for the second image. The transmitting device may receive the encoding selection data for the second image of the video data. The transmitting device may encode the second image based on the encoding selection data to generate second encoded video data. The transmitting device may transmit the second encoded video data to the receiving device. The receiving device may receive the second encoded video data from the transmitting device. The receiving device may reconstruct the second image based on the second encoded video data. In this way, because the transmitting device receives the encoding selection data from the receiving device, the transmitting device does not need to perform resource-intensive analysis and selection processing when encoding the second image, since the analysis and selection processing of the second image has already been performed at the receiving device. This reduces the resource requirements of the transmitting device.

[0273] The transmitting and receiving devices can communicate over low-range, low-power links on ultra-wideband (e.g., high-bandwidth) communication links. In some examples, the transmitting and receiving devices may communicate using time-division duplex (TDD), subband non-overlapping full-duplex (SBFD), or SFFD schemes. The low latency associated with this type of communication allows the transmitting device to receive coded selection data quickly enough to continue transmitting coded video data at a predetermined picture rate.

[0274] Figure 21 This is a block diagram illustrating example components of a delivery device 102 and a receiving device 104 that transmits encoded selection data to the delivery device according to the technology of this disclosure. Figure 21 In the example, the transmitting device 102 may include a video encoder 210, a channel encoder 212, and a punching unit 214. The receiving device 104 may include a de-punching unit 220, a channel decoder 222, a video decoder 224, an image estimation unit 226, and a video encoder 228.

[0275] exist Figure 21In the examples provided above, video encoder 210 can encode images of video data. Unlike some of the examples given above, video encoder 210 can perform full video coding processing that may include intra-frame prediction and inter-frame prediction. In some examples, video encoder 210 may use video codecs (e.g., H.264 / AVC, H.265 / HEVC, H.266 / VVC, Basic Video Coding (EVC), AV1, etc.) to encode video data. Channel encoder 212, puncturing unit 214, de-puncturing unit 220, and channel decoder 222 may operate in the same manner as described elsewhere in this disclosure.

[0276] In addition, Figure 21 In the example, the video decoder 224 of the receiving device 104 can perform video decoding processing on the error-corrected encoded video data generated by the channel decoder 222. The video decoder 224 can perform full video decoding processing including intra-frame prediction and inter-frame prediction. The video decoder 224 can use the same video codec as the video encoder 210.

[0277] After the video decoder 224 reconstructs at least a portion of the images from the video data, the image estimation unit 226 can estimate the corresponding portion of subsequent images following the reconstructed images. The image estimation unit 226 can estimate subsequent images in the same manner as described elsewhere in this disclosure. Furthermore, the video encoder 228 can apply video encoding processing to subsequent images. In the example where both the video encoder 210 and the video decoder 224 use a video codec, the video encoder 228 can use the same codec.

[0278] However, according to one or more techniques of the present invention, the receiving device 104 may transmit encoding selection data 2100 to the transmitting device 102. Encoding selection data 2100 indicates encoding selection for encoding estimated subsequent images. For example, the encoding selection data may include motion parameters of blocks in the estimated subsequent images. The block motion parameters may include motion vectors, reference image indicators, merge candidate indices, affine motion parameters, and other data used to determine predicted blocks in one or more reference images. Thus, in this example, the video encoder 228 may encode the blocks using inter-frame prediction, and the encoding selection data 2100 may include motion parameters instructing the video encoder 228 how to encode the blocks using inter-frame prediction.

[0279] In some examples, the encoding selection data may include intra-prediction parameters for blocks in the estimated subsequent images. The intra-prediction parameters may include data indicating the intra-prediction mode (e.g., planar mode, DC mode, squared prediction mode, etc.) used by the video encoder 228 for intra-prediction of the blocks. In some examples, the encoding selection data may include other information, such as information describing how the video encoder 228 divides the estimated subsequent images into blocks, whether residual prediction is used, whether and how intra-block copying (IBC) is used, and whether specific filters are used.

[0280] Video encoder 210 can use encoding selection data 2100 when encoding actual (non-estimated) subsequent images. That is, instead of searching for different possibilities during video encoding processing, video encoder 210 can use the video encoding selection indicated by encoding selection data 2100. For example, encoding selection data 2100 can indicate that a specific block of a subsequent image is encoded using a specific intra-prediction mode. Therefore, in this example, when encoding a subsequent image, video encoder 210 can utilize the specific intra-prediction mode to encode the specific block without analyzing different potential intra-prediction modes to select the specific intra-prediction mode. In another example, encoding selection data 2100 can indicate the motion vectors and reference image of a specific block of a subsequent image. Therefore, in this example, when encoding a subsequent image, video encoder 210 can use the motion vectors to determine the predicted block in the reference image without analyzing the potential reference image and motion vectors. Video encoder 210 can then use the predicted block to encode the specific block.

[0281] The transmitting device 102 can process the encoded video data of subsequent images in the same manner as other encoded video data. Similarly, the receiving device 104 can process the encoded video data of subsequent images in the same manner as other encoded video data. Therefore, after the video decoder 224 reconstructs at least a portion of the subsequent image, the image estimation unit 226 can predict the corresponding portion of the image following the subsequent image, the video encoder 228 can encode the video data of the estimated subsequent image, and send encoding selection data for the estimated subsequent image; this cycle can be repeated. In this way, some of the burden of encoding video data can be transferred from the video encoder 210 of the transmitting device 102 to the video encoder 228 of the receiving device 104. This can reduce the resource requirements of the transmitting device 102.

[0282] about Figure 21The described processing can be adapted for use with DVC technology. For example, transmitting device 102 may apply a decimation mode to the encoded video data (e.g., according to any of the examples provided elsewhere in this disclosure), such that transmitting device 102 transmits only some of the encoded video data, but still transmits error-corrected data of the decimated video data. The channel decoder 222 of receiving device 104 can receive error-corrected data for a specific image from the transmitting device (e.g., via the de-punching unit 220). In this example, channel decoder 222 may apply error-correction processing to generate error-corrected encoded video data based on the error-corrected data for the specific image and the encoded video data for the specific image generated by the video encoder 228 of receiving device 104. Video decoder 224 may decode the error-corrected encoded video data to reconstruct specific features.

[0283] In some cases, transmitting device 102 needs to transmit encoded video data according to a schedule. For example, transmitting device 102 may need to send encoded video data to receiving device 104 at a predetermined pictures per minute rate to support a specific application. Therefore, it is possible that transmitting device 102 may not receive the image encoding selection data in time for use in encoding and transmitting the encoded video data of the image. Therefore, in some examples, based on the determination that the image encoding selection data has not been received from receiving device 104 before the time limit expires, video encoder 210 may encode the image without using the image encoding selection data. Video encoder 210 may use limited video encoding processing to encode the image. Furthermore, in some examples, the encoded video data of the image may contain encoding selection data generated by video encoder 210. In some examples, the encoded video data of the image may contain data indicating that the encoded video data of the image is not generated based on the encoding selection data generated by receiving device 104. The time limit may be subject to or defined based on the capabilities of transmitting device 102.

[0284] Encoding selection data and time constraints can be defined on a per-image-fragment basis (e.g., slices, regions, etc.). Therefore, in this disclosure, discussions of encoding selection data for images, encoded video data, or other types of data may apply only with respect to individual segments of the image.

[0285] Figure 22 This is a communication diagram illustrating an example data exchange between a transmitting device 102 and a receiving device 104 according to the technology of this disclosure, the data exchange including the transmission and reception of encoded selection data. Figure 22In the example, transmitting device 102 sends encoded video data of image n-1 to receiving device 104. Receiving device 104 can reconstruct image n-1 based on the encoded video data of image n-1. Additionally, receiving device 104 can estimate and encode image n based on image n-1. Receiving device 104 can send encoded selection data of image n to transmitting device 102. Transmitting device 102 can encode image n based on the encoded selection data of image n and send the resulting encoded video data of image n to receiving device 104. This process can be repeated multiple times. Therefore, in Figure 22 In one example, receiving device 104 may reconstruct image n based on encoded video data of image n, estimate image n+1 based on image n and / or one or more other previously reconstructed images, encode image n+1, and send the encoding selection data of image n+1 to transmitting device 102.

[0286] Figure 23 This is a flowchart illustrating an example operation of a transmitting device 102 according to the technology of this disclosure, wherein the transmitting device 102 receives encoded selection data. Figure 23 In the example, video encoder 210 encodes a first image of video data to generate first encoded video data (2300). In some examples, if transmitting device 102 has not yet received encoding selection data for the first image, transmitting device 102 may perform limited video coding processing on the first image. Limited video coding processing can use relatively less computationally intensive coding tools compared to full video coding processing. For example, limited video coding processing can use intra-frame prediction instead of inter-frame prediction.

[0287] The transmitting device 102 may transmit the first encoded video data to the receiving device 104 (2302). In some examples, the transmitting device 102 may apply channel coding processing to the first encoded video data to generate error correction data for the first encoded video data. The transmitting device 102 may transmit the first encoded video data and the error correction data to the receiving device 104.

[0288] Subsequently, transmitting device 102 may receive encoding selection data (2304) for a second image of video data from receiving device 104. The encoding selection data may indicate encoding selections for encoding an estimate of the second image. The second image is decoded after the first image. In some examples, the second image may appear in the output encoder before or after the first image. In some examples, the encoding selection data is entropy-coded. Therefore, in such examples, transmitting device 102 may entropy-decode the encoding selection data. For example, transmitting device 102 may apply CABAC decoding, Golomb-Rice decoding, or another type of entropy decoding to the encoding selection data. In some examples, the encoding selection data is channel-coded. Therefore, transmitting device 102 may apply error correction operations to the encoding selection data based on error correction data for the encoding selection data.

[0289] The video encoder 210 of the transmitting device 102 can encode the second picture based on encoding selection data to generate second encoded video data (2306). For example, the encoding selection data may include data indicating how to divide a particular macroblock into CUs. In this example, the video encoder 210 may divide the macroblock into CUs in the manner indicated by the encoding selection data. In another example, the encoding selection data may indicate an intra-prediction mode for the block (e.g., CU or PU), and the video encoder 210 may use the indicated intra-prediction mode to encode the block. Therefore, in this example, the encoding selection data received from the receiving device 104 may contain intra-prediction parameters for the block of the second picture, and as part of encoding the second picture, the video encoder 210 may perform intra-prediction based on the intra-prediction parameters for the block of the second picture to generate prediction blocks. The second encoded video data may contain encoded video data based on the prediction blocks.

[0290] In another example, the encoded selection data received from receiving device 104 includes motion parameters of blocks of the second image, and as part of encoding the second image, transmitting device 102 can perform motion compensation based on the motion parameters of the blocks of the second image to generate predicted blocks. The second encoded video data includes encoded video data based on the predicted blocks.

[0291] The transmitting device 102 can transmit the second encoded video data to the receiving device (2308). In some examples, the second encoded video data does not contain encoding selection data, which indicates the encoding selection used by the transmitting device 102 to encode the second image or by the receiving device 104 to encode an estimate of the second image. The second encoded video data may not need to include encoding selection data because the receiving device 104 generates and therefore already has encoding selection data.

[0292] In some examples,Figure 23 The operation can be used with DVC technology. For example, transmitting device 102 can receive encoding selection data for a third picture for video data. The encoding selection data for the third picture can indicate encoding selections for encoding an estimate of the third picture. Video encoder 210 can encode the third picture based on the encoding selection data for the third picture to generate third encoded video data. Channel encoder 212 can apply channel coding processing to generate error correction data for the third encoded video data. Transmitting device 102 can transmit error correction data for the third encoded video data to the receiving device without transmitting at least a portion of the third encoded video data.

[0293] Figure 24 This is a flowchart illustrating an example operation of the receiving device 104, wherein the receiving device 104 transmits encoded selection data according to the techniques of this disclosure. Figure 24 In the example, receiving device 104 may receive first encoded video data (2400) from transmitting device.

[0294] The video decoder 224 of the receiving device 104 can reconstruct a first image (2402) of the video data based on the first encoded video data. The image estimation unit 226 of the receiving device 104 can estimate a second image (2404) of the video data based on the first image. The second image may be an image that appears after the first image in the decoding order.

[0295] The video encoder 228 of the receiving device 104 can generate encoding selection data (2406) for estimating the second image. The encoding selection data indicates encoding selections for encoding the estimated second image. For example, as part of encoding the estimated second image, the video encoder 228 can perform motion compensation based on motion parameters of blocks of the second image to generate prediction blocks. In this example, the encoding selection data may include motion parameters of blocks of the second image from one or more processors. In some examples, as part of encoding the second image, the video encoder 228 can perform intra-frame prediction based on intra-frame prediction parameters for blocks of the second image to generate prediction blocks. In this example, the encoding selection data may include intra-frame prediction parameters for blocks of the second image.

[0296] The receiving device 104 can transmit encoded selection data for a second picture to the transmitting device 102 (2408). In some examples, the receiving device 104 can apply entropy coding (e.g., CABAC coding, Golomb-Rice coding, etc.) to the encoded selection data for the second picture before transmitting it. In some examples, the receiving device 104 can perform channel coding processing on the encoded selection data to generate error correction data for the encoded selection data. The receiving device 104 can transmit the encoded selection data and the error correction data for the encoded selection data to the transmitting device 102. In some examples, compared to other data transmissions in the data link (e.g., wireless sidelink channel 112, wireless uplink / downlink channel, etc.) between the receiving device 104 and the transmitting device 102, the communication interface 134 of the receiving device 104 ( Figure 1 The coded selection data can be modulated at a lower modulation order. This increases the likelihood that the transmitting device 102 will correctly receive the coded selection data.

[0297] Subsequently, the receiving device 104 can receive second encoded video data (2410) from the transmitting device. The video decoder 224 can reconstruct a second picture (2412) based on the second encoded video data. In some examples, the second encoded video data does not contain encoding selection data. The video decoder 224 can apply decoding processing, which includes using the encoding selection data to reconstruct the second picture based on the second encoded video data.

[0298] Figure 24 The processing can be used in conjunction with DVC technology. For example, image estimation unit 226 can estimate a third image of video data based on one or more of the first or second images. Video encoder 228 can encode the estimated third image to generate third encoded video data. Receiving device 104 can transmit third encoding selection data to transmitting device 102. The third encoding selection data can indicate encoding selections for encoding the estimated third image. Subsequently, receiving device 104 can receive error correction data for the third image. Channel decoder 222 can apply error correction processing to generate error-corrected encoded video data for the third image based on the error correction data for the third image and the third encoded video data. Video decoder 224 can apply decoding processing to reconstruct the third image based on the error-corrected encoded video data for the third image. In some examples where transmitting device 102 and receiving device 104 use DVC technology, the error-corrected video data for the third image does not contain the third encoding selection data. However, the video decoder 224 can use the third encoding selection data generated by the video encoder 228 of the receiving device 104 to apply decoding processing to reconstruct the third picture based on the error-corrected encoded video data of the third picture.

[0299] Figure 25 This is a conceptual diagram illustrating an example hierarchy of encoded video data according to the technology of this disclosure. More specifically, Figure 25 This describes the layers of the encoded video data generated using the H.264 / AVC video encoding standard. For example... Figure 25 As shown in the example, the Network Abstraction Layer (NAL) is the highest level of the hierarchy. At the NAL, data is organized into NAL units. In some examples, NAL units are assigned to different packets or coding blocks for transmission. NAL units in the NAL can contain Sequence Parameter Sets (SPS) and Picture Parameter Sets (PPS) with high-level syntax. NAL units in the NAL can also contain Video Coding Layer (VCL) NAL units. VCL NAL units can contain slice NAL units containing slice-level data. A slice can be a series of macroblocks within a picture. Slices can include Instantaneous Decoder Refresh (IDR) slices and regular slices. Decoding an IDR slice does not depend on any other slice. Regular slices may have dependencies on other slices.

[0300] Each slice NAL unit can contain a slice header and slice data. The slice header of the slice NAL unit contains information for decoding the slice data of the slice NAL unit. The slice data of the slice NAL unit contains a series of macroblocks (MBs). Skip indicators can be scattered between the MBs. Each MB contains encoded video data of a specific block of the slice. Furthermore, as... Figure 25 As shown, a Block Markup (MB) can contain a type indicator, prediction information, coded block mode, quantization parameters (QP), and coded residual data. If intra-frame prediction is used to code the MB, the prediction data can indicate one or more intra-frame modes used to code the MB. If inter-frame prediction is used to code the MB, the prediction data can indicate one or more reference images and one or more motion vectors. The coded residual data used for the MB can include coded residual data for luminance blocks within the MB, coded residual data for Cb blocks within the MB, and coded residual data for Cr blocks within the MB. Generally, the coded residual data constitutes the largest portion of the coded video data.

[0301] According to the technology of this disclosure, everything in the hierarchy at the macroblock layer except for the encoded residual data can be encoding selection data. Therefore, in some examples, receiving device 104 may send type data, prediction data, encoded block pattern, and QP to transmitting device 102 for each MB of the estimated image. Furthermore, in some examples, transmitting device 102 may send only the encoded residual data of the MB to receiving device 104, without sending the type data, prediction data, encoded block pattern, or QP of the MB. In some examples, the encoding selection data sent by receiving device 104 may include slice header data, SPS data, and PPS data. Transmitting device 102 and receiving device 104 may exchange information, or may be pre-configured with information indicating encoder and decoder capabilities. In some examples, transmitting device 102 may transmit data other than the encoded residual data for some images, some MBs, or some slices. Transmitting device 102 may signal information (e.g., bits) at the network abstraction layer (e.g., as a picture-level control field), which may indicate whether a DVC-based approach is used or whether conventional compression should be used for a particular image.

[0302] Figure 26 This is a block diagram illustrating alternative example components of the transmitting device 102 according to one or more technologies of this disclosure. Figure 26 In the example, transmitting device 102 performs digital encoding and analog encoding on the video data. Transmitting device 102 transmits the digitally encoded video data and the analog-encoded video data to receiving device 104 via channel 230.

[0303] exist Figure 26 The example includes a video encoder 2600, a residual generation unit 2602, an analog encoder 2604, a reliability sorting unit 2606, an interleaving unit 2608, a channel encoder 2610, and a punching unit 2612. The video encoder 2600 can acquire video data and... Figure 2 The video encoder 210 operates in roughly the same way. For example... Figure 26 As shown in example A, the video encoder 2600 can receive values ​​of encoding parameters (e.g., encoding selection parameters) transmitted by the receiving device 104. In some examples, the video encoder 2600 can transmit the values ​​of encoding parameters and / or encoding selection data to the receiving device 104. In this way, the video encoder 2600 of the transmitting device 102, the video encoder of the receiving device 104, and the video decoder of the receiving device 104 can operate based on the same values ​​of the encoding parameters.

[0304] The video encoder 2600 can also output predicted data to the residual generation unit 2602. Furthermore, the video encoder 2600 can apply a higher level of quantization than the video encoder 210. The residual generation unit 2602 can generate residual data based on the predicted data and the video data.

[0305] The analog encoder 2604 can perform analog encoding operations on residual data. Examples of details of the analog encoding operations can be found in the following patents: U.S. Patent 11,553,184, filed December 29, 2020, entitled "Hybrid Digital-Analog Modulation for Transmission of Video Data"; U.S. Patent 11,431,962, filed December 29, 2020, entitled "Analog Modulated Video Transmission with Variable Symbol Rate"; and U.S. Patent 11,457,224, filed December 29, 2020, entitled "Interlaced Coefficients in Hybrid Digital-Analog Modulation for Transmission of Video Data".

[0306] For example, in some examples, the analog encoder 2604 may generate coefficients based on residual data. For instance, the analog encoder 2604 may binarynize the residual data to generate coefficients. The analog encoder 2604 may quantize the coefficients. In other examples of generating coefficients based on video data, the analog encoder 2604 may perform more, fewer, or different steps. For example, in some examples, the analog encoder 2604 may not perform a quantization step. In other examples, the analog encoder 2604 may not perform a step of binarynizing the residual data.

[0307] Furthermore, the analog encoder 2604 can generate a coefficient vector. Each element in the coefficient vector contains n coefficients. The analog encoder 2604 can generate the coefficient vector in one of several ways. For example, in one example, the analog encoder 2604 can generate the coefficient vector as a set of n consecutive coefficients according to the coefficient encoding order. Various coefficient encoding orders can be used, such as raster scan order, zigzag scan order, reverse raster scan order, vertical scan order, etc. In some examples, the coefficient vector may contain one or more negative coefficients and one or more positive coefficients (i.e., signed coefficients). In some examples, the coefficient vector contains only non-negative coefficients (i.e., unsigned coefficients).

[0308] For each of the coefficient vectors, the analog encoder 2604 can determine the amplitude value of the coefficient vector based on a mapping pattern. For each corresponding allowed coefficient vector among a plurality of allowed coefficient vectors, the mapping pattern maps the corresponding allowed coefficient vector to a corresponding amplitude value among a plurality of amplitude values. The corresponding amplitude value is adjacent to at least one other amplitude value among the plurality of amplitude values ​​in n-dimensional space, and the at least one other amplitude value is adjacent to the corresponding amplitude value in the monotonic digital line of the amplitude value.

[0309] In some examples, to determine the amplitude value of the coefficient vector, the analog encoder 2604 can determine a position in n-dimensional space. The coordinates of the position in the n-dimensional space are based on the coefficients of the coefficient vector, and the mapping pattern maps different positions in the n-dimensional space to different amplitude values ​​among a plurality of amplitude values. The analog encoder 2604 can determine the amplitude value of the coefficient vector as the amplitude value corresponding to the determined position in the n-dimensional space.

[0310] Analog encoder 2604 can modulate an analog signal based on the amplitude values ​​of a coefficient vector. For example, analog encoder 2604 can determine an analog symbol based on a pair of amplitude values. The analog symbol may correspond to the phase shift and power of a point in the IQ plane with coordinates indicated by the amplitude value pair. Analog encoder 2604 can modulate the analog signal during the symbol sampling time based on the determined phase shift and power. The modem (e.g., communication interface 118) of transmitting device 102 can be configured to output an analog signal.

[0311] In addition, Figure 26 In example A, the reliability sequencing unit 2606 can obtain encoded video data generated by the video encoder 2600. The reliability sequencing unit 2606 can obtain reliability side information from the video encoder 2600. In some examples, the reliability sequencing unit 2606 can receive values ​​of channel and compression state feedback (CCSF) parameters. The values ​​of the CCSF parameters can provide information about the condition of channel 230 (e.g., signal-to-noise ratio, latency, network bandwidth congestion, etc.). In some examples, the values ​​of the CCSF parameters provide information related to predicted reliability and quality. For example, the values ​​of the CCSF parameters can include a decimation mode indicator. In some examples, the values ​​of the CCSF parameters can enable the channel encoder 2610 to determine the decimation mode.

[0312] Interleaving unit 2608 can perform interleaving processing that ensures reliable and unreliable bits are evenly distributed across code blocks. For example, encoded video data can be divided into code blocks. Channel encoder 2610 can generate a separate error correction dataset for each code block. Before channel encoder 2610 generates error correction data, interleaving unit 2608 can interleave encoded video data between code blocks according to a predefined interleaving pattern. For example, encoded video data representing different neighboring pixels can be interleaved into different code blocks. After channel decoding processing is applied, deinterleaving processing performed at receiving device 104 is the reverse of interleaving processing. Therefore, if a code block is corrupted during transmission, pixels decoded from the corrupted code block can be spatially scattered within the image among pixels decoded from uncorrupted code blocks.

[0313] The channel encoder 2610 of the transmitting device 102 can perform channel coding processing on the video data obtained from the interleaving unit 2608. The channel encoder 2610 can, according to the information provided by the channel encoder 212 (…), Figure 2 Channel coding processing can be performed using any example provided by the channel encoder 2610. The puncturing unit 2612 can perform bit puncturing operations on the error correction data generated by the channel encoder 2610. The puncturing unit 2612 can perform bit puncturing operations based on the information provided by the puncturing unit 214 (…). Figure 2 Any example provided performs bit punching on the error correction data. Transmitting device 102 may transmit encoded video data and error correction data (e.g., bit-punched error correction data) to receiving device 104 via channel 230.

[0314] Figure 27 This is a block diagram illustrating example alternative components of a receiving device 104 according to one or more technologies of this disclosure. Figure 27 The version of the receiver 104 shown can be used with Figure 26 The version of the transmitting device 102 shown is compatible. Figure 27 In the example, the receiving device 104 includes an analog decoder 2700, a de-puncturing unit 2702, a channel decoder 2704, a deinterleaving unit 2706, a video decoder 2708, a reconstruction unit 2710, an image estimation unit 2712, a video encoder 2714, a reliability unit 2716, and a feedback unit 2718.

[0315] The analog decoder 2700 can acquire analog encoded video data. The analog decoder 2700 can perform analog decoding operations to reconstruct residual data. Example details of the analog decoding operations can be found in U.S. Patents 11,553,184, 11,431,962, and 11,457,224.

[0316] For example, in some examples, the analog decoder 2700 can determine the amplitude values ​​of multiple coefficient vectors based on an analog signal. For instance, the analog decoder 2700 can determine the phase shift and power at the symbol sampling time of the analog signal. The analog decoder 2700 can then determine a point in the IQ plane indicated by the determined phase shift and power. The analog decoder 2700 can then determine the amplitude value pairs as coordinates of the point in the IQ plane.

[0317] For each of the coefficient vectors, the analog decoder 2700 can determine the coefficients in the coefficient vector based on the amplitude values ​​and mapping patterns of the coefficient vector. For each corresponding allowed coefficient vector among a plurality of allowed coefficient vectors, the mapping pattern maps the corresponding allowed coefficient vector to a corresponding amplitude value among a plurality of amplitude values. The corresponding amplitude value is adjacent to at least one other amplitude value among the plurality of amplitude values ​​in n-dimensional space, and the at least one other amplitude value is adjacent to the corresponding amplitude value in the monotonic digital line of the amplitude value. Each of the coefficient vectors may contain n coefficients. The value n may be greater than or equal to 2. In some examples, the analog decoder 2700 can determine the coefficients in the coefficient vector as coordinates of positions in n-dimensional space corresponding to amplitude values. The mapping pattern maps different positions in n-dimensional space to different amplitude values ​​among the plurality of amplitude values. In some instances, the coefficient vector contains one or more negative coefficients and one or more positive coefficients. In other examples, the coefficient vector may contain only non-negative coefficients.

[0318] In some examples, as part of determining the coefficients, the analog decoder 2700 may obtain sign values, where the sign value indicates the positive / negative sign of the coefficients in the coefficient vector. In such examples, the analog decoder 2700 may determine the absolute values ​​of the coefficients in the coefficient vector based on the magnitude values ​​of the coefficient vector and the mapping pattern. The analog decoder 2700 may reconstruct the coefficients in the coefficient vector at least partially by applying the sign value to the absolute values ​​of the coefficients in the coefficient vector. In some examples, as part of determining the coefficients, the analog decoder 2700 may obtain data representing shift values. In such examples, the shift value indicates the most negative coefficient in the coefficient vector. Additionally, in such examples, the analog decoder 2700 may determine the intermediate values ​​of the coefficients in the coefficient vector based on the magnitude values ​​of the coefficient vector and the mapping pattern. The analog decoder 2700 may reconstruct the coefficients in the coefficient vector at least partially by adding shift values ​​to each of the intermediate values ​​of the coefficients in the coefficient vector.

[0319] Furthermore, the analog decoder 2700 can generate residual data based on the coefficients in the coefficient vector. For example, in one instance, the analog decoder 2700 can dequantize the coefficients of the coefficient vector. In this example, the analog decoder 2700 can perform debinarization to convert the coefficients into digital sample values. For instance, the analog decoder 2700 can apply an inverse DCT to the coefficients to convert them into digital sample values. In this way, the analog decoder 2700 can generate digital residual sample values.

[0320] The de-puncturing unit 2702 can obtain encoded video data and bit-punctured error correction data. The de-puncturing unit 2702 can apply de-puncturing processing to the bit-punctured error correction data to reconstruct the error correction data. The de-puncturing unit 2702 can, according to other parts of this disclosure... Figure 2 Any example provided by the de-drilling unit 220 for applying de-drilling processing.

[0321] Channel decoder 2704 can perform channel decoding processing that modifies encoded video data (e.g., encoded video data received via channel 230 or encoded video data generated by video encoder 2714 and modified by reliability unit 714 in some examples) based on error correction data. Channel decoder 2704 may be configured according to other parts of this disclosure regarding... Figure 2 Any example provided by the channel decoder 222 to perform channel decoding processing.

[0322] Deinterleaving unit 2706 can perform deinterleaving operations on the error-corrected encoded video data generated by channel decoder 2704. For example, deinterleaving processing can reverse the interleaving processing performed by interleaving unit 2608 of transmitting device 102. For example, deinterleaving processing can be performed according to the interleaving mode used by interleaving unit 2608.

[0323] Video decoder 2708 can obtain encoded video data (e.g., deinterleaved encoded video data generated by deinterleaving unit 2706). Video decoder 2708 can perform video decoding processing on the encoded video data to reconstruct a picture of the video data. The video decoding processing performed by video decoder 2708 can be the same as the video decoding processing described elsewhere in this disclosure with respect to video decoder 224. Reconstruction unit 2710 of receiving device 104 can add residual data generated by analog decoder 2700 to corresponding samples of the reconstructed video data generated by video decoder 2708, thereby completely reconstructing a picture of the video data.

[0324] In addition, Figure 27In the example, image estimation unit 2712 may estimate one or more images based on a previously reconstructed image. As previously mentioned in this disclosure, the discussion of an image may be applied relative to a segment (e.g., a slice) of an image. Image estimation unit 2712 may estimate an image according to any of the examples provided elsewhere in this disclosure concerning image estimation unit 226. Video encoder 2714 may perform video encoding processing on the estimated images. As part of performing video encoding processing, video encoder 2714 may determine encoding selection data, such as encoding selection data 2100, as previously described. Receiving device 104 may transmit encoding selection data to transmitting device 102. In some examples, video encoder 2714 may transmit to transmitting device 102 data such as information about... Figure 4 and Figure 5 For the encoding parameters discussed, the video encoder 2600 of the transmitting device 102 can perform limited encoding processing in the same manner as the video encoder 2714 of the receiving device 104. In some examples, the video encoder 2714 can determine a multi-view encoding cue and transmit it to the transmitting device 102.

[0325] Reliability unit 2716 can be used in conjunction with reliability unit 1002 ( Figure 10 It operates in a similar manner. The feedback unit 2718 can transmit predicted quality feedback (e.g., CCSF parameters) to the transmitting device 102 based on the output of the reliability unit 1002.

[0326] In some examples of this disclosure, the video encoder 2600 of the transmitting device 102 generates predicted data for a set of images, and the residual generation unit 2602 can generate residual data based on the first predicted data and the first set of images. The video encoder 2600 can apply a transform to the predicted data to generate a transform block, quantize the transform coefficients of the transform block, and apply entropy coding to the syntax elements representing the quantized transform coefficients to generate entropy-coded syntax elements. The first encoded video data may contain entropy-coded syntax elements. The channel encoder 2610 can perform channel coding processing to generate error correction data for the encoded video data, including entropy-coded syntax elements. The analog encoder 2604 can perform analog modulation on the residual data to generate analog modulated residual data. The communication interface of the transmitting device 102 can transmit the analog modulated residual data, the error correction data, and the encoded video data.

[0327] The communication interface of the receiving device 104 can receive analog modulation residual data and error correction data and decimated video data from the transmitting device. The decimated video data may contain encoded video data with the decimation mode applied. Encoded video data is generated from a set of images based on the video data. The channel decoder 2704 can apply error correction processing to generate error-corrected encoded video data based on the encoded video data and the error correction data. The error-corrected encoded video data contains entropy-encoded syntax elements representing quantized transform coefficients. The video decoder 2708 can perform decoding processing to reconstruct a second set of images based on the error-corrected encoded video data. As part of applying decoding processing to reconstruct the image set, the video decoder 2708 can apply entropy decoding to the syntax elements to obtain quantized transform coefficients, inverse quantize the quantized transform coefficients to generate inverse quantized transform coefficients, and apply an inverse transform to the inverse quantized transform coefficients to generate prediction data. The analog decoder 2700 can demodulate the analog modulation residual data to obtain residual data. The reconstruction unit 2710 can reconstruct the image set based on the prediction data and the residual data.

[0328] The following is a non-limiting list of one or more technologies under the terms of this disclosure.

[0329] Clause 1A. A method for decoding video data, the method comprising: obtaining error correction data at a receiving device and from a transmitting device, wherein the error correction data provides error correction information and is generated based on encoded video data of one or more blocks of a picture of the video data; generating prediction data of the picture at the receiving device using one or more encoding tools not used to generate the one or more blocks of the encoded video data, wherein the prediction data of the picture includes predictions of the blocks of the picture based at least in part on one or more previously reconstructed blocks of the video data; generating encoded video data at the receiving device based on the prediction data of the picture; generating error-corrected encoded video data at the receiving device using the error correction data to perform an error correction operation on the encoded video data; and performing a reconstruction operation at the receiving device to reconstruct the blocks of the picture based on the error-corrected encoded video data, wherein the reconstruction operation is controlled by values ​​of one or more parameters.

[0330] Clause 2A. The method according to Clause 1A further includes receiving the value of the parameter from the transmitting device at the receiving device.

[0331] Clause 3A. The method according to Clause 1A further includes determining the value of the parameter at the receiving device without receiving the value of the parameter from the transmitting device.

[0332] Clause 4A. The method according to any one of Clauses 1A-3A, wherein: the parameters include one or more quantization parameters, generating coded video data includes using the quantization parameters to quantize transform coefficients generated from image-based prediction data, and performing the reconstruction operation includes using the quantization parameters to inverse quantize the transform coefficients of the error-corrected coded video data.

[0333] Clause 5A. The method according to Clause 4A, wherein the method further comprises: calculating the quantization parameter based on the entropy ratio of the quantized transform coefficients to the unquantized transform coefficients.

[0334] Clause 6A. The method according to any one of Clauses 1A-5A, wherein: the parameters include a transform size parameter, generating the encoded video data includes applying a forward transform with a transform size indicated by the transform size parameter to the sampled domain data of the picture, and performing the reconstruction operation includes applying an inverse transform with a transform size indicated by the transform size parameter to the transform coefficients of the error-corrected encoded video data.

[0335] Clause 7A. The method according to any one of Clauses 1A-6A, wherein: the parameters include parameters indicating the number of transform coefficients; generating the encoded video data includes including a set of transform coefficients in the encoded video data, the set of transform coefficients containing the indicated number of transform coefficients; and performing the reconstruction operation includes parsing the set of transform coefficients from the error-corrected encoded video data, the set of transform coefficients containing the indicated number of transform coefficients.

[0336] Clause 8A. Obtaining the encoded video data and the error correction data includes receiving the encoded video data and the error correction data from the transmitting device via a communication channel at the receiving device, and the method further includes applying an optimization process to determine the number of transform coefficients based on the signal-to-noise ratio of the data transmitted on the communication channel.

[0337] Clause 9A. A method according to any one of Clauses 1A-8A, wherein: the parameters include a bit width parameter for a plurality of index values, and for each corresponding index value among the plurality of index values: performing the reconstruction operation includes parsing a first set of bits from the error-corrected encoded video data, wherein the first set of bits indicates transform coefficients having the corresponding index value, and the number of bits in the first set of bits is equal to the bit width indicated by the bit width parameter for the corresponding index value. Generating the encoded video data includes including a second set of bits in the encoded video data, wherein the second set of bits indicates transform coefficients having the corresponding index value, and the number of bits in the second set of bits is equal to the bit width indicated by the bit width parameter for the corresponding index value, and parsing a third set of bits from the error-corrected encoded video data. wherein the third set of bits indicates transform coefficients having the corresponding index value, and the number of bits in the third set of bits is equal to the bit width indicated by the bit width parameter for the corresponding index value.

[0338] Clause 10A. The method according to any one of Clauses 1A-9A, wherein the method further comprises performing the bit depunting operation on the error correction data before generating the error correction encoded video data.

[0339] Clause 11A. The method according to any one of Clauses 1A-10A, wherein the parameter comprises one or more of the following: color space, transform size, quantization parameter, number of transform coefficients in the first encoded video data, or number of bits for each transform coefficient in the first encoded video data.

[0340] Clause 12A. The method according to any one of Clauses 1A-11A, wherein: the extraction mode defines the mode of anchored transform blocks and non-anchored transform blocks in an image, the method further comprising receiving at a receiving device system bits of anchored transform blocks instead of system bits of said non-anchored transform blocks, the system bits of anchored transform blocks representing transform coefficients in the anchored transform blocks, and the system bits of non-anchored transform blocks representing reduced bit-depth versions of the original transform coefficients in the non-anchored transform blocks; error correction data including error correction data for anchored transform blocks and error correction data for non-anchored transform blocks, wherein the error correction data for non-anchored transform blocks is based on the original transform coefficients in the non-anchored transform blocks, generating error-corrected encoded video data comprising: performing error correction on the system bits of the anchored transform blocks using the error correction data for the anchored transform blocks; and performing error correction on a portion of the encoded video data corresponding to said non-anchored transform blocks using said error correction data for said non-anchored transform blocks.

[0341] Clause 13A. Determining the extraction mode at the receiving device; and transmitting the extraction mode to the transmitting device at the receiving device.

[0342] Clause 14A. The method according to any one of Clauses 1A-13A, wherein: the extracted pattern defines the pattern of anchored transform blocks and non-anchored transform blocks in an image, the transform coefficients in the non-anchored transform blocks having a reduced bit depth relative to the anchored transform blocks; at the receiving device, the system bits of the anchored transform blocks, the system bits of the non-anchored transform blocks, and a correlation matrix are received, the system bits of the anchored transform blocks representing the transform coefficients in the anchored transform blocks, and the system bits of the non-anchored transform blocks representing a reduced bit depth version of the original transform coefficients in the non-anchored transform blocks; the reconstructing operation includes, for each non-anchored transform coefficient in the non-anchored transform blocks: at the receiving device, calculating an interpolated value of the non-anchored transform coefficient based on the correlation matrix and the corresponding anchored transform coefficient; and at the receiving device, calculating a reconstructed value of the non-anchored transform coefficient based on the interpolated value of the non-anchored transform coefficient in the error-corrected encoded video data and the value of the non-anchored transform coefficient.

[0343] Clause 15A. A method for encoding video data, the method comprising: obtaining video data from a video source at a transmitting device; generating encoded video data of a first image of the video data and encoded video data of a second image of the video data based on a parameter set at the transmitting device; performing channel coding on the encoded video data of the first image and the encoded video data of the second image at the transmitting device to generate error correction data for the first image and error correction data for the second image; and transmitting the encoded video data of the first image, the error correction data for the first image, and the error correction data for the second image at the transmitting device.

[0344] Clause 16A. The method according to Clause 15A further includes transmitting the value of the parameter from the transmitting device to the receiving device.

[0345] Clause 17A. The method according to Clause 16A, wherein: the parameters include one or more quantization parameters, and generating the encoded video data includes using the quantization parameters to quantize the transform coefficients of the first image and the transform coefficients of the second image.

[0346] Clause 18A. The method according to any one of Clauses 16A-17A, wherein: the parameters include a transform size parameter, and generating the coded video data includes applying a forward transform to residual data blocks of the first image and residual data blocks of the second image, wherein the forward transform has a transform size indicated by the transform size parameter.

[0347] Clause 19A. The method according to any one of Clauses 16A-18A, wherein: the parameters include parameters indicating the number of transform coefficients, and the coded video data for generating the first image and the second image includes a set of transform coefficients in the coded video data for the first image and the coded video data for the second image, the set of transform coefficients including the indicated number of transform coefficients.

[0348] Clause 20A. The method according to any one of Clauses 16A to 10A, wherein the parameter comprises one or more of the following: color space, transform size, quantization parameter, number of transform coefficients in the encoded video data, or number of bits per transform coefficient in the encoded video data.

[0349] Clause 21A. A method for encoding video data, the method comprising: obtaining video data from a video source at a transmitting device; generating transform blocks based on the video data at the transmitting device; determining, at the transmitting device, which transform blocks are anchored transform blocks; calculating a correlation matrix of a set of transform blocks at the transmitting device; generating a bit-reduced non-anchored transform matrix at the transmitting device; and transmitting the anchored transform blocks, the non-anchored transform blocks, and the correlation matrix to a receiving device at the transmitting device.

[0350] Clause 22A. The method according to Clause 21A further includes: at the transmitting device, receiving an indication of a extraction mode from the receiving device.

[0351] Clause 23A. An apparatus comprising: a memory configured to store video data; a communication interface; and one or more processes implemented in a circuit and coupled to the memory, the one or more processors being configured to perform the method of any one of Clauses 1A-22A.

[0352] Clause 24A. An apparatus comprising means for performing the method of any one of Clauses 1A-22A.

[0353] Clause 25A. A computer-readable data storage medium having instructions stored thereon, which, when executed, cause a device to perform any one of Clauses 1A-22A.

[0354] Clause 1B. An apparatus for processing video data, the apparatus comprising: a memory configured to store video data; and a communication interface configured to obtain error correction data from a transmitting means, wherein the error correction data provides error correction information about images of the video data; one or more processors implemented in circuitry and coupled to the memory, the one or more processors being configured to: generate prediction data for images, wherein the prediction data for images includes predictions of blocks of images at least partially based on one or more previously reconstructed images of the video data; generate encoded video data based on the prediction data for images, wherein the encoded video data includes transform blocks, the transform blocks including transform coefficients; scale the bits of the transform coefficients of the transform blocks based on reliability values ​​of bit positions; generate error-corrected encoded video data using the error correction data to perform an error correction operation on the scaled bits of the transform coefficients of the transform blocks; and reconstruct the images based on the error-corrected encoded video data.

[0355] Clause 2B. The apparatus according to Clause 1B, wherein the one or more processors are further configured to generate the reliability value at the receiving device.

[0356] Clause 3B. The apparatus according to Clause 2B, wherein the one or more processors are configured to generate the reliability value based on statistics regarding the occurrence of errors in the bit positions.

[0357] Clause 4B. The apparatus according to any one of Clauses 2B or 3B, wherein the one or more processors are configured to generate the reliability value based on the reliability characteristics of individual regions of the images of the video data.

[0358] Clause 5B. An apparatus pursuant to any of Clauses 2B-4B, wherein one or more processors are configured to generate reliability values ​​based on a noise model.

[0359] Clause 6B. The apparatus according to any one of Clauses 1B-5B, wherein the communication interface is further configured to transmit a reliability value to the transmitting device.

[0360] Clause 7B. An apparatus pursuant to any of Clauses 1B-5B, wherein the communication interface is further configured to receive a reliability value from the transmitting device.

[0361] Clause 8B. An apparatus for processing video data, the method comprising: a memory configured to store video data; and one or more processors implemented in a circuit and coupled to the memory, the one or more processors configured to: acquire the video data; acquire predictive quality feedback, wherein the predictive quality feedback is based on the reliability of an estimated picture generated by a receiving device; adapt one or more of video coding parameters or channel coding parameters based on the predictive quality feedback; perform video coding processing to generate encoded video data based on the one or more pictures of the acquired video data, wherein the video coding processing is controlled by the video coding parameters; perform channel coding processing on the encoded video data to generate channel-coded data, wherein the channel coding processing is controlled by the channel coding parameters; and a communication interface configured to transmit the channel-coded data to the receiving device.

[0362] Clause 9B. The apparatus according to Clause 8B, wherein: the video coding parameters include quantization parameters, the one or more processors are configured to adapt the quantization parameters as part of adapting the video coding parameters, and the one or more processors are configured to use the quantization parameters to quantize the transform coefficients of transform blocks of the one or more images as part of performing the video coding process.

[0363] Clause 10B. An apparatus according to any one of Clauses 8B-9B, wherein: the channel coding parameters include a low-density parity-check (LDPC) graph, the one or more processors are configured to: adapt the LDPC graph as part of adapting the channel coding parameters, and the one or more processors are configured to: use the LDPC graph to generate codewords included in the channel coding data as part of performing the channel coding process.

[0364] Clause 11B. An apparatus according to any one of Clauses 8B-10B, wherein: the channel coding data includes error correction data, one or more processors are further configured to adapt one or more bit puncturing parameters based on predictive quality feedback, and one or more processors are configured to perform bit puncturing processing on the error correction data, wherein the bit puncturing processing is controlled by one or more bit puncturing parameters.

[0365] Clause 12B. A method of processing video data, the method comprising: obtaining error correction data at a receiving device and from a transmitting device, wherein the error correction data provides error correction information about an image of the video data; generating prediction data of the image at the receiving device, wherein the prediction data of the image includes predictions of blocks of the image based at least in part on one or more previously reconstructed images of the video data; generating encoded video data at the receiving device based on the prediction data of the image, wherein the encoded video data includes transform blocks, the transform blocks including transform coefficients; scaling bits of the transform coefficients of the transform blocks based on reliability values ​​of bit positions at the receiving device; generating error-corrected encoded video data at the receiving device using the error correction data to perform an error correction operation on the scaled bits of the transform coefficients of the transform blocks; and reconstructing an image at the receiving device based on the error-corrected encoded video data.

[0366] Clause 13B. The method according to Clause 12B further includes generating the reliability value at the receiving device.

[0367] Clause 14B. The method according to Clause 13B, wherein generating the reliability value includes: at the receiving device, generating the reliability value based on statistics regarding the occurrence of errors in bit positions.

[0368] Clause 15B. The method according to any one of Clauses 13B or 14B, wherein generating a reliability value includes generating a reliability value at the receiving device based on the reliability characteristics of individual regions of a picture of video data.

[0369] Clause 16B. The method according to any one of Clauses 13B-15B, wherein generating the reliability value includes generating the reliability value based on a noise model at the receiving device.

[0370] Clause 17B. The method according to any one of Clauses 12B-16B further includes transmitting a reliability value to the delivery device at the receiving device.

[0371] Clause 18B. The method pursuant to any one of Clauses 12B-16B further includes receiving a reliability value from the transmitting device at the receiving device.

[0372] Clause 19B. A method for processing video data, the method comprising: acquiring video data; acquiring predictive quality feedback, wherein the predictive quality feedback is based on the reliability of estimated pictures generated by a receiving device; adapting one or more of video coding parameters or channel coding parameters based on the predictive quality feedback; performing video coding processing to generate encoded video data based on one or more pictures of the acquired video data, wherein the video coding processing is controlled by the video coding parameters; performing channel coding processing on the encoded video data to generate channel-coded data, wherein the channel coding processing is controlled by the channel coding parameters; and transmitting the channel-coded data to the receiving device.

[0373] Clause 20B. The method according to Clause 19B, wherein: the video coding parameters include quantization parameters, adjusting the video coding parameters includes adjusting the quantization parameters, and performing the video coding process includes using the quantization parameters to quantize the transform coefficients of the transform blocks of the one or more images.

[0374] Clause 21B. The method according to any one of Clauses 19B-20B, wherein: the channel coding parameters include a low-density parity-check (LDPC) graph, the adapted channel coding parameters include an adapted LPPC graph, and performing the channel coding process includes using the LDPC graph to generate codewords included in the channel coding data.

[0375] Clause 22B. The method according to any one of Clauses 19B-21B, wherein: the channel-coded data includes error-correcting data, and the method further comprises: adjusting one or more bit puncturing parameters based on prediction quality feedback, and performing bit puncturing processing on the error-correcting data, wherein the bit puncturing processing is controlled by one or more bit puncturing parameters.

[0376] Clause 23B. An apparatus comprising means for performing the method of any one of Clauses 12B-22B.

[0377] Clause 24B. A computer-readable data storage medium having instructions stored thereon, which, when executed, cause a device to perform any one of Clauses 12B-22B.

[0378] Clause 1C. An apparatus comprising: a memory configured to store video data; and one or more processors implemented in circuitry and coupled to the memory, the one or more processors configured to: obtain a first set of multi-view images of the video data, wherein the first set of multi-view images includes a first image and a second image, the first image being from a first viewpoint and the second image being from a second viewpoint; transmit first encoded video data to a receiving device, wherein the first encoded video data is based on the first set of multi-view images; receive a multi-view encoding prompt from the receiving device; obtain a second set of multi-view images of the video data, wherein the second set of multi-view images includes a third image and a fourth image, the third image being from the first viewpoint and the fourth image being from the second viewpoint; perform multi-view encoding processing on the second set of multi-view images based on the multi-view encoding prompt received from the receiving device to generate second encoded video data, wherein the multi-view encoding processing reduces inter-view redundancy between the third image and the fourth image; and transmit the second encoded video data to the receiving device.

[0379] Clause 2C. The apparatus according to Clause 1C, wherein the one or more processors are further configured to: after transmitting the second encoded video data to the receiving device, receive an updated multi-view encoding cue from the receiving device; obtain a third set of multi-view images of the video data, wherein the third set of multi-view images comprises a fifth image and a sixth image, the fifth image being from the first viewpoint and the sixth image being from the second viewpoint; encode the third set of multi-view images based on the updated multi-view encoding cue received from the receiving device to generate third encoded video data; and transmit the third encoded video data to the receiving device.

[0380] Clause 3C. An apparatus according to any one of Clauses 1C-2C, wherein the multi-view encoding prompt includes one or more of the following: relative shift between blocks of a first image and a second image, brightness correction between the first image and the second image, inter-block shift between anchor blocks and reconstructed blocks, or motion data for reference shift.

[0381] Clause 4C. An apparatus according to any one of Clauses 1C-3C, wherein: the apparatus is an extended reality (XR) head-mounted device, and the one or more processors are configured to: receive from the receiving device virtual element data generated based on a first set of multi-view images and a second set of multi-view images; and output the virtual element data for display in an XR scene.

[0382] Clause 5C. An apparatus comprising: a memory configured to store video data; and one or more processors implemented in circuitry and coupled to the memory, the one or more processors configured to: obtain first encoded video data from a transmitting device, wherein the first encoded video data is a first set of multi-view images based on the video data, the first set of multi-view images including a first image and a second image, the first image being from a first viewpoint and the second image being from a second viewpoint; determine a multi-view encoding cue based on the first encoded video data; transmit the multi-view encoding cue to the transmitting device; and obtain second encoded video data from the transmitting device, wherein the second encoded video data is based on a second set of multi-view images including a third image and a fourth image, the second encoded video data being encoded using multi-view encoding processing that reduces inter-view redundancy between the third image and the fourth image based on the multi-view encoding cue.

[0383] Clause 6C. The apparatus according to Clause 5C, wherein the one or more processors are further configured to decode the second encoded video data.

[0384] Clause 7C. The apparatus according to any one of Clauses 5C-6C, wherein the multi-view encoding hint is a first multi-view encoding hint, and the one or more processors are further configured to: determine a second multi-view encoding hint based on the second encoded video data; transmit the second multi-view encoding hint to the transmitting device; and obtain third encoded video data from the transmitting device, wherein the third encoded video data is based on a third set of multi-view images including a fifth image and a sixth image, and the third encoded video data is encoded using the multi-view encoding process that reduces inter-view redundancy between the fifth image and the sixth image based on the second multi-view encoding hint.

[0385] The apparatus of any one of Clauses 8C, 5C-7C, wherein: the multi-view encoding hint includes a depth map indicating the depth of an object represented in the first and second images, and one or more processors are configured to determine the depth map based on the first and second images as part of determining the multi-view encoding hint.

[0386] Clause 9C. The apparatus according to Clauses 5C to 8C, wherein the multi-view encoded cue includes one or more lighting compensation factors, and the one or more processors are configured to determine the lighting compensation factors based on the first image and the second image as part of determining the multi-view encoded cue.

[0387] Clause 10C. An apparatus according to any one of Clauses 5C-9C, wherein: the transmitting means is an extended reality (XR) head-mounted device, and the one or more processors are further configured to: process the second set of images to generate virtual element data; and transmit the virtual element data to the XR head-mounted device.

[0388] Clause 11C. A method for processing video data, the method comprising: obtaining a first set of multi-view images of the video data, wherein the first set of multi-view images includes a first image and a second image, the first image being from a first viewpoint and the second image being from a second viewpoint; transmitting first encoded video data to a receiving device, wherein the first encoded video data is based on the first set of multi-view images; receiving a multi-view encoding prompt from the receiving device; obtaining a second set of multi-view images of the video data, wherein the second set of multi-view images includes a third image and a fourth image, the third image being from the first viewpoint and the fourth image being from the second viewpoint; performing multi-view encoding processing on the second set of multi-view images based on the multi-view encoding prompt received from the receiving device to generate second encoded video data, wherein the multi-view encoding processing reduces inter-view redundancy between the third image and the fourth image; and transmitting the second encoded video data to the receiving device.

[0389] Clause 12C. After transmitting the second encoded video data to the receiving device, receiving an updated multi-view encoding prompt from the receiving device; obtaining a third set of multi-view images of the video data, wherein the third set of multi-view images includes a fifth image and a sixth image, the fifth image being from the first viewpoint and the sixth image being from the second viewpoint; encoding the third set of multi-view images based on the updated multi-view encoding prompt received from the receiving device to generate third encoded video data; and transmitting the third encoded video data to the receiving device.

[0390] Clause 13C. The method according to any one of Clauses 11C-12C, wherein the multi-view encoding hint includes one or more of the following: relative shift between blocks of the first and second images, brightness correction between the first and second images, inter-block shift between anchor blocks and reconstructed blocks, or motion data for reference shift.

[0391] Clause 14C. The method of any one of Clauses 11C-13C, wherein the method further comprises: receiving virtual element data generated from a receiving device based on first and second sets of multi-view images; and outputting the virtual element data for display in an extended reality (XR) scene.

[0392] Clause 15C. A method for processing video data, the method comprising: obtaining first encoded video data from a transmitting device, wherein the first encoded video data is based on a first set of multi-view images of the video data, the first set of multi-view images including a first image and a second image, the first image being from a first viewpoint and the second image being from a second viewpoint; determining a multi-view encoding cue based on the first encoded video data; sending the multi-view encoding cue to the transmitting device; and obtaining second encoded video data from the transmitting device, wherein the second encoded video data is based on a second set of multi-view images including a third image and a fourth image, the second encoded video data being encoded using a multi-view encoding process, the multi-view encoding process reducing inter-view redundancy between the third image and the fourth image based on the multi-view encoding cue.

[0393] Clause 16C. The method described in Clause 15C further includes decoding the second encoded video data.

[0394] Clause 17C. The method according to any one of Clauses 15C-16C, wherein the multi-view encoding hint is a first multi-view encoding hint, and the method further comprises: determining a second multi-view encoding hint based on the second encoded video data; transmitting the second multi-view encoding hint to the transmitting device; and obtaining third encoded video data from the transmitting device, wherein the third encoded video data is based on a third set of multi-view images including a fifth image and a sixth image, and the third encoded video data is encoded using the multi-view encoding process that reduces inter-view redundancy between the fifth image and the sixth image based on the second multi-view encoding hint.

[0395] Clause 18C. The method of any one of Clauses 15C-17C, wherein: the multi-view encoding hint includes a depth map indicating the depth of objects represented in the first and second images, and determining the multi-view encoding hint includes determining the depth map based on the first and second images.

[0396] Clause 19C. The method according to Clauses 15C-18C, wherein the multi-view encoded hint includes one or more lighting compensation factors, and determining the multi-view encoded hint includes determining the lighting compensation factors based on the first image and the second image.

[0397] Clause 20C. The method according to any one of Clauses 15C-19C, wherein: the transmitting device is an extended reality (XR) head-mounted device, and the method further comprises: processing the second set of images to generate virtual element data; and transmitting the virtual element data to the XR head-mounted device.

[0398] Clause 21C. An apparatus comprising: means for acquiring a first set of multi-view images of video data, wherein the first set of multi-view images includes a first image and a second image, the first image being from a first viewpoint and the second image being from a second viewpoint; means for transmitting first encoded video data to a receiving means, wherein the first encoded video data is based on the first set of multi-view images; means for receiving a multi-view encoding prompt from the receiving means; means for acquiring a second set of multi-view images of the video data, wherein the second set of multi-view images includes a third image and a fourth image, the third image being from the first viewpoint and the fourth image being from the second viewpoint; means for performing multi-view encoding processing on the second set of multi-view images based on the multi-view encoding prompt received from the receiving means to generate second encoded video data, wherein the multi-view encoding processing reduces inter-view redundancy between the third image and the fourth image; and means for transmitting the second encoded video data to the receiving means.

[0399] Clause 22C. An apparatus comprising: means for obtaining first encoded video data from a transmitting means, wherein the first encoded video data is based on a first set of multi-view images of the video data, the first set of multi-view images including a first image and a second image, the first image being from a first viewpoint and the second image being from a second viewpoint; means for determining a multi-view encoding cue based on the first encoded video data; means for transmitting the multi-view encoding cue to the transmitting means; and means for obtaining second encoded video data from the transmitting means, wherein the second encoded video data is based on a second set of multi-view images including a third image and a fourth image, the second encoded video data being encoded using a multi-view encoding process that reduces inter-view redundancy between the third image and the fourth image based on the multi-view encoding cue.

[0400] Clause 1D. An apparatus comprising: a memory configured to store video data; and one or more processors implemented in a circuit and coupled to the memory, the one or more processors configured to: encode a first set of images of the video data to generate first encoded video data; transmit the first encoded video data to a receiving device; receive from the receiving device a decimation mode indication indicating a decimation mode determined based on the first set of images, the decimation mode being a mode in which the encoded video data is not transmitted; encode a second set of images of the video data to generate second encoded video data; apply the decimation mode to the second encoded video data to generate decimated video data; and transmit the decimated video data to the receiving device.

[0401] Clause 2D. The apparatus according to Clause 1D, wherein the one or more processors are configured to: generate first error correction data based on the first encoded video data; transmit the first error correction data to the receiving device; generate second error correction data based on the second encoded video data; and transmit the second error correction data to the receiving device.

[0402] Clause 3D. An apparatus according to any one of Clauses 1D-2D, wherein the extraction mode indicates a mode for skipping the transmission of encoded video data of a complete picture.

[0403] Clause 4D. An apparatus according to any one of Clauses 1D-3D, wherein the extraction mode indicates a mode for skipping the transmission of encoded video data in a specific region within a picture.

[0404] Clause 5D. An apparatus according to any one of Clauses 1D-4D, wherein the video data is multi-view video data and the extraction mode indicates a mode for skipping the transmission of encoded video data from pictures of a particular view.

[0405] Clause 6D. An apparatus according to any one of Clauses 1D-5D, wherein: the decimation mode indication is a first decimation mode indication, the mode in which encoded video data is not transmitted is a first mode in which encoded video data is not transmitted, the decimated video data is first decimated video data, and the one or more processors are further configured to: encode a third set of images of the video data to generate third encoded video data; determine a second decimation mode indicating a second mode in which encoded video data is not transmitted; apply the second decimation mode to the third encoded video data to generate second decimated video data; transmit the second decimated video data to the receiving device; and transmit a second decimation mode indication to the receiving device, the second decimation mode indication indicating that the second decimation mode is applied to the third encoded video data.

[0406] Clause 7D. An apparatus according to any one of Clauses 1D-6D, wherein: the one or more processors are configured, as part of encoding the first set of images: to generate first prediction data for the first set of images; to generate residual data based on the first prediction data and the first set of images; to apply a transform to the first prediction data to generate a transform block; to quantize the transform coefficients of the transform block; to apply entropy coding to a syntax element representing the quantized transform coefficients to generate a first entropy-coded syntax element, wherein the first encoded video data includes the first entropy-coded syntax element; the one or more processors are further configured to perform analog modulation on the residual data to generate first analog modulated residual data, and the apparatus further includes a communication interface configured to transmit the first analog modulated residual data and the first encoded video data.

[0407] Clause 8D. An apparatus according to any one of Clauses 1D-7D, wherein: the apparatus is an extended reality (XR) head-mounted device and includes a display system, and the one or more processors are further configured to: receive virtual element data from the receiving device; and the display system is configured to display one or more virtual elements in an XR scene based on the virtual element data.

[0408] Clause 9D. An apparatus comprising: a memory configured to store video data; and one or more processors implemented in a circuit and coupled to the memory, the one or more processors being configured to: receive first encoded video data from a transmitting device; perform decoding processing to reconstruct a first set of images based on the first encoded video data; determine, based on the first set of images, an extraction mode indicating a mode in which the encoded video data has not been transmitted; send to the transmitting device an extraction mode indication indicating the determined extraction mode; receive extracted video data from the transmitting device, wherein the extracted video data includes second encoded video data to which the extraction mode has been applied, wherein the second encoded video data is generated based on a second set of images of the video data; and perform the decoding processing to reconstruct the second set of images based on the second encoded video data.

[0409] Clause 10D. The apparatus according to Clause 9D, wherein the one or more processors are further configured to: receive first error correction data from the transmitting device; apply error correction processing to modify the first encoded video data based on the first error correction data to generate first error-corrected encoded video data; wherein the one or more processors are configured to perform the decoding processing to reconstruct the first set of images based on the first error-corrected encoded video data; wherein the one or more processors are further configured to: receive second error correction data from the transmitting device; apply the error correction processing to generate second error-corrected encoded video data based on the second encoded video data and the second error correction data; and wherein the one or more processors are configured to perform the decoding processing to reconstruct the second set of images based on the second error-corrected encoded video data.

[0410] Clause 11D. An apparatus pursuant to any of Clauses 9D-10D, wherein the extraction mode indicates a mode for skipping the transmission of encoded video data of a complete picture.

[0411] Clause 12D. An apparatus according to any one of Clauses 9D-11D, wherein, as part of determining the decimation pattern, the one or more processors are configured to: apply the decimation pattern to the first encoded video data to generate decimated encoded video data; apply error correction processing to modify the decimated encoded video data based on the first error correction data to generate trial error-corrected video data; apply the decoding processing to reconstruct the first set of images based on the trial error-corrected video data; and determine whether the decimation pattern meets a criterion based on a comparison between the first set of images reconstructed as based on the trial error-corrected video data and the first set of images reconstructed as based on the first video data.

[0412] Clause 13D. An apparatus pursuant to any of Clauses 9D-12D, wherein the extraction mode indicates a mode for skipping the transmission of coded video data in a specified region within a picture.

[0413] Clause 14D. An apparatus pursuant to any one of Clauses 9D-13D, wherein the video data is multi-view video data and the extraction mode indicates a mode that skips the transmission of encoded video data from pictures of a specified view.

[0414] Clause 15D. An apparatus according to any one of Clauses 9D-14D, wherein: the decimation mode indication is a first decimation mode indication, the mode in which encoded video data is not transmitted is a first mode in which encoded video data is not transmitted, the decimated video data is first decimated video data, and the one or more processors are further configured to: receive a second decimation mode indication indicating a second mode in which encoded video data is not transmitted; receive second decimated video data from the transmitting device, wherein the second decimated video data includes third encoded video data to which the second decimation mode has been applied, wherein the third encoded video data is generated based on a third set of images of the video data; and apply decoding processing to reconstruct the third set of images based on the third encoded video data.

[0415] Clause 16D. An apparatus according to any one of Clauses 9D to 15D, wherein: the apparatus further includes a communication interface configured to receive analog modulation residual data, the second encoded video data comprising entropy-coded syntax elements representing quantized transform coefficients; the one or more processors are configured as part of applying the decoding process to reconstruct the second set of images: applying entropy decoding to the syntax elements to obtain the quantized transform coefficients; inverse-quantizing the quantized transform coefficients to generate inverse-quantized transform coefficients; applying an inverse transform to the inverse-quantized transform coefficients to generate prediction data; demodulating the analog modulation residual data to obtain residual data; and reconstructing the second set of images based on the prediction data and the residual data.

[0416] Clause 17D. An apparatus according to any one of Clauses 9D-16D, wherein: the one or more processors are further configured to process the second set of images to generate virtual element data, and the transmitting means is an extended reality (XR) head-mounted device configured to display one or more virtual elements in an XR scene based on the virtual element data.

[0417] Clause 18D. A method comprising: encoding a first set of images of video data to generate first encoded video data; transmitting the first encoded video data to a receiving device; receiving from the receiving device the sampling mode indication indicating a sampling mode determined based on the first set of images, the sampling mode being a mode in which the encoded video data is not transmitted; encoding a second set of images of the video data to generate second encoded video data; applying the sampling mode to the second encoded video data to generate sampling video data; and transmitting the sampling video data to the receiving device.

[0418] Clause 19D. The method according to Clause 18D, comprising generating first error correction data based on the first encoded video data; transmitting the first error correction data to the receiving device; generating second error correction data based on the second encoded video data; and transmitting the second error correction data to the receiving device.

[0419] Clause 20D. The method according to any one of Clauses 18D-19D, wherein the extraction mode indicates a mode for skipping the transmission of encoded video data of the complete picture.

[0420] Clause 21D. The method according to any one of Clauses 18D-20D, wherein the extraction mode indicates a mode for skipping the transmission of encoded video data in a specific region within a picture.

[0421] Clause 22D. The method according to any one of Clauses 18D-21D, wherein the video data is multi-view video data and the extraction mode indicates a mode for skipping the transmission of encoded video data from pictures of a particular view.

[0422] Clause 23D. A method according to any one of Clauses 18D to 22D, wherein: the decimation mode indication is a first decimation mode indication, the mode of not transmitting encoded video data is a first mode of not transmitting encoded video data, the decimated video data is first decimated video data, and the method further comprises: a third set of images of encoded video data to generate third encoded video data; determining a second decimation mode indicating a second mode of not transmitting encoded video data; applying the second decimation mode to the third encoded video data to generate second decimated video data; transmitting the second decimated video data to the receiving device; and transmitting a second decimation mode indication to the receiving device, the second decimation mode indication indicating that the second decimation mode is applied to the third encoded video data.

[0423] Clause 24D. A method according to any one of Clauses 18D-23D, wherein: encoding a first set of images comprises: generating first prediction data for the first set of images; generating residual data based on the first prediction data and the first set of images; applying a transform to the first prediction data to generate a transform block; quantizing the transform coefficients of the transform block; applying entropy coding to a syntax element representing the quantized transform coefficients to generate a first entropy-coded syntax element, wherein the first encoded video data includes the first entropy-coded syntax element; the method further comprises: performing analog modulation on the residual data to generate first analog-modulated residual data, and transmitting the first analog-modulated residual data and the first encoded video data.

[0424] Clause 25D. The method according to any one of Clauses 18D-24D, wherein the method further comprises: receiving virtual element data from the receiving device; and displaying one or more virtual elements in an extended reality (XR) scene based on the virtual element data.

[0425] Clause 26D. A method comprising: receiving first encoded video data from a transmitting device; applying decoding processing to reconstruct a first set of images based on the first encoded video data; determining, based on the first set of images, an extraction mode indicating a mode in which the encoded video data has not been transmitted; sending, to the transmitting device, an extraction mode indication indicating the determined extraction mode; receiving extracted video data from the transmitting device, wherein the extracted video data includes second encoded video data to which the extraction mode has been applied, wherein the second encoded video data is generated based on a second set of images of the video data; and performing the decoding processing to reconstruct the second set of images based on the second encoded video data.

[0426] Clause 27D. The method according to Clause 26D, wherein the method further comprises: receiving first error correction data from a transmitting device; applying error correction processing to modify the first encoded video data based on the first error correction data to generate first error-corrected encoded video data; wherein performing the decoding processing to reconstruct the first set of images comprises performing the decoding processing based on the first error-corrected encoded video data to reconstruct the first set of images; wherein the method further comprises: receiving second error correction data from the transmitting device; applying the error correction processing to generate second error-corrected encoded video data based on the second encoded video data and the second error correction data; and wherein performing the decoding processing to reconstruct the second set of images comprises performing the decoding processing based on the second error-corrected encoded video data to reconstruct the second set of images.

[0427] Clause 28D. The method according to any one of Clauses 26D-27D, wherein the extraction mode indicates a mode for skipping the transmission of encoded video data of the complete picture.

[0428] Clause 29D. The method according to any one of Clauses 26D-28D, wherein determining the extraction pattern comprises: applying the extraction pattern to the first encoded video data to generate extracted encoded video data; applying error correction processing to modify the extracted encoded video data based on the first error correction data to generate trial error-corrected video data; applying the decoding processing to reconstruct the first set of images based on the trial error-corrected video data; and determining whether the extraction pattern satisfies a standard based on a comparison between the first set of images reconstructed as based on the trial error-corrected video data and the first set of images reconstructed as based on the first video data.

[0429] Clause 30D. The method according to any one of Clauses 26D-29D, wherein the extraction mode indicates a mode for skipping the transmission of coded video data in a specified region within the picture.

[0430] Clause 31D. The method according to any one of Clauses 26D-30D, wherein the video data is multi-view video data, and the extraction mode indicates a mode for skipping the transmission of encoded video data from pictures of a specified view.

[0431] Clause 32D. The method according to any one of Clauses 26D-31D, wherein: the decimation mode indication is a first decimation mode indication, the mode in which encoded video data is not transmitted is a first mode in which encoded video data is not transmitted, the decimated video data is first decimated video data, and the method further comprises: receiving a second decimation mode indication indicating a second mode in which encoded video data is not transmitted; receiving second decimated video data from the transmitting device, wherein the second decimated video data includes third encoded video data to which the second decimation mode has been applied, wherein the third encoded video data is generated based on a third set of images of the video data; and applying decoding processing to reconstruct the third set of images based on the third encoded video data.

[0432] Clause 33D. The method according to any one of Clauses 26D to 32D, wherein: the method further comprises receiving analog modulation residual data, the second encoded video data comprising entropy-coded syntax elements representing quantized transform coefficients; applying the decoding process to reconstruct the second set of pictures comprises: applying entropy decoding to the syntax elements to obtain the quantized transform coefficients; inverse-quantizing the quantized transform coefficients to generate inverse-quantized transform coefficients; applying the inverse transform to the inverse-quantized transform coefficients to generate prediction data; demodulating the analog modulation residual data to obtain residual data; and reconstructing the second set of pictures based on the prediction data and the residual data.

[0433] Clause 34D. The method according to any one of Clauses 26D to 33D, wherein: the method further comprises processing the second set of images to generate virtual element data, and the transmitting device is an extended reality (XR) head-mounted device configured to display one or more virtual elements in an XR scene based on the virtual element data.

[0434] Clause 35D. An apparatus comprising: means for encoding a first set of images of video data to generate first encoded video data; means for transmitting the first encoded video data to a receiving means; means for receiving from the receiving means an indication of a decimation mode based on the first set of images, the decimation mode being a mode in which the encoded video data is not transmitted; means for encoding a second set of images of the video data to generate second encoded video data; means for applying the decimation mode to the second encoded video data to generate decimated video data; and means for transmitting the decimated video data to the receiving means.

[0435] Clause 36D. An apparatus comprising: means for receiving first encoded video data from a transmitting means; means for performing decoding processing to reconstruct a first set of pictures based on the first encoded video data; means for determining an extraction mode based on the first set of pictures that indicates a mode in which the encoded video data has not been transmitted; means for delivering to the transmitting means the extraction mode indication that indicates the determined extraction mode; means for receiving extracted video data from the transmitting means, wherein the extracted video data includes second encoded video data to which the extraction mode has been applied, wherein the second encoded video data is generated based on a second set of pictures of the video data; and means for performing decoding processing to reconstruct a second set of pictures based on the second encoded video data.

[0436] Clause 1E. An apparatus comprising: a memory configured to store video data; and one or more processors implemented in a circuit and coupled to the memory, the one or more processors configured to: encode a first image of the video data to generate first encoded video data; transmit the first encoded video data to a receiving device; receive encoding selection data of a second image of the video data from the receiving device, wherein: the encoding selection data of the second image indicates encoding selections for encoding an estimate of the second image, and the second image is after the first image in decoding order; encode the second image based on the encoding selection data of the second image to generate second encoded video data; and transmit the second encoded video data to the receiving device.

[0437] Clause 2E. The apparatus according to Clause 1E, wherein: the encoded selection data received from the receiving device includes motion parameters of blocks of the second picture as part of encoding the second picture, the one or more processors are configured to perform motion compensation based on the motion parameters of the blocks of the second picture to generate predicted blocks, and the second encoded video data includes encoded video data based on the predicted blocks.

[0438] Clause 3E. An apparatus according to any one of Clauses 1E-2E, wherein: the encoding selection data received from the receiving device includes intra-frame prediction parameters for blocks of the second picture, the one or more processors are configured to perform intra-frame prediction based on the intra-frame prediction parameters for blocks of the second picture to generate prediction blocks as part of encoding the second picture, and the second encoded video data includes encoded video data based on the prediction blocks.

[0439] Clause 4E. An apparatus pursuant to any of Clauses 1E-3E, wherein the second encoded video data does not include encoding selection data.

[0440] Clause 5E. An apparatus according to any one of Clauses 1E-4E, wherein one or more processors are configured to perform entropy decoding on the encoded selection data of the second picture.

[0441] Clause 6E. The apparatus according to any one of Clauses 1E-5E, wherein the one or more processors are further configured to: generate first error correction data based on the first encoded video data; and transmit the first encoded video data and the first error correction data to the receiving apparatus.

[0442] Clause 7E. The apparatus according to any one of Clauses 1E-6E, wherein the one or more processors are further configured to: encode the third image without using the encoding selection data of the third image, based on determining that no encoding selection data of the third image has been received from the receiving device before the time limit expires.

[0443] Clause 8E. An apparatus according to any one of Clauses 1E-7E, wherein one or more processors are further configured to: receive coding selection data of a third picture of video data, wherein the coding selection data of the third picture indicates an estimated coding selection for encoding the third picture; encode the third picture based on the coding selection data of the third picture to generate third encoded video data; apply channel coding processing to generate error correction data for the third encoded video data; and transmit the error correction data for the third encoded video data to the receiving apparatus without transmitting at least a portion of the third encoded video data.

[0444] Clause 9E. An apparatus according to any one of Clauses 1E-8E, wherein: the apparatus is an extended reality (XR) head-mounted device and includes a display system, and the one or more processors are further configured to: receive virtual element data from the receiving device; and the display system is configured to display one or more virtual elements in an XR scene based on the virtual element data.

[0445] Clause 10E. An apparatus comprising: a memory configured to store video data; and one or more processors implemented in a circuit and coupled to the memory, the one or more processors being configured to: receive first encoded video data from a transmitting device; reconstruct a first picture of the video data based on the first encoded video data; estimate a second picture of the video data based on the first picture, the second picture being a picture that appears after the first picture in decoding order; generate encoding selection data for the estimated second picture, wherein the encoding selection data indicates encoding selections for encoding the estimated second picture; transmit the encoding selection data for the second picture to the transmitting device; receive second encoded video data from the transmitting device; and reconstruct the second picture based on the second encoded video data.

[0446] Clause 11E. The apparatus according to Clause 10E, wherein: the one or more processors are configured to perform motion compensation based on motion parameters of the blocks of the second image as part of encoding the estimated second image to generate predicted blocks, and the encoding selection data includes the motion parameters of the blocks of the second image, and the second encoded video data includes encoded video data based on the predicted blocks.

[0447] Clause 12E. An apparatus according to any one of Clauses 10E-11E, wherein: the one or more processors are configured to, as part of encoding the second picture, perform intra-prediction based on the intra-prediction parameters for the block of the second picture to generate a prediction block, the encoding selection data includes the intra-prediction parameters for the block of the second picture, and the second encoded video data includes encoded video data based on the prediction block.

[0448] Clause 13E. An apparatus according to any one of Clauses 10E-12E, wherein: the second encoded video data does not include the encoding selection data; and the one or more processors are configured to use the encoding selection data to reconstruct the second picture based on the second encoded video data as part of applying the decoding process.

[0449] Clause 14E. An apparatus according to any one of Clauses 10E-13E, wherein one or more processors are configured to entropy encode the encoding selection data of the second picture before transmitting the encoding selection data of the second picture.

[0450] Clause 15E. An apparatus according to any one of Clauses 10E-14E, wherein: the one or more processors are further configured to: estimate a third picture of the video data based on one or more of the first picture or the second picture; perform an encoding process that encodes the estimated third picture to generate third encoded video data, wherein third encoding selection data indicates encoding selection for encoding the estimated third picture; transmit the third encoding selection data to a transmitting device; receive error correction data of the third picture from the transmitting device; apply error correction processing to generate error-corrected encoded video data for the third picture based on the error correction data for the third picture and the third encoded video data; and apply a decoding process that reconstructs the third picture based on the error-corrected encoded video data of the third picture.

[0451] Clause 16E. The apparatus according to Clause 15E, wherein: the error-corrected encoded video data of the third picture does not include the third encoding selection data, and the one or more processors are configured to, as part of applying the decoding process, reconstruct the third picture using the third encoding selection data based on the error-corrected encoded video data of the third picture.

[0452] Clause 17E. The apparatus according to any one of Clauses 10E-16E, wherein the one or more processors are configured to: apply channel coding processing to the coding selection data of the second picture to generate error correction data for the coding selection data of the second picture; and transmit the error correction data for the coding selection data of the second picture to the transmitting means.

[0453] Clause 18E. An apparatus according to any one of Clauses 10E-17E, wherein the apparatus includes a communication interface configured to modulate the coded selection data at a lower modulation order than other data transmissions in the data link between the apparatus and the transmitting apparatus.

[0454] Clause 19E. An apparatus according to any one of Clauses 10E-18E, wherein: the one or more processors are further configured to process the second set of images to generate virtual element data, and the transmitting means is an XR head-mounted device configured to display one or more virtual elements in an extended reality (XR) scene based on the virtual element data.

[0455] Clause 20E. A method for processing video data, the method comprising: encoding a first picture of the video data to generate first encoded video data; transmitting the first encoded video data to a receiving device; receiving encoding selection data of a second picture of the video data from the receiving device, wherein: the encoding selection data of the second picture indicates encoding selections for encoding an estimate of the second picture, and the second picture is after the first picture in decoding order; encoding the second picture based on the encoding selection data of the second picture to generate second encoded video data; and transmitting the second encoded video data to the receiving device.

[0456] Clause 21E. The method according to Clause 20E, wherein: the encoded selection data received from the receiving device includes motion parameters of blocks of the second picture, encoding the second picture includes performing motion compensation based on the motion parameters of the blocks of the second picture to generate predicted blocks, and the second encoded video data includes encoded video data based on the predicted blocks.

[0457] Clause 22E. The method according to any one of Clauses 20E-21E, wherein: the encoding selection data received from the receiving device includes intra-frame prediction parameters for blocks of the second picture, encoding the second picture includes performing intra-frame prediction based on the intra-frame prediction parameters for the blocks of the second picture to generate prediction blocks, and the second encoded video data includes encoded video data based on the prediction blocks.

[0458] Clause 23E. The method according to any one of Clauses 20E to 22E, wherein the second encoded video data does not contain the encoding selection data.

[0459] Clause 24E. The method according to any one of Clauses 20E-23E further includes entropy decoding of the encoded selection data used for the second picture.

[0460] Clause 25E. The method according to any one of Clauses 20E to 24E further comprises: generating first error correction data based on the first encoded video data; and transmitting the first encoded video data and the first error correction data to the receiving device.

[0461] Clause 26E. The method according to any one of Clauses 20E-25E further includes: encoding the third picture without using the encoding selection data of the third picture, based on determining that no encoding selection data of the third picture has been received from the receiving device before the time limit expires.

[0462] Clause 27E. The method according to any one of Clauses 20E-26E further comprises: receiving coding selection data of a third picture of video data, wherein the coding selection data of the third picture indicates coding selection for encoding an estimate of the third picture; encoding the third picture based on the coding selection data of the third picture to generate third encoded video data; applying channel coding processing to generate error correction data for the third encoded video data; and transmitting the error correction data for the third encoded video data to the receiving device without transmitting at least a portion of the third encoded video data.

[0463] Clause 28E. The method according to any one of Clauses 20E-27E, wherein: the apparatus is an extended reality (XR) head-mounted device and includes a display system, and the method further includes: receiving virtual element data from the receiving device; and displaying one or more virtual elements in an XR scene on the display system based on the virtual element data.

[0464] Clause 29E. A method for processing video data, the method comprising: receiving first encoded video data from a transmitting device; reconstructing a first picture of the video data based on the first encoded video data; estimating a second picture of the video data based on the first picture, the second picture being a picture that appears after the first picture in decoding order; generating encoding selection data for the estimated second picture, wherein the encoding selection data indicates encoding selections for encoding the estimated second picture; transmitting the encoding selection data for the second picture to the transmitting device; receiving second encoded video data from the transmitting device; and reconstructing the second picture based on the second encoded video data.

[0465] Clause 30E. The method according to Clause 29E, wherein: encoding the estimated second picture includes performing motion compensation based on motion parameters of the blocks in the second picture to generate predicted blocks, and the encoded selection data includes the motion parameters of the blocks in the second picture, and the second encoded video data includes encoded video data based on the predicted blocks.

[0466] Clause 31E. The method according to any one of Clauses 29E-30E, wherein: encoding the second picture includes performing intra-prediction based on the intra-prediction parameters for the block of the second picture to generate a prediction block, the encoding selection data includes the intra-prediction parameters for the block of the second picture, and the second encoded video data includes encoded video data based on the prediction block.

[0467] Clause 32E. The method according to any one of Clauses 29E to 31E, wherein: the second encoded video data does not include the encoding selection data; and applying the decoding process includes using the encoding selection data to reconstruct the second picture based on the second encoded video data.

[0468] Clause 33E. The method according to any one of Clauses 29E-32E, wherein entropy encoding of the encoding selection data for the second picture occurs before the encoding selection data for the second picture is transmitted.

[0469] Clause 34E. The method according to any one of Clauses 29E-33E further comprises: estimating a third image of the video data based on one or more of the first image or the second image; performing an encoding process that encodes the estimated third image to generate third-encoded video data, wherein third encoding selection data indicates encoding selection for encoding the estimated third image; transmitting the third encoding selection data to a transmitting device; receiving error correction data of the third image from the transmitting device; applying error correction processing to generate error-corrected encoded video data for the third image based on the error correction data for the third image and the third-encoded video data; and applying a decoding process that reconstructs the third image based on the error-corrected encoded video data of the third image.

[0470] Clause 35E. The method according to any one of Clauses 34E, wherein: the error-corrected encoded video data of the third picture does not include the third encoding selection data, and applying the decoding process includes using the third encoding selection data to reconstruct the third picture based on the error-corrected encoded video data of the third picture.

[0471] Clause 36E. The method according to any one of Clauses 29E-35E further includes: applying channel coding processing to the coding selection data of the second picture to generate error correction data for the coding selection data of the second picture; and transmitting the error correction data for the coding selection data of the second picture to the transmitting means.

[0472] Clause 37E. The method according to any one of Clauses 29E-36E further comprises: modulating the coded selection data at a lower modulation order compared to other data transmissions in the data link between the means and the transmitting means.

[0473] Clause 38E. The method according to any one of Clauses 29E-37E, wherein: it further comprises processing the second set of images to generate virtual element data, and the transmitting device is an extended reality (XR) head-mounted device configured to display one or more virtual elements in an XR scene based on the virtual element data.

[0474] Clause 39E. An apparatus comprising: means for encoding first picture of video data to generate first encoded video data; means for transmitting the first encoded video data to a receiving means; means for receiving from the receiving means encoding selection data for a second picture of the video data, wherein: the encoding selection data for the second picture indicates encoding selections for encoding an estimate of the second picture, and the second picture is after the first picture in decoding order; means for encoding the second picture based on the encoding selection data for the second picture to generate second encoded video data; and means for transmitting the second encoded video data to the receiving means.

[0475] Clause 40E. An apparatus comprising: means for receiving first encoded video data from a transmitting device; means for reconstructing a first picture of the video data based on the first encoded video data; means for estimating a second picture of the video data based on the first picture, the second picture being a picture that appears after the first picture in decoding order; means for generating encoding selection data for the second picture, wherein the encoding selection data for the second picture indicates encoding selections for encoding the second picture; means for transmitting the encoding selection data for the second picture to the transmitting device; means for receiving second encoded video data from the transmitting device; and means for reconstructing the second picture based on the second encoded video data.

[0476] It should be recognized that, depending on the instance, certain actions or events of any of the techniques described herein may be performed in a different order, and may be added, combined, or omitted entirely (e.g., practicing the techniques does not require all of the described actions or events). Furthermore, in some instances, actions or events may be performed simultaneously rather than sequentially, for example, through multithreaded processing, interrupt handling, or multiple processors.

[0477] In one or more instances, the described ...

Claims

1. An apparatus for processing video data, the apparatus comprising: The memory is configured to store video data; as well as A communication interface configured to receive error correction data from a transmitting device, wherein the error correction data provides error correction information about images of the video data; One or more processors, implemented in a circuit and coupled to the memory, are configured to: Generate prediction data for the image, wherein the prediction data for the image includes predictions of blocks of the image based at least in part on one or more previously reconstructed images of the video data; Encoded video data is generated based on the predicted data of the image, wherein the encoded video data includes transform blocks, and the transform blocks include transform coefficients; The transform coefficients of the transform block are scaled based on the reliability value of the bit position; The error correction data is used to generate error-corrected encoded video data to perform error correction on the scaled bits of the transform coefficients of the transform block; as well as The image is reconstructed based on the error-corrected encoded video data.

2. The apparatus according to claim 1, wherein, The one or more processors are also configured to generate the reliability value.

3. The apparatus according to claim 2, wherein, The one or more processors are configured to generate the reliability value based on statistics regarding the occurrence of errors in the bit positions.

4. The apparatus of claim 2, wherein the one or more processors are configured to generate the reliability value based on the reliability characteristics of individual regions of the images of the video data.

5. The apparatus of claim 2, wherein the one or more processors are configured to generate the reliability value based on a noise model.

6. The apparatus according to claim 1, wherein, The communication interface is also configured to transmit the reliability value to the transmitting device.

7. The apparatus according to claim 1, wherein, The communication interface is also configured to receive the reliability value from the transmitting device.

8. An apparatus for processing video data, the apparatus comprising: A memory configured to store the video data; as well as One or more processors, implemented in a circuit and coupled to the memory, are configured to: Obtain video data; Obtain predictive quality feedback, wherein the predictive quality feedback is based on the reliability of the estimated image generated by the receiving device; Based on the predicted quality feedback, adapt one or more of the video coding parameters or channel coding parameters; Perform video encoding processing to generate encoded video data based on one or more images of the obtained video data, wherein the video encoding processing is controlled by the video encoding parameters; Channel coding processing is performed on encoded video data to generate channel-coded data, wherein the channel coding processing is controlled by the channel coding parameters; as well as A communication interface configured to send the channel-coded data to the receiving device.

9. The apparatus according to claim 8, wherein: The video encoding parameters include quantization parameters. The one or more processors are configured to: adapt the quantization parameters as part of adapting the video encoding parameters, and The one or more processors are configured to, as part of performing video encoding processing, use quantization parameters to quantize the transform coefficients of transform blocks of one or more images.

10. The apparatus according to claim 8, wherein: The channel coding parameters include a low-density parity-check (LDPC) graph. The one or more processors are configured to: adapt the LPDC graph as part of adapting the channel coding parameters, and The one or more processors are configured to use the LDPC graph to generate codewords included in the channel-coded data as part of performing the channel coding process.

11. The apparatus according to claim 8, wherein: The channel-coded data includes error correction data. The one or more processors are further configured to adapt one or more bit puncturing parameters based on the predicted quality feedback, and The one or more processors are configured to perform bit punching on the error correction data, wherein the bit punching is controlled by the one or more bit punching parameters.

12. A method for processing video data, the method comprising: Error correction data is obtained at the receiving device and from the transmitting device, wherein the error correction data provides error correction information about images of the video data; Prediction data for the image is generated at the receiving device, wherein the prediction data for the image includes predictions of blocks of the image based at least in part on one or more previously reconstructed images of the video data; At the receiving device, encoded video data is generated based on the predicted data of the image, wherein the encoded video data includes transform blocks, and the transform blocks include transform coefficients; At the receiving device, the bits of the transform coefficients of the transform block are scaled based on the reliability value of the bit position; At the receiving device, the error correction data is used to generate error-corrected encoded video data to perform error correction on the scaled bits of the transform coefficients of the transform block; as well as The image is reconstructed at the receiving device based on the error-corrected encoded video data.

13. The method of claim 12, further comprising: The reliability value is generated at the receiving device.

14. The method according to claim 13, wherein, Generating the reliability value includes: at the receiving device, generating the reliability value based on statistics regarding the occurrence of errors in the bit positions.

15. The method of claim 13, wherein generating the reliability value comprises generating the reliability value at the receiving device based on the reliability characteristics of individual regions of the image of the video data.

16. The method according to claim 13, wherein, Generating the reliability value includes generating the reliability value based on a noise model at the receiving device.

17. The method of claim 12, further comprising: The reliability value is transmitted from the receiving device to the transmitting device.

18. The method of claim 12, further comprising: At the receiving device, the reliability value is received from the transmitting device.

19. A method for processing video data, the method comprising: Obtain video data; Obtain predictive quality feedback, wherein the predictive quality feedback is based on the reliability of the estimated image generated by the receiving device; Based on the predicted quality feedback, one or more of the video coding parameters or channel coding parameters are adapted; Perform video encoding processing to generate encoded video data based on one or more images of the obtained video data, wherein the video encoding processing is controlled by the video encoding parameters; Channel coding processing is performed on the encoded video data to generate channel-coded data, wherein the channel coding processing is controlled by the channel coding parameters; as well as The channel-coded data is sent to the receiving device.

20. The method of claim 19, wherein: The video encoding parameters include quantization parameters. Adapting the video encoding parameters includes adapting the quantization parameters, and Performing the video encoding process includes using the quantization parameters to quantize the transform coefficients of the transform blocks of the one or more images.

21. The method according to claim 19, wherein: The channel coding parameters include a low-density parity-check (LDPC) graph. Adapting the channel coding parameters includes adapting the LPDC diagram, and Performing channel coding processing involves using an LDPC graph to generate codewords that are included in the channel-coded data.

22. The method according to claim 19, wherein: The channel-coded data includes error correction data, and The method further includes: Based on the predicted quality feedback, one or more bit puncturing parameters are adapted, and Bit punching is performed on the error correction data, wherein the bit punching is controlled by one or more bit punching parameters.

Images