Augmented layer processing for extended reality (XR) content

Abstract

The base layer encoding of an image (500) is acquired. The image (500) includes XR content and is intended for an XR display device. A first enhancement layer processing is performed on a first image region (506) of the image (500). Performing the first enhancement layer processing includes acquiring the enhancement layer (700) encoding of the first image region (506). The enhancement layer (700) encoding of the first image region (506) is performed according to a first encoding priority. A second enhancement layer processing is performed on a second image region (508) of the image (500). Performing the second enhancement layer processing may include suppressing the acquisition of the enhancement layer (700) encoding of the second image region (508). Alternatively, performing the second enhancement layer processing includes acquiring the enhancement layer (700) encoding of the second image region (508). The enhancement layer (700) encoding of the second image region (508) is performed according to a second, lower encoding priority.

Description

Technical Field

[0001] This disclosure relates to the processing of augmentation layers for extended reality (XR) content. Background Technology

[0002] Images can be encoded to provide base layer encoding and enhancement layer encoding. Signals (e.g., as non-limiting examples, images, videos, stereo videos, depth maps, point clouds, etc.) can be encoded to provide base layer encoding and enhancement layer encoding, the base layer encoding corresponding to a second quality level of presentation of the signal, and the enhancement layer encoding providing data to be processed together with the base layer encoding to reconstruct the signal at a first (higher) quality level. The base layer encoding can be decoded and displayed without enhancement through the enhancement layer. Alternatively, the enhancement layer encoding can be decoded and used to enhance the visual quality of the decoded base layer.

[0003] Enhanced visual quality is particularly relevant (but not limited to) in XR scenarios, which include augmented reality (AR) and virtual reality (VR). Specifically, XR content is typically viewed on a screen (usually a high-resolution screen) very close to the viewer. Low visual quality can significantly degrade the viewer's experience.

[0004] However, in addition to visual quality, other factors can also have a significant impact on performance and viewer experience in XR scenarios.

[0005] For example, wireless communication technologies used for XR pixel streaming have limited bandwidth. For instance, the bandwidth might be 30 megabits per second (Mbps). For example, wireless communication technologies used for XR pixel streaming (e.g., where an XR display receives video generated in real-time by a remote device based on the viewer's motion and actions) have limited bandwidth. Furthermore, even with more bandwidth available, end-to-end latency and packet loss (both of which degrade the quality of experience) increase disproportionately with bandwidth. For example, in practical Wi-Fi and / or 5G scenarios, the bandwidth constraints to guarantee reliable low-latency transmission might be as low as 30 to 40 megabits per second (Mbps). However, the display resolution of XR headsets is increasing far faster than the bandwidth of wireless communication technologies. These limitations of wireless communication technologies mean that low bit rates must still be used in XR pixel streaming, even as display resolution continues to increase. State-of-the-art video encoding methods cannot guarantee high-quality, ultra-low-latency 8K video transmission under the practical limitations of wireless bandwidth. Summary of the Invention

[0006] The various aspects of this disclosure are set forth in the appended claims.

[0007] Further features and advantages will become apparent from the following description of preferred embodiments, which are given by way of example only with reference to the accompanying drawings. Attached Figure Description

[0008] Figure 1 A schematic block diagram of an example image processing system is shown;

[0009] Figure 2A and Figure 2B A schematic block diagram of another example of an image processing system is shown;

[0010] Figure 3 A schematic diagram showing an example of an image;

[0011] Figure 4 A schematic diagram showing an example of an enhancement layer is provided.

[0012] Figure 5 A schematic diagram showing another example of an image;

[0013] Figure 6 A schematic diagram showing another example of an enhancement layer;

[0014] Figure 7 A schematic diagram showing another example of an enhancement layer;

[0015] Figure 8 A schematic diagram showing another example of an enhancement layer;

[0016] Figure 9 A schematic diagram showing another example of an enhancement layer;

[0017] Figure 10 A schematic diagram showing another example of an image;

[0018] Figure 11 A schematic diagram showing an example of an image sequence; and

[0019] Figure 12 A schematic block diagram of an example device is shown. Detailed Implementation

[0020] refer to Figure 1 An example of a signal processing system 100 is shown. The signal processing system 100 is used to process signals. Examples of signal types include, but are not limited to, video signals, multi-view video signals, image signals, audio signals, volumetric signals (such as signals used for medical, scientific, or holographic imaging), or other multidimensional signals.

[0021] The signal processing system 100 includes a first device 102 and a second device 104. The first device 102 and the second device 104 may have a client-server relationship, wherein the first device 102 performs the functions of a server device, and the second device 104 performs the functions of a client device. The signal processing system 100 may include at least one additional device (not shown). The first device 102 and / or the second device 104 may contain one or more components. These components can be implemented in hardware and / or software. In the signal processing system 100, these components may be located in the same location or located remotely from each other. Examples of device types include, but are not limited to, computerized devices, handheld or laptop computers, tablet computers, mobile devices, game consoles, smart TVs, set-top boxes, XR headsets (including AR and / or VR headsets), etc.

[0022] The first device 102 is communicatively connected to the second device 104 via a data communication network 106. Examples of the data communication network 106 include, but are not limited to, the Internet, a local area network (LAN), and a wide area network (WAN). The first and / or second devices 102, 104 may establish wired and / or wireless connections with the data communication network 106.

[0023] In this example, the first device 102 includes an encoder 108. The encoder 108 is configured to encode data contained in and / or derived from a signal, hereinafter referred to as "signal data". For example, when the signal is a video signal, the encoder 108 is configured to encode video data. Video data comprises a sequence of multiple images or frames. In addition to encoding the signal data, the encoder 108 may perform one or more other functions. The encoder 108 can be implemented in a variety of different ways. For example, the encoder 108 can be implemented in hardware and / or software. The encoder 108 may encode metadata associated with the signal. The first device 102 may use one or more encoders 108.

[0024] Although in this example, the first device 102 includes the encoder 108, in other examples, the first device 102 is separate from the encoder 108. In such examples, the first device 102 is communicatively connected to the encoder 108. The first device 102 may be implemented as one or more software functions and / or hardware modules.

[0025] In this example, the second device 104 includes a decoder 110. The decoder 110 is configured to decode signal data. In addition to decoding signal data, the decoder 110 may perform one or more other functions. The decoder 110 can be implemented in a variety of different ways. For example, the decoder 110 can be implemented in hardware and / or software. The decoder 110 can decode metadata associated with the signal. The second device 104 may use one or more decoders 110.

[0026] Although in this example the second device 104 includes the decoder 110, in other examples the second device 104 is separate from the decoder 110. In such examples, the second device 104 is communicatively connected to the decoder 110. The second device 104 may be implemented as one or more software functions and / or hardware modules.

[0027] Encoder 108 encodes the signal data and transmits the encoded signal data to decoder 110 via data communication network 106. Decoder 110 decodes the received encoded signal data and generates decoded signal data. Decoder 110 may output the decoded signal data or output data derived using the decoded signal data. For example, decoder 110 may output such data for display on one or more display devices associated with second device 104. The one or more display devices may be components of second device 104 or may be otherwise associated with second device 104. The one or more display devices are operable to display XR content and may therefore be referred to as XR display devices.

[0028] In some examples described herein, encoder 108 transmits to decoder 110 a representation of the signal at a given quality level, along with information that decoder 110 can use to reconstruct a representation of part or all of the signal at one or more higher quality levels. This information may be referred to as “reconstruction data.” In some examples, “reconstruction” of the representation involves obtaining a representation that is not an exact copy of the original representation. The degree to which this representation is identical to the original representation can depend on various factors, including but not limited to the quantization level. A representation of the signal at a given quality level can be considered as a presentation, version, or depiction of the data contained within the signal at that given quality level. In some examples, the reconstruction data is included in the signal data encoded by encoder 108 and transmitted to decoder 110. For example, the reconstruction data may be in the form of metadata. In some examples, the reconstruction data is encoded and transmitted separately from the signal data.

[0029] The information used by decoder 110 to reconstruct a representation of the signal at one or more higher quality levels may include residual data, as described in more detail below. Residual data is an example of reconstructed data. The information used by decoder 110 to reconstruct a representation of the signal at one or more higher quality levels may also include configuration data related to the processing of the residual data. The configuration data may indicate how the residual data has been processed by encoder 108 and / or how the residual data will be processed by decoder 110. The configuration data may be communicated to decoder 110, for example, in the form of metadata.

[0030] The first and / or second devices 102, 104 may be configured to perform some or all of the techniques described herein. A computer program may be configured to perform some or all of the techniques described herein.

[0031] refer to Figure 2A and Figure 2B An example of a signal processing system 200 is schematically illustrated. The signal processing system 200 includes a first device 202 and a second device 204. In this example, the first device 202 includes an encoder, and the second device 204 includes a decoder. However, as mentioned above, in other examples, the encoder is not included in the first device 202, and / or the decoder is not included in the second device 204. In each of the first device 202 and the second device 204, items are shown at two logical levels. These two levels are separated by dashed lines. Items at the first (highest) level relate to data at a first quality level. Items at the second (lowest) level relate to data at a second quality level. The first quality level is higher than the second quality level. The first and second quality levels relate to a hierarchical structure with multiple quality levels. In some examples, the hierarchical structure includes more than two quality levels. In such examples, the first device 202 and the second device 204 may include more than two distinct levels. Figure 2A and 2B One or more other layers may exist above and / or below the layer depicted in the text. As described herein, in some cases, quality levels may correspond to different spatial resolutions.

[0032] First refer to Figure 2AThe first device 202 obtains a first representation 206 of an image at a first quality level. A representation of a given image is a representation of data contained in the image. The image may be a given frame of a video. The first representation 206 of the image at the first quality level will be referred to hereinafter as "input data" because, in this example, the first representation is data provided as input to the encoder in the first device 202. The first device 202 may receive the input data 206. For example, the first device 202 may receive the input data 206 from at least one other device. The first device 202 may be configured to receive consecutive portions of the input data 206, such as consecutive frames of a video, and perform the operations described herein for each consecutive frame. For example, the video may include frames F1, F2, ... F... T Furthermore, the first device 202 can process each of these frames in sequence.

[0033] The first device 202 derives data 212 based on input data 206. In this example, data 212 based on input data 206 is a representation 212 of the image at a second lower quality example level. In this example, data 212 is derived by performing a downsampling operation on input data 206, and will therefore be referred to hereinafter as "downsampled data". In other examples, data 212 is derived by performing an operation other than downsampling on input data 206, or data 212 is the same as input data 206 (i.e., input data 206 is unprocessed, e.g., unsampled).

[0034] In this example, the downsampled data 212 is processed to generate processed data 213 of a second quality level. In other examples, the downsampled data 212 is not processed at the second quality level. Thus, the first device 202 can generate data of a second quality level, wherein the data of the second quality level includes either the downsampled data 212 or the processed data 213.

[0035] In some examples, generating processed data 213 involves encoding downsampled data 212. This encoding can occur within the first device 202, or the first device 202 can output processed data 213 to an external encoder. Encoding the downsampled data 212 produces an encoded image of a second quality level. The first device 202 can output the encoded image, for example, for transmission to a second device 204. A series of encoded images output for transmission to the second device 204 (e.g., forming an encoded video) can be referred to as a “base” stream or a “base” layer. As explained above, the encoded image can be generated by an encoder separate from the first device 202, rather than within the first device 202. The encoded image can be part of HEVC-encoded video, H.264 (AVC) video, or other video. Generating processed data 213 can, for example, involve generating consecutive video frames output by a separate encoder, such as an HEVC video encoder or an H.264 (AVC) video encoder. Intermediate datasets used for generating processed data 213 can include the output of such an encoder, rather than any intermediate data generated by that separate encoder.

[0036] Generating processed data 213 of a second quality level may further involve decoding the encoded image of the second quality level. Decoding operations can be performed to simulate decoding operations at the second device 204, as will become apparent below. Decoding the encoded image produces a decoded image of the second quality level. In some examples, the first device 202 decodes the encoded image of the second quality level to produce a decoded image of the second quality level. In other examples, the first device 202 receives the decoded image of the second quality level, for example, from an encoder and / or decoder separate from the first device 202. The encoded image can be decoded using an HEVC decoder (or, for example, an H.264, AVC decoder). Decoding by a separate decoder may include inputting encoded video (such as an encoded data stream configured for transmission to a remote decoder) into a separate black-box decoder implemented with the first device 202 to generate successive decoded video frames. The processed data 213 may therefore include video data frames generated via a complex nonlinear encoding and decoding process, where the encoding and decoding process may involve modeling spatiotemporal correlations according to a specific encoding standard such as HEVC (or H.264, AVC). However, since the output of any encoder is fed into the corresponding decoder, this complexity is effectively hidden from the first device 202.

[0037] In the example, generating the processed data 213 of the second quality level further involves obtaining correction data based on a comparison between the downsampled data 212 and the decoded image obtained by the first device 202, such as based on the difference between the downsampled data 212 and the decoded image. The correction data can be used to correct encoder-decoder errors (which may also be referred to as “encode-decoder errors”), i.e., errors introduced during the encoding and decoding of the downsampled data 212. In some examples, the first device 202 outputs correction data (e.g., for transmission to the second device 204) along with the encoded signal. This allows the receiver to correct for encoder-decoder errors introduced during the encoding and decoding of the downsampled data 212. This correction data can also be referred to as the “first enhancement” stream. Since the correction data can be based on the difference between the downsampled data 212 and the decoded image, it can be considered a form of residual data (e.g., a different residual dataset than described later below). A residual data item can be referred to as a “residual”.

[0038] In some examples, generating the processed data 213 at the second quality level further involves correcting the decoded image using correction data. For example, the correction data output for transmission can be placed in a form suitable for combination with the decoded image and then added to the decoded image. This can be performed frame by frame. In other examples, the first device 202 uses downsampled data 212 without using correction data to correct the decoded image. For example, in some cases, only the encoded and then decoded data can be used, while in others, encoding and decoding can be replaced by other processing.

[0039] In some examples, generating processed data 213 involves performing one or more operations in addition to the encoding, decoding, acquisition, and correction actions described above.

[0040] The first device 202 acquires data 214 based on data of a second quality level. As indicated above, the data of the second quality level may include processed data 213, or downsampled data 212 if it was not processed at a lower level. As described above, in some cases, processed data 213 may include a reconstructed video stream (e.g., from an encoder-decoder operation) corrected using correction data. Figure 2A and Figure 2BIn the example, data 214 is a second representation of an image at a first quality level, and the first representation of the image at the first quality level is input data 206. The second representation of the first quality level can be considered a preliminary or predicted representation of the image at the first quality level. In this example, the first device 202 derives data 214 by performing an upsampling operation on the data at the second quality level. Data 214 will be referred to hereinafter as "upsampled data". However, in other examples, one or more other operations may be used to derive data 214, for example, where data 212 is not derived by downsampling the input data 206.

[0041] Input data 206 and upsampled data 214 are used to obtain residual data 216. Residual data 216 is associated with an image. Residual data 216 can be in the form of a set of residual elements, which may be referred to as a "residual frame" or "residual image." Residual elements may be referred to as "residuals." Residual elements in the residual element set 216 can be associated with corresponding image elements in the input data 206. An example of an image element is a pixel.

[0042] In this example, the given residual element is obtained by subtracting the value of the image element in the upsampled data 214 from the value of the corresponding image element in the input data 206. Thus, the residual data 216 can be used in conjunction with the upsampled data 214 to reconstruct the input data 206. The residual data 216 can also be referred to as "reconstructed data" or "enhanced data." In one case, the residual data 216 can form part of a "second enhancement" stream. The residual data 216 can therefore generate an upsampler-downsampler asymmetry. Upsampler-downsampler asymmetry can also be referred to as "upsample-downsample asymmetry," "upsample-downsample" asymmetry, etc. In some examples, the enhancement data (e.g., the residual data 216) can include information to guide the post-processing operations to be performed on the reconstructed image to obtain a first quality level of rendering.

[0043] The first device 202 acquires configuration data relating to the processing of residual data 216. The configuration data indicates how the residual data 216 has been processed and / or generated by the first device 202, and / or how the residual data 216 will be processed by the second device 204. The configuration data may include a set of configuration parameters. The configuration data can be used to control how the second device 204 processes the data and / or how it reconstructs the input data 206 using the residual data 216. The configuration data may relate to one or more characteristics of the residual data 216. The configuration data may relate to one or more characteristics of the input data 206. Different configuration data may result in different processing of the residual data 216 and / or different processing using the residual data 216. The configuration data can therefore be used to reconstruct the input data 206 using the residual data 216. As described below, in some cases, the configuration data may also relate to the correction data described herein.

[0044] In this example, the first device 202 transmits data based on downsampled data 212, data based on residual data 216, and configuration data (or data based on configuration data) to the second device 204 so that the second device 204 can reconstruct the input data 206.

[0045] See now Figure 2B The second device 204 receives data 220 based on (e.g., derived from) downsampled data 212. The second device 204 also receives data based on residual data 216. For example, the second device 204 may receive a “base” stream (data 220), a “first enhancement stream” (any correction data), and a “second enhancement stream” (residual data 216). The base stream may be referred to as the “base layer.” The first and / or second enhancement streams may be referred to as “enhancement layers,” and / or may be included in the “enhancement layers.” The second device 204 also receives configuration data relating to the processing of the residual data 216. The data 220 based on downsampled data 212 may be downsampled data 212 itself, processed data 213, or data derived from downsampled data 212 or processed data 213. The data based on residual data 216 may be residual data 216 itself or data derived from residual data 216.

[0046] In some examples, the received data 220 includes processed data 213, which may include encoded image and / or correction data at a second quality level. In some examples, for instance, where the first device 202 has processed the downsampled data 212 to generate processed data 213, the second device 204 processes the received data 220 to generate processed data 222. Such processing performed by the second device 204 may include decoding the encoded image (e.g., forming part of a “base” encoded video stream) to produce a decoded image at a second quality level. In some examples, the processing performed by the second device 204 includes correcting the decoded image using acquired correction data. Therefore, the processed data 222 may include frames of corrected data at a second quality level. In some examples, the encoded image at a second quality level is decoded by a decoder separate from the second device 204. The encoded image at a second quality level may be decoded using an HEVC decoder (or, for example, an H.264, AVC decoder).

[0047] In other examples, the received data 220 includes downsampled data 212 but does not include processed data 213. In some such examples, the second device 204 does not process the received data 220 to generate processed data 222.

[0048] The second device 204 uses data of the second quality level to derive upsampled data 214. As indicated above, the data of the second quality level may include processed data 222, or received data 220 without the second device 204 processing the received data 220 at the second quality level. Upsampled data 214 is a preliminary representation of the image of the first quality level. Upsampled data 214 can be derived by performing an upsampling operation on the data of the second quality level.

[0049] The second device 204 acquires residual data 216. The residual data 216 can be used with the upsampled data 214 to reconstruct the input data 206. The residual data 216 indicates a comparison between the input data 206 and the upsampled data 214.

[0050] The second device 204 also acquires configuration data relating to the processing of the residual data 216. The configuration data can be used by the second device 204 to reconstruct the input data 206. For example, the configuration data may indicate features or attributes relating to the residual data 216 that affect how the residual data 216 will be used and / or processed, or whether the residual data 216 will be used at all. In some examples, the configuration data includes the residual data 216.

[0051] Several considerations exist regarding this type of processing. One such consideration is the amount of information generated, stored, transmitted, and / or processed. The more information used, the greater the amount of resources potentially involved in processing it. Examples of such resources include transmission resources, storage resources, and processing resources. Some signal processing techniques allow for the use of relatively small amounts of information. This can reduce the amount of data transmitted via data communication network 106. This saving can be particularly meaningful when the data involves high-quality video data, where the amount of information transmitted can be especially high.

[0052] Another factor to consider is latency. Complex image processing can introduce latency, which can negatively impact performance.

[0053] Another factor to consider is the limitations of wireless communication technology.

[0054] Other considerations include the decoder's ability to perform image reconstruction accurately, reliably, and / or efficiently. Accurate and reliable image reconstruction can affect the final visual quality of the displayed image, and therefore, the observer's engagement with the image and / or the video containing it. This can be particularly meaningful for XR. Efficient reconstruction is especially effective for mobile computing devices that can be readily used in XR applications.

[0055] refer to Figure 3 The diagram illustrates an example of image 300. Image 300 can be a standalone image or a frame from video content. Image 300 can, for example, represent a frame from a movie, television program, sporting event, etc.

[0056] Basic layer processing can be performed on image 300. This basic layer processing produces basic layer encoding of image 300. For example, the basic layer encoding can correspond to the reference above. Figure 2A The processed data described is 213.

[0057] The base layer code can be obtained in various ways. In some examples, it is obtained by generating the base layer code. In some examples, it is obtained by instructing the generation of the base layer code. In some examples, it is obtained by configuring an encoder to generate the base layer code. In some examples, it is obtained by receiving the base layer code. In some examples, it is obtained by retrieving the base layer code from memory. In other examples, the base layer code can be obtained in yet another way.

[0058] Enhancement layer processing can be performed on image 300. This enhancement layer processing results in enhancement layer coding of image 300. For example, the enhancement layer coding may correspond to the reference above. Figure 2A The described correction data and / or residual data 216.

[0059] Enhancement layer codes can be obtained in various ways. In some examples, they are obtained by generating enhancement layer codes. In some examples, they are obtained by instructing the generation of enhancement layer codes. In some examples, they are obtained by configuring an encoder to generate enhancement layer codes. In some examples, they are obtained by receiving enhancement layer codes. In some examples, they are obtained by retrieving enhancement layer codes from memory. In other examples, enhancement layer codes can be obtained in yet another way.

[0060] More generally, references to acquiring encoding herein include generating encoding, instructing the generation of encoding, configuring an encoder to enable the encoder to generate encoding, receiving encoding, retrieving encoding from memory, and acquiring encoding in any other manner.

[0061] The base layer code can be generated by the base layer encoder. The enhancement layer code can be generated by the enhancement layer encoder. The base layer encoder can be different from the enhancement layer encoder. In particular, the base layer encoder and the enhancement layer encoder can use different codecs.

[0062] In some examples, enhancement layer residuals arise from upsampler-downsampler asymmetry. Such residuals can correspond to those mentioned in the reference above. Figure 2A The residual data described is 216.

[0063] In some examples, enhancement layer residuals produce autoencoder-decoder errors. Such residuals can correspond to those mentioned in the reference above. Figure 2A The described correction data.

[0064] In some examples, the base layer encoding and enhancement layer encoding are output. In some examples, the base layer encoding and enhancement layer encoding are stored.

[0065] The base layer encoding and enhancement layer encoding can be decoded, combined, and displayed, as shown in the reference above. Figure 2B Described.

[0066] refer to Figure 4 A schematic diagram illustrating an example of enhancement layer 400 is shown. In this example, enhancement layer 400 is based on... Figure 3 The example image 300 shown is generated.

[0067] In this example, enhancement layer 400 includes residual set 402.

[0068] Residuals 402 can take many different forms. Residuals are more generally referred to as "correction data" or "enhancing data".

[0069] In some examples, Residual 402 includes MPEG-5 Part 2 Low Complexity Enhanced Video Coding (LCEVC) residuals, which will be described in more detail below. In some examples, Residual 402 includes residuals used in scalable codecs, such as SMPTE ST 2117-1 (often referred to as "VC-6"). In some examples, the scalable codec is a multi-layered, scalable codec. Therefore, the encoding and / or decoding described herein may be derived from a multi-layered, scalable codec. In some examples, Residual 402 includes residuals used in scalable MPEG codecs such as scalable AVC (SVC), scalable HEVC (SHVC), scalable AV1, and scalable VVC.

[0070] There are no restrictions on the location where the residual 402 can be positioned within the enhancement layer 400. As explained above, the enhancement layer 400 can be used to enhance the visual quality of the base layer encoding of the example image 300. In particular, the residual 402 enhances the visual quality at its respective locations. Thus, visual quality enhancement can be easily provided across the entire image 300.

[0071] In many practical situations, the entire image 300 is visible to the observer. For example, if image 300 is a frame from a movie, sporting event, television program, video game, etc., the entire image 300 is visible to the observer. The observer may be looking at any part of image 300. Furthermore, the observer can focus on different parts of the video at different times. For example, an observer may focus on an object in the center of image 300 during a portion of the video, and on an object in the upper right corner of image 300 during a later portion of the video, and so on. The foveal fixation characteristic of human vision makes it impossible to view the entire field of view at maximum retinal resolution at any given time. Different observers can focus on different parts of any given image 300. For example, some observers may focus on an object in the lower left corner of image 300, while other users may focus on a different object in the upper right corner of image 300. High visual quality across the entire image thus enhances the observer's experience and perceived visual quality. For example, by default, residuals can be applied to any and every location within the image. By further example, by default, in order to satisfy all possible focal areas, residuals can be applied to any and every location of the image.

[0072] refer to Figure 5 The diagram shows another example of image 500.

[0073] Various items are shown schematically as part of image 500. However, this is merely for the purpose of aiding in the understanding of this disclosure. In practice, these items may not be visible in image 500 itself, and image 500 may represent other content.

[0074] In this example, image 500 includes XR content and is used with an XR display device. As indicated above, the XR content may include AR content and / or VR content. For example, the XR display device may correspond to the reference above. Figure 1 The second device 104 is described. As will be described above, the base layer code and enhancement layer code generated using image 500 can be output for display on the XR display device.

[0075] In this example, image 500 is a stereoscopic image. In this example, the image includes a left image portion 502 and a right image portion 504.

[0076] In this specific example, image 500 is rectangular and has a width and height In this specific example, the left image portion 502 and the right image portion 504 are also rectangular, with a width and height However, in other examples, image 500 may have a different shape. Additionally, in other examples, the left image portion 502 and the right image portion 504 may have different proportions and / or shapes and / or sizes. In some examples, image 500 includes a different number of image portions.

[0077] Figure 5 The example image 500 shown includes a first image region 506. In this example, the first image region 502 includes a first sub-region 506-1 and a second sub-region 506-2. In this example, the first sub-region 506-1 is in the left image portion 502, and the second sub-region 506-2 is in the right image portion 504.

[0078] As will be described in more detail below, in some examples, the first image region 506 is the foveal fixation region because the first image region 506 corresponds to the foveal fixation of the observed eye, or at least the region expected to be the foveal fixation of the observed eye. The region outside the foveal region can be the peripheral region, that is, the region corresponding to the observer's peripheral vision, or at least the region expected to be the observer's peripheral vision.

[0079] In this example, the first sub-region 506-1 is centered in the left image portion 502, and the second sub-region 506-2 is centered in the right image portion 504. Specifically, neither the first sub-region 506-1 nor the second sub-region 506-2 is centered in the image 500 itself.

[0080] In this example, the first sub-region 506-1 is rectangular and has a width and height ,in and In this example, the second sub-region 506-2 is rectangular, with a width... and height ,in and In this example, and In other examples, the first sub-region 506-1 and / or the second sub-region 506-2 may have different shapes and / or sizes. In some examples, the first sub-region and / or the second sub-region 506-1 is rectangular and has a height smaller than its width. In other examples, the sub-regions have more rounded shapes, approximating the typical shape of small glasses. In still other examples, the shapes of the sub-regions may be discontinuous to allow for the consistent presence of detail in specific areas of the viewport frequently viewed via rapid eye movements (e.g., a HUD overlay with key game statistics).

[0081] In this example, the first sub-region 506-1 and the second sub-region 506-2 do not overlap. The first region 506 can therefore be considered discontinuous.

[0082] Figure 5 The example image 500 shown also includes a second image region 508. In this example, the second image region 508 surrounds the first image region 506. In this example, the second image region 508 is the complement of the first image region 506 in image 500.

[0083] In this example, the first image region 506 corresponds to the observer's expected field of view (FOV) of image 500. The expected FOV is the FOV that the observer expects to have. The FOV described in this example and elsewhere (e.g., the first image region 506) may correspond to the observer's (e.g., expected) "fovea" view region of image 500, which may also be referred to as the fovea (e.g., expected) FOV. The fovea FOV is the region in an image that includes the area that the observer can see with maximum detail. The expected fovea FOV is the region in an image that includes the area that the observer expects to see with maximum detail. For example, the FOV may include regions at the observer's view / visual edge (regions that cannot be seen with high detail because they are at the view edge). The foveal FOV may exclude such marginal regions (“marginal regions are areas located at the edge of the observer’s view / visual field of vision”). The foveal FOV may (e.g., only) include the area of ​​the image that the observer can see with high detail (because such areas are closer to the observer’s gaze or expected gaze than marginal regions). The expected FOV may differ from the observer’s actual FOV. For example, last-minute (e.g., rapid) eye movements can alter the observer’s actual FOV (compared to the expected one), for example, based on the observer’s recent FOV, eye-tracking data, or otherwise. The expected FOV may also be referred to as the “possible,” “predicted,” “central,” or “focal” FOV, and for this reason, it may often be larger than the actual area that the observer can see with maximum detail.

[0084] In this example, the observer of image 500 is the user of the XR device.

[0085] More specifically, in this example, the first sub-region 506-1 corresponds to the left-eye view of the observer of image 500, and the second sub-region 506-2 corresponds to the right-eye view of the observer of image 500. In this example, part or all of the second image region 508 is outside the observer's field of view (FOV) of image 500.

[0086] A boundary 510 exists between the left image portion 502 and the right image portion 504. In this example, the second image region 508 includes the boundary 510.

[0087] Boundary 510 can appear as a dark line in image 500. For example, the left-eye view (corresponding to the left image portion 502) can transition from a relatively light color on its left to a relatively dark color on its right. The right-eye view (corresponding to the right image portion 504) can similarly transition from a relatively light color on its left to a relatively dark color on its right. The boundary 510 between the left image portion 502 and the right image portion 504 will therefore appear as a vertical dark line. In some examples (e.g., including but not limited to...) Figure 5(For example), multi-view video coding can be used in the base layer (e.g., MV-HEVC instead of HEVC), so that the left-eye view and the right-eye view are encoded with separate image layers.

[0088] In some examples, the second image region 508 includes at least a portion of the edge 512 of the image 500. The edge 512 may be referred to as a “boundary,” “perimeter,” etc. The edge may similarly include dark lines or dark areas, especially in the case of XR content.

[0089] In some examples, the positions of the first image region 506 and the second image region 508 are independent of the pixel values ​​of the pixels in image 500, and therefore can be said to be pixel value independent. Thus, such examples differ from those in which pixel analysis is performed to determine the positions of the first image region 506 and the second image region 508.

[0090] In some examples, the positions of the first image region 506 and the second image region 508 are independent of the genre of the XR content of image 500, and can therefore be considered type-independent. Such examples are therefore different from those in which knowledge of the genre of the XR content affects the positions of the first image region 506 and the second image region 508.

[0091] Therefore, in some examples, the positions of the first image region 506 and the second image region 508 are content-independent.

[0092] The locations of the alternative first image region 506 and the second image region 508 are independent of pixel values ​​and / or XR content type. Pre-analysis can be performed on image 500 to determine the locations of the first image region 506 and / or the second image region 508. However, such pre-analysis introduces latency. For XR content, reduced latency generally improves performance and user experience.

[0093] refer to Figure 6 This diagram illustrates another example of enhancement layer 600. In this example, enhancement layer 600 is based on... Figure 5 The example image 500 shown is generated. Figure 6 Reference numerals used for the same or similar features in the figures Figure 5 The reference numerals used in the figures are the same, but 100 has been added.

[0094] Figure 6 The example enhancement layer 600 shown generally corresponds to Figure 4 The example enhancement layer 400 shown is located in the residual set of the residuals in Figure 6 The corresponding position in the text.

[0095] However, Figure 6A first subset 614 of the residuals (represented by residual 614) is shown in the first region 606 of the enhancement layer 600, and a second subset 616 of the residuals (represented by residual 616) is shown in the second region 608 of the enhancement layer 600. The first region 606 and the second region 608 of the enhancement layer 600 correspond, in terms of their positions, to the first image region 506 and the second image region 508 of the image 500. The residuals in the first region 606 of the enhancement layer 600 can therefore be said to correspond to the first image region 506 of the image 500, and the residuals in the second region 608 of the enhancement layer 600 can therefore be said to correspond to the second image region 508 of the image 500.

[0096] As explained above, for many types of images, the entire image may be visible to the observer. However, the entire image may not be visible, such as for XR images. Furthermore, the entire image may not be visible at its maximum detail, as with XR images.

[0097] The examples that will now be described can improve bitrate control, performance, and / or visual quality in XR.

[0098] Such examples selectively enhance XR content. For instance, a portion of the XR content may be enhanced, while another portion may be enhanced to a lesser extent, or even not at all. This selective enhancement is particularly (but not exclusively) effective for XR content where not all of the XR content provided to the XR display device is within the user's field of view (FOV). XR content outside the user's FOV can be enhanced less, or even not at all, than XR content within the user's FOV. In some examples, the XR display device can still display XR content outside the user's FOV based on the base layer encoding of such content, even if the corresponding enhancement layer encoding for such content is unavailable. Lower visual quality of XR content within the FOV has a more significant impact on the user experience than outside the FOV. Therefore, visual quality outside the FOV can be sacrificed to at least some extent relative to visual quality within the FOV without significantly negatively impacting performance and user experience. This can improve bitrate control because less enhancement layer data can be delivered to the XR display device. Sometimes it is preferable to sacrifice the size of the field of view (FOV) rather than the quality enhancement within the FOV, allowing the rate controller to dynamically adjust the FOV size frame by frame. In some examples, some or all of the savings gained by reducing enhancements outside the FOV and / or reducing the size of the FOV are used to further enhance the visual quality within the FOV.

[0099] More specifically, in one possible implementation, the residual is applied to the object in some frames but not in others. This can improve bitrate control but may affect the visual quality associated with the relevant object.

[0100] Furthermore, as explained above, stereoscopic images can have artificially created sharp boundaries. Significant residuals can be used to attempt to make these boundaries perfect lines. However, when viewing stereoscopic images on XR display devices, precisely defined boundary lines do not improve visual quality because the boundaries are outside the viewer's field of view (FOV). This applies accordingly to dark lines or dark areas at the edges of the image.

[0101] The example described below addresses this issue by either not generating residuals outside the observer's FOV, or by generating lower-priority residuals outside the observer's FOV. Boundaries will fall outside the observer's FOV, and therefore will either not consume residuals at the boundaries, or will consume smaller residuals at the boundaries.

[0102] In some examples, residuals for only regions of the XR image within the observer's field of view (FOV) are generated, encoded, quantized to non-zero values, and / or transmitted. Such regions may be located at the center of each half of the stereoscopic image. In other examples, the location of such regions may vary frame-by-frame based on real-time information received about the observer's eye movements, and the size of such regions may similarly vary frame-by-frame based on rate control decisions informed by data including available bitrate and image complexity metrics.

[0103] When residuals (such as these residuals) are quantized, the quantization step size can be based on the size of the region within the observer's FOV, the target bit rate, and / or one or more other factors.

[0104] refer to Figure 7 The diagram shows another example of the enhancement layer 700.

[0105] Figure 7 The example enhancement layer 700 shown generally corresponds to Figure 6 Example enhancement layer 600 is shown. Figure 7 Reference numerals used for the same or similar features in the figures Figure 6 The reference numerals used in the figures are the same, but 100 has been added.

[0106] However, Figure 7 This demonstrates that the first subset 714 of the residual is processed in a different manner than the second subset 716 of the residual. This is in Figure 7 The image is depicted using different background patterns of the first subset 714 and the second subset 716 of the residual.

[0107] In more detail, Figure 5 The base layer encoding of the example image 500 shown is obtained.

[0108] In some examples, the base layer coding of image 500 has consistent coding quality across the image.

[0109] In other examples, the base layer coding of image 500 includes a first base layer coding portion corresponding to a first image region 506 and a second base layer coding portion corresponding to a second image region 508. The first base layer coding portion has a first coding quality, and the second base layer coding portion has a second, lower coding quality. When the coding is decoded, the lower coding quality can result in lower visual quality.

[0110] A first enhancement layer processing can be performed on a first image region 506 of image 500. Performing the first enhancement layer processing includes obtaining the enhancement layer 700 encoding of the first image region 506. The enhancement layer 700 encoding of the first image region 506 is performed according to a first encoding priority. In this example, a first subset 714 of the residual, which is encoded in the first region 706 of the enhancement layer 700, is encoded according to the first encoding priority.

[0111] Encoding priority can correspond to visual importance. For example, higher encoding priority can correspond to higher visual importance, and lower encoding priority can correspond to lower visual importance. A higher encoding priority relative to another code with a lower encoding priority can correspond to a higher priority ordering for a code. A higher encoding priority relative to another code with a lower encoding priority can correspond to a lower quantization step size for a code. A higher encoding priority relative to another code with a lower encoding priority can correspond to using a codec with higher visual quality for a code.

[0112] A second enhancement layer is performed on the second image region 508 of image 500.

[0113] The first enhancement layer processing and the second enhancement layer processing can be part of a single enhancement coding process.

[0114] In some examples, performing the second enhancement layer processing includes suppressing the encoding of the enhancement layer 700 that acquires the second image region 508. Thus, although the second subset 716 of the residual in the second region 708 of the enhancement layer 700... Figure 7 The image is depicted, but only for ease of understanding. The second subset 716 of the residual may not have been generated at all.

[0115] Suppressing the encoding of the enhancement layer 700 that acquires the second image region 508 can be implemented as an encoding condition. This encoding condition can be that the residual encoding logic does not operate for residuals with specific locations.

[0116] This differs from the priority graph implementation, which will be described in more detail below, in which residuals are generated for all locations, but residuals at specific locations are de-prioritized and quantized to zero.

[0117] In this regard, reference is made to WO 2020 / 188229, the contents of which are incorporated herein by reference. WO 2020 / 188229 describes, for example, an encoder analyzing the input video and preparing a set of residual masks for each frame of the video. WO 2020 / 188229 also describes that, instead of performing such pre-analysis on the input video by the encoder, a central server can propose residual weighted masks based on the type of the input video. The central server can provide a set of residual weighted masks covering different types (e.g., sports, movies, news, etc.). The residual masks can then prioritize regions of the image that require detail. For example, the residual masks can thus prioritize regions of the image that require detail in response to regions of view that can be characterized as unaffected by head movement (e.g., graphic overlays). However, such pre-analysis and / or type dependence can introduce processing latency.

[0118] The techniques described in this article also differ from WO 2018 / 015764, the contents of which are incorporated herein by reference. In WO2018 / 015764, one or more enhancement layers can be provided to the decoder for the entire image, and the decoder can choose which parts of the enhancement layers to use.

[0119] In some examples, suppressing the encoding of the enhancement layer 700 for acquiring the second image region 508 includes causing the enhancement layer mask to be applied to the second image region 508.

[0120] In such examples, the enhancement layer mask may correspond in shape to the second image region 508. In such examples, the enhancement layer mask is applied to one or more lower-priority image regions, such that one or more higher-priority image regions are not masked and are outside the enhancement layer mask. The residuals corresponding to the masked lower-priority image regions and the unmasked higher-priority image regions can be processed accordingly.

[0121] In other examples, suppressing the encoding of the enhancement layer 700 for acquiring the second image region 508 includes causing an enhancement layer mask to be applied to the first image region 506. In such examples, the enhancement layer mask may correspond in shape to the first image region 506. In such examples, the enhancement layer mask is applied to one or more higher priority image regions such that one or more lower priority image regions are not masked and are outside the enhancement layer mask. The residuals corresponding to the masked higher priority image regions and the unmasked lower priority image regions can be processed accordingly.

[0122] In some examples, the enhancement layer mask has a specific geometry. For instance, the enhancement layer mask can be rectangular (such as a square), circular, elliptical, or it can have more complex shapes. Less complex shapes can have improved computational performance compared to more complex shapes. However, less complex shapes can be less accurately similar to the field of view (FOV) compared to more complex shapes.

[0123] The geometry and / or size of the shape can be determined based on one or more factors. Examples of such factors include, but are not limited to, available bandwidth, target bitrate, scene complexity, and visual quality targets. Therefore, bitrate control can be provided by changing the geometry and / or size of the shape. Consequently, bitrate control can be provided by (e.g., dynamically) changing the geometry and / or size of the shape and enhancing the quantization of the data, thereby effectively balancing the size of the field of view (FOV) with the quality level within the FOV for a given bitrate target.

[0124] More specifically, the shape can be predetermined to change in response to a predetermined change in conditions. These conditions can be communication network conditions. For example, the size of the shape can be reduced in response to a predetermined reduction in bandwidth. This predetermined reduction in bandwidth could correspond to a bandwidth drop below a threshold bandwidth value, or otherwise. Advantageously, this allows the rate controller to quickly reduce the bit rate allocation to enhanced data without necessarily producing visible degradation in the central region of the field of view (FOV). By another example, the size of the shape can be increased in response to a predetermined increase in bandwidth. This predetermined increase in bandwidth could correspond to a bandwidth rise above a threshold bandwidth value, or otherwise.

[0125] The size, shape, and / or position of the enhancement layer mask can be fixed. For example, the size, shape, and / or position of the enhancement layer mask can be the same for different images in an image sequence.

[0126] Alternatively, the size, shape, and / or position of the enhancement layer mask can be dynamic. For example, the size, shape, and / or position of the enhancement layer mask can differ relative to different images in an image sequence.

[0127] The enhancement layer mask can be thought of as a foveal priority map. The foveal priority map provides a graph of different priorities (in this example, encoding priorities) relative to foveal imaging. In foveal imaging, as explained in this paper, visual quality varies throughout the image. The foveal priority map can also be referred to as the foveal priority map region.

[0128] In a specific example, the enhancement layer mask (in other words, the concave priority map region) is a rectangular shape whose height is less than its width. The concave priority map region can therefore be shaped to be more "horizontal" or "lateral" in orientation than more "vertical" or "vertical," and thus can resemble typical eyeglasses. The concave priority map region can therefore be considered rectangular and lateral in orientation. In such examples, the long side of the rectangle can be aligned with (e.g., parallel to) the top and bottom edges of the image frame and / or view. In such examples, the short side of the rectangle can be aligned with (e.g., parallel to) the left and right edges of the image frame and / or view. This can be particularly effective in the context of XR content. This is because, when the observer's head is not moved, their eyes typically cover a greater distance on the left-right (horizontal) axis compared to the up-down (vertical) axis. This is because observers typically read a scene like reading a book, moving their eyes from left to right, while they typically move their heads up and down rather than straining their pupils to look significantly upward or downward.

[0129] Therefore, the enhancement layer mask can be thought of as a foveal priority map that prioritizes one or more regions of the field of view. For example, for each of the left and right eye views, the foveal priority map could correspond to a rectangle, a circle, or any other shape at the center of the field of view. In other examples where XR devices can provide eye-tracking data, the encoder dynamically adjusts the position of the foveal priority map based on the specific location where the observer is focusing from time to time.

[0130] In some examples, performing the second enhancement layer processing includes obtaining the enhancement layer 700 encoding of the second image region 508. In such examples, the enhancement layer 700 encoding of the second image region 508 follows a second encoding priority. The second encoding priority is lower than the first encoding priority.

[0131] In some examples, encoding the enhancement layer 700 for the second image region 508 includes obtaining a residual set 716 corresponding to the second image region 508, wherein the residual set 716 corresponding to the second image region 508 is quantized to zero. In this example, the residual set 716 corresponding to the second image region 508 is a second subset 716 of the residuals. Thus, a second subset 716 of the residuals can be generated, and its value can be set to zero during quantization.

[0132] In some examples, encoding the enhancement layer 700 for the first image region 506 includes obtaining a residual set 714 corresponding to the first image region 506, wherein the residual set 714 corresponding to the first image region 506 is quantized using a first quantization step size. In this example, the residual set 714 corresponding to the first image region 506 is a first subset 714 of the residuals. In such examples, encoding the enhancement layer 700 for the second image region 508 includes obtaining a residual set 716 corresponding to the second image region 508, wherein the residual set 716 corresponding to the second image region 508 is quantized using a second higher quantization step size. In this example, the residual set 716 corresponding to the second image region 508 is a second subset 716 of the residuals.

[0133] In some examples, the rate controller is configured to determine one or more coding properties of the residual in a given image region and to determine one or more geometric properties of the region in which the residual will be used. For example, the rate controller may be configured to determine the quantization step size (an example of a coding parameter) of the residual in a given image region and the size (an example of a geometric property) of the region in which the residual will be used. In other words, the rate controller may be configured to make trade-offs between the size and / or shape of the concave region and one or more coding parameters. An example of such coding parameters is the quantization step size of the residual in the concave region. Another example of such coding parameters is the quantization step size of the residual outside the concave region. Another example of such coding parameters is the bit rate allocation to the base layer. Another example of such coding parameters is the bit rate allocation to the enhancement layer. Another example of such coding parameters is a measure of image complexity.

[0134] Therefore, the residual set corresponding to the first image region can be quantized using a given quantization step size.

[0135] A given quantization step size can be based on the size of a first image region and / or a target bit rate. Some examples involve determining the given quantization step size based on the size of the first image region and / or the target bit rate. Some examples involve quantizing the residual set corresponding to the first image region according to the determined given quantization step size.

[0136] The size of the first image region can be based on a given quantization step size and / or target bit rate. Some examples include determining the size of the first image region based on a given quantization step size and / or target bit rate. Some examples include selecting, setting, and / or adjusting the size of the first image region based on a given quantization step size and / or target bit rate.

[0137] Therefore, measures (such as methods, systems, and computer programs) are provided for processing images including XR content and for use in XR display devices. The base layer encoding of the image is obtained. A first enhancement layer processing may be performed on a first image region of the image. Performing the first enhancement layer processing may include a rate controller determining one or more encoding attributes of a first residual set in the first image region of the image, and determining one or more attributes (e.g., geometric attributes) of a first region in which the first residual set will be used. Such determination may include a rate controller weighing one or more encoding attributes against one or more geometric attributes. A second enhancement layer processing may be performed on a second image region of the image. Performing the second enhancement layer processing may include a rate controller determining one or more encoding attributes of a second residual set in the second image region of the image, and determining one or more geometric attributes of a second region in which the second residual set will be used.

[0138] Therefore, the same high-quality residual can be used for a relatively small area, rather than degrading visual quality by using (lower-quality) residuals throughout the image. The quality of the residual can correspond to its quantization level, such that higher quantization corresponds to lower quality, and lower quantization corresponds to higher quality. In particular, lower-quality residuals can be more heavily quantized compared to higher-quality residuals. If the fovea region reaches a given (small) size, and if the rate controller determines that the size of the residual still needs to be reduced (e.g., by increasing the quantization step size), the rate controller can adjust this tradeoff so that it seeks to maintain a constant size of the fovea region and increase the quantization step size to achieve the target bit rate. In this way, the bit rate controller can dynamically trade off the visual quality provided in the fovea region against the actual size of the fovea region on a frame-by-frame basis.

[0139] Without performing the second enhancement layer processing described above, the second subset 716 of the residuals would actually be wasted. This is because they are not important, or at least have limited importance, in terms of the visual quality perceived by the observer.

[0140] The first and second enhancement layer processing can be performed as part of a single enhancement layer processing procedure. For example, the first and second enhancement layer encodings can be generated by the same enhancement layer encoder.

[0141] In some examples, these codes are output for display on XR display devices. In some examples, these codes are stored.

[0142] These encodings include base layer encoding and enhancement layer 700 encoding for the first image region 506. When the enhancement layer 700 encoding for the second image region 508 is acquired, these encodings may include that enhancement layer 700 encoding for the second image region 508. These encodings may be output to an XR display device for display on the XR display device, or they may be output to another device for display on the XR display device. Such output may include streaming. An example of such another device is a computing device paired with an XR display device in the form of an XR head-mounted device.

[0143] In some examples, the base layer encoding of image 500 can be decoded and / or displayed independently of the enhancement layer 700 encoding.

[0144] In terms of decoder-side processing, the base layer code of image 500 can be obtained. The enhancement layer code of the first image region 506 (which follows a first encoding priority) can also be obtained. The enhancement layer code of the second image region 508 (which follows a second encoding priority) can be obtained, or the enhancement layer code of the second image region 508 can be omitted. This acquisition or omission can be performed by the XR display device. The acquired codes can be used to reconstruct image 500. The reconstructed image can then be displayed on the XR display device.

[0145] In some examples, residuals 714 and 716 are intra-frame independent and can therefore be described as “intra-frame independent.” Thus, for such residuals 714 and 716, there may be no “intra-frame” dependency. This allows for the use of residuals not being used or residuals quantized to zero for a portion of enhancement layer 700. Codecs that do not perform intra-frame coding (e.g., intra-frame predictive coding), such as LCEVC, can be used to generate such intra-frame independent residuals because the codec is suitable for frame-by-frame varying concave priority regions. LCEVC is described in WO 2020 / 188273 (PCT / GB2020 / 050695) and WO 2019 / 111010 (PCT / GB2018 / 053552), the entire contents of which are incorporated herein by reference. Furthermore, LCEVC can also allow for the streaming of XR content with ultra-low latency within the limitations of wireless communication technologies (such as those described above). If residuals 714 and 716 are intra-frame dependent, decoding the values ​​of some residuals may require the values ​​of others. For example, decoding the values ​​of some residuals 714 in the first region 706 of enhancement layer 700 may require the values ​​of some residuals 716 in the second region 708 of enhancement layer 700. If the residuals 716 in the second region 708 of enhancement layer 700 are not provided at all, or are requantized, enhancement layer 700 may be unusable, or at least less effective. For example, if the residuals 716 in the second region 708 of enhancement layer 700 are requantized, intra-frame prediction will be less accurate, but usable predictions will still be provided. However, if the residuals 716 in the second region 708 of enhancement layer 700 are completely omitted or entirely quantized to zero, these residuals will not be available for intra-frame prediction.

[0146] refer to Figure 8 The diagram shows another example of the enhancement layer 800.

[0147] Figure 8 The example enhancement layer 800 shown generally corresponds to Figure 7 Example enhancement layer 700 is shown. Figure 8 Reference numerals used for the same or similar features in the figures Figure 7 The reference numerals used in the figures are the same, but 100 has been added.

[0148] However, Figure 8 Only the first subset 814 of the residuals is described.

[0149] from Figure 8 As can be seen, compared to Figure 7 The example enhancement layer 700 shown can use fewer residuals without significantly or at all negatively impacting visual quality.

[0150] refer to Figure 9The diagram shows another example of the enhancement layer 900.

[0151] Figure 9 The example enhancement layer 900 shown generally corresponds to Figure 7 and Figure 8 The example enhancement layers shown are 700 and 800. Figure 9 Reference numerals used for the same or similar features in the figures Figure 7 and Figure 8 The reference numerals used are the same, but 200 and 100 are added respectively.

[0152] However, in this example, the residuals that would normally be generated outside the observer's FOV are actually redistributed inside the observer's FOV.

[0153] More specifically, in some examples, data-saving measures are determined. The data-saving metric represents the amount of data saved by performing a second enhancement layer process on the second image region 508 (e.g., compared to performing a first enhancement layer process on the second image region 508). Some or all of the data saved in this way can be allocated to the first enhancement layer process on the first image region 506.

[0154] In particular, Figure 7 Six residuals 714 in the first region 706 of the reinforcement layer 700 and seven residuals 716 in the second region 708 of the reinforcement layer 700 are shown, for a total of thirteen residuals.

[0155] In comparison, Figure 9 The diagram shows thirteen residuals 918 in the first region 906 of the reinforcement layer 900 and zero residuals in the second region 908 of the reinforcement layer 900, which is still a total of thirteen residuals.

[0156] By limiting the enhancement layer regions that allow residuals (or at least residuals with limited quantization), the visual quality of the most important regions of the image can be improved, potentially without increasing the amount of data transmitted, compared to allowing residuals in any part of the enhancement layer.

[0157] By placing the residual primarily or solely in the concave region, it is possible to significantly improve quality at a relatively low bit rate. This can be particularly effective in bandwidth-constrained systems, such as those using low-bandwidth wireless communication technologies (especially for very high-resolution signals). For example, if the residual is confined to an area of ​​one-fifth of the total screen area, an equivalent quality enhancement of up to five times can be achieved without increasing the bit rate.

[0158] refer to Figure 10 The diagram shows another example of image 1000.

[0159] Figure 10 The example image 1000 shown generally corresponds to Figure 5 Example image 500 is shown. Figure 10 Reference numerals used for the same or similar features in the figures Figure 5 The reference numerals used in the figures are the same, but 500 has been added.

[0160] However, in this example, image 1000 includes a third image region 1014.

[0161] In this example, the third image region 1014 includes a first sub-region 1014-1 and a second sub-region 1014-2. In this example, the first sub-region 1014-1 is in the left image portion 1002, and the second sub-region 1014-2 is in the right image portion 1004.

[0162] In this example, the first sub-region 1014-1 is centered in the left image portion 1002, and the second sub-region 1014-2 is centered in the right image portion 1004. Therefore, the first sub-region 1014-1 can be said to be in the center of the left image portion 1002, and the second sub-region 1014-2 can be said to be in the center of the right image portion 1004.

[0163] In this example, the first sub-region 1014-1 is rectangular and has a width and height ,in and In this example, the second sub-region 1014-2 is rectangular and has a width and height ,in and In this example, and In other examples, the first sub-region 1014-1 and / or the second sub-region 1014-2 may have different shapes and / or sizes.

[0164] In this example, the first sub-region 1014-1 and the second sub-region 1014-2 of the third image region 1014 do not overlap. The third image region 1014 can therefore be considered discontinuous.

[0165] In some examples, a third enhancement layer process is performed on the third image region 1014. Performing the third enhancement layer process may include obtaining the enhancement layer code for the third image region 1014 according to a third coding priority. The third coding priority may differ from the one described above. Figure 7 The first and second encoding priorities are described.

[0166] In some examples, the third image region 1014 is located between the first image region 1006 and the second image region 1008. In some examples, the third coding priority is located between the first coding priority and the second coding priority.

[0167] The third image region 1014 may correspond to the periphery of the user's field of view (FOV) of the XR display device.

[0168] Therefore, more than two encoding priorities can be used. Some residuals may not be used at all. In other words, such residuals may not be generated at all, or they may be generated but quantized to zero. These will be the lowest priority residuals. Residuals in the peripheral regions may have intermediate priorities and may be encoded but requantized. Residuals in the center field of view may have the highest priority.

[0169] In some examples, the residual is therefore also used outside the concave priority map corresponding to the first image region 1006. However, in some examples, such residuals are restricted to only one or more static regions of the screen. Examples of static regions of the screen are regions where the visual content does not change between at least some consecutive images in an image sequence. This allows static graphics overlays to have consistently good quality, for example, while using a very limited bitrate.

[0170] Reference Figure 11 The diagram shows an example of image sequence 1100. Figure 11 Some of the figure marks used are similar to Figure 5 The same reference numerals are used for the same or similar features in the figures.

[0171] In this example, the base layer encoding and enhancement layer encoding of the additional images are created, for example, for output to be displayed on an XR display device. In this example, there are three additional images 1102, 1104, and 1106. However, in practice, there can be significantly more than three additional images in image sequence 1100. The additional images 1102, 1104, and 1106 are... Figure 5 As shown in and Figure 11 The example image 500 reproduced in the image is an additional image. Image 500 and additional images 1102, 1104, and 1106 are part of an image sequence 1100 that includes XR content for an XR display device. Image sequence 1100 may correspond to an XR stream. The XR stream may include a stereoscopic XR stream.

[0172] In some examples, the first and second image regions (506 and 508 for image 500) have fixed positions on image sequence 1110. This reduces or even eliminates the need for resources to determine the positions of these regions on image sequence 1110. Fixed positions may also be referred to as "static," "set," or "set frame" positions. Using a fixed priority map thus reduces or eliminates the computational costs associated with positioning. The first and second image regions (506 and 508 for image 500) may have fixed shapes and / or sizes on image sequence 1110.

[0173] In some examples, the first and second image regions (506 and 508 for image 500) have dynamic positions, shapes, and / or sizes on image sequence 1110. The dynamic positions, shapes, and / or sizes can be determined in a variety of different ways.

[0174] In some examples, eye-tracking data is received from an XR display device, and the position, shape, and / or size of a first image region and a second image region (506, 508 for image 500) are determined based on the eye-tracking data. The eye-tracking data may also be referred to as "eye-tracking information." As explained above, the XR display device may be or may include an XR head-mounted device. In such examples, the first and second image regions (506, 508 for image 500) may be centered or may not be centered.

[0175] Therefore, the encoder can receive feedback from the XR display device, such that the first image region and the second image region (506, 508 for image 500) have a position, shape and / or size associated with eye tracking.

[0176] Therefore, the residual can be placed within a selected portion of the entire screen. This allows detail to be focused where it is most important, following the natural foveal fixation characteristics of the human visual system. Real-time feedback can be provided to the encoder using eye-tracking data. This introduces a delay. However, this delay can be tolerable, such as 50-100 ms. The encoder can thus precisely place detail around the area the observer is actually looking at, rather than assuming the observer is roughly looking at the center of the FOV.

[0177] LCEVC is particularly effective in this regard. For example, LCEVC features enable this level of control in a way that is impossible, or at least not very effective, for at least some single-layer codecs. Therefore, in the context of XR pixel streaming, using eye-tracking data to inform the LCEVC encoder to select a specific combination of residual sets is particularly effective.

[0178] In the absence of eye tracking and eye-tracking data, the fovea region can be positioned at a fixed center in a frame sequence. The fovea region can function like a pair of (variable-sized) glasses on the observer's face. When eye tracking is available, the encoder can receive real-time information about where the observer is looking. The encoder can then center the fovea region there.

[0179] In some examples, the shape of the fovea region is at least partially based on the location the observer is currently viewing. In other examples, the shape of the fovea region is alternatively or additionally at least partially based on the location the observer is likely to view next. Such predictions can be based on eye movements. For example, predictions can be based on recent eye movements. This can allow for tens of milliseconds of latency, thus balancing end-to-end system delays.

[0180] The examples described in this article can be used in various use cases of XR pixel streaming. Examples of such use cases include, but are not limited to, game streaming, six degrees of freedom (6DoF) narratives with pre-rendered view areas, and design reviews of digital twins and / or digital avatars.

[0181] refer to Figure 12 A schematic block diagram of an example of device 1200 is shown.

[0182] In one example, device 1200 includes an encoder. In another example, device 1200 includes a decoder. In still other examples, device 1200 includes neither an encoder nor a decoder, but is configured to communicate with an encoder and / or a decoder.

[0183] Examples of device 1200 include, but are not limited to, mobile computers, personal computer systems, wireless devices, base stations, telephone equipment, desktop computers, laptop computers, notebook computers, netbook computers, mainframe computers, handheld computers, workstations, network computers, application servers, storage devices, consumer electronic devices such as cameras, camcorders, mobile devices, video game consoles, handheld video game devices, XR headsets, or generally any type of computing or electronic device.

[0184] In this example, device 1200 includes one or more processors 1201 configured to process information and / or instructions. The one or more processors 1201 may include a central processing unit (CPU). The one or more processors 1201 are coupled to a bus 1202. Operations performed by the one or more processors 1201 may be implemented by hardware and / or software. The one or more processors 1201 may include multiple co-located processors or multiple distributed processors.

[0185] In this example, device 1200 includes computer-available volatile memory 1203 configured to store information and / or instructions for one or more processors 1201. Computer-available volatile memory 1203 is coupled to bus 1202. Computer-available volatile memory 1203 may include random access memory (RAM).

[0186] In this example, device 1200 includes computer-available non-volatile memory 1204 configured to store information and / or instructions for one or more processors 1201. Computer-available non-volatile memory 1204 is coupled to bus 1202. Computer-available non-volatile memory 1204 may include read-only memory (ROM).

[0187] In this example, device 1200 includes one or more data storage units 1205 configured to store information and / or instructions. The one or more data storage units 1205 are coupled to bus 1202. The one or more data storage units 1205 may include, for example, a magnetic disk or optical disk and a disk drive or solid-state drive (SSD).

[0188] In this example, device 1200 includes one or more input / output (I / O) devices 1206 configured to communicate information to and / or from one or more processors 1201. The one or more I / O devices 1206 are coupled to a bus 1202. The one or more I / O devices 1206 may include at least one network interface. This at least one network interface enables device 1200 to communicate via one or more data communication networks. Examples of data communication networks include, but are not limited to, the Internet and local area networks (LANs). The one or more I / O devices 1206 enable a user to provide input to device 1200 via one or more input devices (not shown). The one or more input devices may include, for example, a remote control, one or more physical buttons, etc. The one or more I / O devices 1206 enable information to be provided to the user via one or more output devices (not shown). The one or more output devices may include, for example, a display screen.

[0189] Various other entities are shown for device 1200. For example, when present, operating system 1207, image processing module 1208, one or more further modules 1209, and data 1210 are shown residing in one or a combination of computer-usable volatile memory 1203, computer-usable non-volatile memory 1204, and one or more data storage units 1205. Data signal processing module 1208 may be implemented by computer program code stored in a memory location within computer-usable non-volatile memory 1204, in a computer-readable storage medium within one or more data storage units 1205, and / or other tangible computer-readable storage media. Examples of tangible computer-readable storage media include, but are not limited to: optical media (e.g., CD-ROM, DVD-ROM, or Blu-ray), flash memory cards, floppy disks, or hard disks, or any other medium capable of storing computer-readable instructions such as firmware or microcode in at least one ROM or RAM or programmable ROM (PROM) chip, or as an application-specific integrated circuit (ASIC).

[0190] The apparatus 1200 may therefore include a data signal processing module 1208 executable by one or more processors 1201. The data signal processing module 1208 can be configured to include instructions to implement at least some of the operations described herein. During operation, one or more processors 1201 initiate, run, execute, interpret, or otherwise perform the instructions in the signal processing module 1208.

[0191] Although at least some aspects of the examples described herein with reference to the accompanying drawings encompass computer processes executed in a processing system or processor, the examples described herein also extend to computer programs, such as computer programs on or within a carrier, which are suitable for putting the examples into practice. A carrier can be any entity or device capable of carrying a program.

[0192] It should be understood that device 1200 may include, compared to Figure 12 The components described in the text include more, fewer, and / or different components.

[0193] The device 1200 can be located in a single location or distributed across multiple locations. Such locations can be local or remote.

[0194] The techniques described herein can be implemented using software or hardware, or a combination of both. They may include devices configured to implement and / or support any or all of the techniques described herein.

[0195] A bitstream can be provided. The bitstream may include a base layer encoding of an image, which includes XR content and is intended for an XR display device. The bitstream may include enhancement layer encoding of a first image region of the image. The enhancement layer encoding of the first image region may have already been generated according to a first encoding priority. In some examples, the bitstream includes enhancement layer encoding of a second image region, wherein the enhancement layer encoding of the second image region has been generated according to a second, lower encoding priority. In other examples, the bitstream does not include enhancement layer encoding of the second image region.

[0196] In a specific example, codec features carrying local user data are used for post-processing operations. This codec can be a multi-layered, hierarchical codec. LCEVC is an example of such a codec with these features. For example, the user data may specify one or more specific regions, to which the decoder will apply one or more given post-processing operations. For example, deringing and / or AI-based post-processing operations can be costly for an entire 8K pixel display. However, applying post-processing operations (such as deringing and / or AI-based post-processing operations) to a given foveal priority region during decoding can be readily feasible. The foveal priority region signaled to the decoder for post-processing can be different from the foveal priority region used by the encoder to prioritize residuals. Furthermore, as explained above, the foveal priority region used for prioritizing residuals may or may not depend on real-time eye-tracking data received from the head-mounted device.

[0197] Therefore, measures (such as methods, systems, and computer programs) are provided for processing images, including XR content, for use in XR display devices. The base layer encoding of the image is obtained. Enhancement layer processing can be performed on a first image region of the image. Performing such enhancement layer processing may include obtaining the enhancement layer encoding of the first image region. The enhancement layer encoding of the first image region can be prioritized. Such prioritization can be for enhancement layer encodings of one or more further image regions of the image. For example, the first image region may correspond to a first fovea priority region. User data can be generated. The user data may specify a second image region of the image, to which the decoder will apply one or more given post-processing operations. The first and second image regions of the image may be different from each other. The user data can be signaled to the decoder. The techniques described herein can be applied to depth maps. For cases where the combination of video and depth is used for depth-based reprojection and thus for frame rate adaptation to the display frame rate, the depth map can be sent along with the video data. For the depth map, and in a manner similar to that described above for video data, residuals can be prioritized for fovea regions. Such prioritization can involve sending only the residuals for the fovea region, using a smaller quantization step size for the fovea region relative to the quantization step size for the residuals outside the fovea region, or other methods. The fovea region of the depth map may or may not depend on eye-tracking data. In this way, accurate depth processing can be provided where it is most efficient, without wasting a large amount of bitrate. In a specific example, the depth map can be sent at a low resolution. The boundaries of the depth map can be aligned to the object using any available method based on luminance or chrominance information. The depth map can then be corrected using the residual data for the fovea region that the user is viewing. This makes the depth map more accurate in the fovea region.

[0198] Therefore, the base layer encoding of the depth map can be obtained. The depth map can indicate depth information associated with the image. A first enhancement layer processing can be performed on a first region of the depth map. Performing the first enhancement layer processing on the first region of the depth map can include obtaining the enhancement layer encoding of the first region of the depth map. The enhancement layer encoding of the first region of the depth map can be performed according to a first depth map encoding priority. A second enhancement layer processing can be performed on a second region of the depth map. Performing the second enhancement layer processing on the second region of the depth map can include: suppressing the acquisition of the enhancement layer encoding of the second region of the depth map; or acquiring the enhancement layer encoding of the second region of the depth map, wherein the enhancement layer encoding of the second region of the depth map is performed according to a second lower depth map encoding priority. The first region of the depth map can correspond to a first image region of the image. The second region of the depth map can correspond to a second image region of the image.

[0199] In the example described above, the image is a stereoscopic image. However, other types of images with left and right image portions can be used.

[0200] For example, one of the left and right image portions may include a view of the scene, and the other may include a corresponding depth map. For example, sacrificing some quality of the depth map for the sake of the view of the scene may be acceptable.

[0201] In another example, one of the left and right image portions may include a view of the sporting event from a first angle, and the other of the left and right image portions may include a view of the sporting event from a second, different angle. Sacrificing some quality in one of the views of the sporting event (e.g., zooming out) may be acceptable for the purpose of providing another view of the sporting event.

[0202] Instead of the example techniques described above, a base layer encoding of the image can be obtained, where a first image region of the image is undistorted and a second image region of the image is distorted. This distortion may involve removing some pixel values ​​from the second image region. The decoder can then interpolate the remaining pixel values ​​in the second image region to attempt to reconstruct the removed pixel values. In cases where the second image region is outside the observer's field of view (FOV), the artifacts resulting from self-interpolation may be tolerable in the second image region. However, such interpolation-based techniques can introduce latency that can negatively impact performance in XR scenarios.

[0203] The example techniques described above can be combined with the methods described in (PCT / GB2017 / 05214, WO2018 / 015764). For example, a device (e.g., a user device, an XR display device, an XR head-mounted device, etc.) can be configured to receive one or more of a base layer and an enhancement layer. As described above, the enhancement layer can be acquired, for example, by performing a first enhancement layer processing on a first image region of an image, wherein performing the first enhancement layer processing includes acquiring enhancement layer encoding of the first image region according to a first encoding priority; and performing a second enhancement layer processing on a second image region of an image, wherein performing the second enhancement layer processing includes: suppressing the acquisition of enhancement layer encoding of the second image region; or acquiring enhancement layer encoding of the second image region according to a second lower encoding priority. The device can be configured to acquire the decoding of the base coding layer. The device can be configured to acquire the decoding of the enhancement coding layer. The device can be configured to combine the decoding of the base coding layer and the decoded enhancement coding layer to acquire an enhanced signal (e.g., an image). Specifically, the device can be configured to decode and / or apply (e.g., only) a portion of the received enhancement layer. This portion can be smaller than the whole. The device can be configured to decode only a portion of the received enhancement layer. The device can be configured to apply only a portion of the decoded enhancement layer. This portion can correspond to the area of ​​the image that the observer is viewing and / or expects to view. This portion can correspond to the area of ​​the image that the observer is viewing and / or expects to view during decoding and / or display. In other words, this portion can be defined by the FOV and / or the concave FOV and / or the expected FOV and / or the expected concave FOV.

[0204] In this way, efficient decoding and efficient encoding based on real-time eye-tracking information are combined (by encoding only enhancements within FOV, foveal FOV, expected FOV, and expected foveal FOV).

[0205] This synergistic combination can be regarded as a "two-pass" process: in the first pass, as described above, residual coding is performed only for a (usually central) portion (assumed to be X%), which can be executed using independent residual tiles. Then, as the second pass, for example, based on eye tracking information, only the residuals (e.g., tiles) that are in focus (e.g., FOV, foveal FOV, expected FOV, expected foveal FOV, etc.) are transmitted and / or decoded and / or applied at a given time. In this way, there is only encoded enhancement data (i.e., enhancement layer) for X% of the image, and only Y% of the image has the enhancement data decoded and / or applied thereto (Y<X). This can be used to achieve full resolution of the display screen only where the user is viewing, without encoding and transmitting full-resolution data over the entire image. Since there are some errors in eye tracking prediction, enhancement data can be generated for X% of the image (centered around the predicted eye direction), however, only a part of this enhancement data can be decoded and / or applied, and this part can be centered around the updated predicted eye direction (i.e., gaze). The updated predicted eye direction is calculated at a time closer to the display time than when the enhancement data was encoded.

[0206] The following clauses are described. The described clauses can be combined with any content in the specification and any content in the claims. In particular, the target region of the clauses may correspond to a subset of the first image region. The enhancement data of the clauses may correspond to enhancement layer coding.

[0207] 1. A decoder device, configured to:

[0208] receive data that can be used to generate data for representing a data signal at a first quality level;

[0209] receive enhancement data that can be used to generate, based on the data for representing the data signal at the first quality level, data for representing the data signal at a second quality level, the second quality level being higher than the first quality level;

[0210] generate data for a target region representing the data signal at a target quality level using a selected part of the received enhancement data, the selected part of the received enhancement data being associated with the target region of the data signal, the target quality level being higher than the first quality level; and

[0211] generate data for further regions of the data signal representing quality levels lower than the target quality level.

[0212] 2. The decoder device according to clause 1, wherein the target quality level is the second quality level.

[0213] 3. The decoder device according to Clause 1, wherein the target quality level is between a first quality level and a second quality level.

[0214] 4. The decoder device according to any one of Clauses 1 to 3, wherein the quality level of a further region of the data signal is between the first quality level and the target quality level.

[0215] 5. The decoder device according to any one of Clauses 1 to 4, wherein the decoder device is configured to generate data for representing a further region of the data signal using a selected further portion of the enhancement data, the selected further portion of the enhancement data being associated with the further region of the data signal.

[0216] 6. The decoder device according to any one of Clauses 1 to 3, wherein the quality level of a further region of the data signal is a first quality level.

[0217] 7. The decoder device according to any one of Clauses 1 to 6, wherein a further region of the data signal at least partially surrounds the target region of the data signal.

[0218] 8. The decoder device according to any one of Clauses 1 to 7, wherein the decoder device is configured to identify a target region of a data signal using data associated with a field of view and / or data associated with one or more gaze positions.

[0219] 9. The decoder device as described in Clause 8, wherein the decoder device is configured to select a target region of a data signal so that it is within the field of view.

[0220] 10. The decoder device according to Clause 8 or Clause 9, wherein the decoder device is configured to select at least a portion of a further region of the data signal to place it within the field of view.

[0221] 11. The decoder device according to any one of Clauses 8 to 10, wherein the decoder device is configured to:

[0222] Monitoring the field of view and / or one or more fixation locations at multiple time points; and

[0223] The location of the target region in the subsequent data signal is determined by the field of view and / or one or more gaze positions at the time points associated with the subsequent data signal.

[0224] 12. The decoder device according to Clause 11, wherein a target region of a data signal is associated with one or more data signal blocks, and a further region of the data signal is associated with one or more data signal blocks, wherein at least one of the data signal blocks associated with the further region of the data signal is within the target region in a subsequent data signal.

[0225] 13. The decoder device according to any one of Clauses 1 to 12, wherein the decoder device is configured to:

[0226] Generate data for representing the target area of ​​the data signal at a first quality level; and

[0227] Use the generated data to represent the target area of ​​the data signal at the first quality level to generate data to represent the target area of ​​the data signal at the target quality level.

[0228] 14. The decoder device according to any one of Clauses 1 to 13, the decoder device being configured to operate in accordance with a layered data signal processing arrangement comprising at least one layer having a set of sublayers, each sublayer being associated with a respective quality level.

[0229] 15. The decoder device as described in Clause 14, wherein the decoder device is configured to not use the enhancement data associated with at least one of the sublayers in a first operating mode of the decoder device.

[0230] 16. The decoder device according to Clause 14, wherein the decoder device is configured to generate data for a target region representing a data signal of a target quality level using enhanced data associated with all sublayers in a second operating mode of the decoder device.

[0231] 17. The decoder device according to any one of Clauses 1 to 16, the decoder device being configured to operate in accordance with a layered data signal processing arrangement comprising a first layer having a first set of sublayers and a second layer having a set of sublayers, each sublayer being associated with a respective quality level.

[0232] 18. The decoder device according to Clause 17, wherein the decoder device is configured to generate data for a target region representing a data signal of a target quality level using enhanced data associated with at least one sublayer of the first and second layers in a third operating mode of the decoder device.

[0233] 19. The decoder device according to any one of Clauses 14 to 18, wherein the first quality level corresponds to the quality level associated with the lowest sublayer in the hierarchical data signal processing arrangement.

[0234] 20. The decoder device according to any one of Clauses 14 to 19, wherein the second quality level corresponds to the quality level associated with the highest sublayer in the hierarchical data signal processing arrangement.

[0235] 21. The decoder device according to any one of Clauses 14 to 20, wherein the target quality level corresponds to the quality level associated with the sublayer between the highest and lowest sublayers in the hierarchical data signal processing arrangement.

[0236] 22. The decoder device according to any one of Clauses 1 to 21, wherein the data signal includes image data.

[0237] 23. The decoder device according to any one of Clauses 1 to 22, wherein the data signal includes video data.

[0238] 24. The decoder device according to any one of Clauses 1 to 23, wherein the decoder device is configured to identify a target area of ​​a data signal.

[0239] 25. The decoder device according to any one of Clauses 1 to 24, wherein the decoder device is configured to select a portion of enhanced data associated with a target region of the data signal.

[0240] 26. The decoder device according to any one of Clauses 1 to 25, wherein the decoder device is included in a virtual reality device, a medical imaging device, a machine vision device and / or a mobile communication device.

[0241] 27. A virtual reality device comprising a decoder device according to any one of Clauses 1 to 25.

[0242] 28. A medical imaging apparatus comprising a decoder device according to any one of Clauses 1 to 25.

[0243] 29. A machine vision device comprising a decoder device according to any one of Clauses 1 to 25.

[0244] 30. A mobile communication device comprising a decoder device according to any one of Clauses 1 to 25.

[0245] 31. A method comprising, at a decoder device:

[0246] Receive data that can be used to generate a data signal representing a first quality level;

[0247] Receive enhanced data, which can be used to generate data for representing data signals at a second quality level, based on data signals used to represent data at a first quality level, the second quality level being higher than the first quality level;

[0248] Data for representing a target region of a data signal at a target quality level is generated using a selected portion of the received augmented data, wherein the selected portion of the received augmented data is associated with the target region of the data signal, and the target quality level is higher than a first quality level; and

[0249] Generate data for a further region of the data signal to represent the data signal at a quality level lower than the target quality level.

[0250] 32. The method described in Clause 31, wherein the target quality level is the second quality level.

[0251] 33. The method according to Clause 31, wherein the target quality level is between a first quality level and a second quality level.

[0252] 34. The method according to any one of Clauses 31 to 33, wherein the quality level of a further region of the data signal is between the first quality level and the target quality level.

[0253] 35. The method according to any one of Clauses 31 to 34, wherein generating data for representing a further region of the data signal includes using selected further portions of enhanced data associated with the further region of the data signal.

[0254] 36. The method according to any one of Clauses 31 to 33, wherein the quality level of a further region of the data signal is a first quality level.

[0255] 37. The method according to any one of Clauses 31 to 36, wherein a further region of the data signal at least partially surrounds the target region.

[0256] 38. The method according to any one of Clauses 31 to 37, the method comprising using data associated with a field of view and / or data associated with one or more gaze positions to identify a target region of a data signal.

[0257] 39. The method according to Clause 38, the method comprising selecting a target region of a data signal so that it is within the field of view.

[0258] 40. The method according to Clause 38 or Clause 39, the method comprising selecting at least a portion of a further region of the data signal so that it is within the field of view.

[0259] 41. The method according to any one of Clauses 38 to 40, the method comprising:

[0260] Monitoring the field of view and / or one or more fixation locations at multiple time points; and

[0261] The location of the target region in the subsequent data signal is determined by the field of view and / or one or more gaze positions at the time points associated with the subsequent data signal.

[0262] 42. The method according to Clause 41, wherein a target region of the data signal is associated with one or more data signal blocks, and a further region of the data signal is associated with one or more data signal blocks, wherein at least one of the data signal blocks associated with the further region of the data signal is within the target region in a subsequent data signal.

[0263] 43. The method according to any one of Clauses 31 to 42, the method comprising:

[0264] Generate data for representing the target area of ​​the data signal at a first quality level; and

[0265] Use the generated data to represent the target area of ​​the data signal at the first quality level to generate data to represent the target area of ​​the data signal at the target quality level.

[0266] 44. The method according to any one of Clauses 31 to 43, the method comprising operating in accordance with a hierarchical data signal processing arrangement, the hierarchical data signal processing arrangement comprising at least one layer having a set of sublayers, each sublayer being associated with a respective quality level.

[0267] 45. The method according to Clause 44, wherein the method includes, in a first operating mode of the decoder device, not using enhancement data associated with at least one of the sublayers.

[0268] 46. ​​The method according to Clause 44, wherein the method includes, in a second operating mode of the decoder device, using enhanced data associated with all sublayers to generate data for a target region of a data signal representing a target quality level.

[0269] 47. The method according to any one of Clauses 31 to 46, the method comprising operating in accordance with a hierarchical data signal processing arrangement, the hierarchical data signal processing arrangement comprising a first layer having a first set of sublayers and a second layer having a set of sublayers, each sublayer being associated with a respective quality level.

[0270] 48. The method according to Clause 47, the method comprising, in a third operating mode of the decoder device, using enhanced data associated with at least one sublayer of the first and second layers to generate data for a target region of a data signal representing a target quality level.

[0271] 49. The method according to any one of Clauses 44 to 48, wherein the first quality level corresponds to the quality level associated with the lowest sublayer in the hierarchical data signal processing arrangement.

[0272] 50. The method according to any one of Clauses 44 to 49, wherein the second quality level corresponds to the quality level associated with the highest sublayer in the hierarchical data signal processing arrangement.

[0273] 51. The method according to any one of Clauses 44 to 50, wherein the target quality level corresponds to the quality level associated with the sublayer between the highest and lowest sublayers in the hierarchical data signal processing arrangement.

[0274] 52. The method according to any one of Clauses 31 to 51, wherein the data signal includes image data.

[0275] 53. The method according to any one of Clauses 31 to 52, wherein the data signal includes video data.

[0276] 54. The method according to any one of Clauses 31 to 53, the method comprising identifying a target area of ​​a data signal.

[0277] 55. The method according to any one of Clauses 31 to 54, the method comprising selecting a portion of augmented data associated with a target area of ​​the data signal.

[0278] 56. A computer program comprising instructions that, when executed, cause a decoder device to perform the method according to any one of clauses 31 to 55.

[0279] 57. A computer-readable medium comprising a computer program as described in Clause 56.

[0280] It should be understood that any feature described with respect to any embodiment may be used alone or in combination with other described features, and may also be used in combination with one or more features of any other embodiment, or in any combination of any other embodiment. Furthermore, equivalents and modifications not described above may be employed without departing from the scope of the invention as defined by the appended claims.

Claims

1. A method for processing an image, the image including extended reality (XR) content and for use in an XR display device, the method comprising: Obtain the base layer encoding of the image; Perform a first enhancement layer processing on a first image region of the image, wherein performing the first enhancement layer processing includes obtaining the enhancement layer code of the first image region, and the enhancement layer code of the first image region is performed according to a first encoding priority; as well as Perform a second enhancement layer process on a second image region of the image, wherein performing the second enhancement layer process includes: Suppress the encoding of the enhancement layer used to acquire the second image region; or The enhancement layer encoding of the second image region is obtained, and the enhancement layer encoding of the second image region is performed according to a second lower encoding priority.

2. The method according to claim 1, wherein the positions of the first image region and the second image region are independent of the pixel values ​​of the pixels of the image.

3. The method according to claim 1 or 2, comprising: The output is encoded for display on the XR display device.

4. The method according to any one of claims 1 to 3, comprising: The base layer encoding and enhancement layer encoding of the output supplementary image are used for display on the XR display device. The image and the supplementary image are part of an image sequence comprising XR content for the XR display device. The first image region and the second image region have fixed positions, shapes and / or sizes in the image sequence.

5. The method according to any one of claims 1 to 3, comprising: The base layer encoding and enhancement layer encoding of the output supplementary image are used for display on the XR display device. The image and the supplementary image are part of an image sequence comprising XR content for the XR display device. The first image region and the second image region have dynamic positions, shapes and / or sizes on the image sequence.

6. The method of claim 5, further comprising: Receive eye-tracking data from the XR display device; as well as The position, shape, and / or size of the first image region and the second image region are determined based on the eye-tracking data.

7. The method according to any one of claims 1 to 6, wherein the first image region has a rectangular shape, and wherein the height of the rectangular shape of the first image region is less than the width of the rectangular shape.

8. The method of claim 7, wherein the first image region has a lateral orientation.

9. The method according to any one of claims 1 to 8, wherein performing the second enhancement layer processing comprises: Suppress the encoding of the enhancement layer used to acquire the second image region.

10. The method of claim 9, wherein obtaining the enhancement layer encoding of the first image region comprises: Obtain the residual set corresponding to the first image region, and quantize the residual set corresponding to the first image region using a given quantization step size. in: The given quantization step size is based on the size of the first image region and / or the target bit rate; and / or The size of the first image region is based on the given quantization step size and / or the target bit rate.

11. The method according to any one of claims 1 to 8, wherein performing the second enhancement layer processing comprises: Obtain the enhancement layer encoding of the second image region.

12. The method according to any one of claims 1 to 11, wherein obtaining the enhancement layer encoding of the second image region comprises: Obtain the residual set corresponding to the second image region, and the residual set corresponding to the second image region is quantized to zero.

13. The method according to any one of claims 1 to 11, wherein obtaining the enhancement layer encoding of the first image region comprises: Obtain the residual set corresponding to the first image region, and quantize the residual set corresponding to the first image region using a first quantization step size. The enhancement layer encoding that generates the second image region includes: Obtain the residual set corresponding to the second image region, and the residual set corresponding to the second image region is quantized using a second higher quantization step size.

14. The method of claim 13, wherein: The first quantization step size is based on the size of the first image region and / or the target bit rate; and / or The size of the first image region is determined based on the first quantization step size and / or the target bit rate.

15. The method according to any one of claims 1 to 14, wherein the second image region surrounds the first image region.

16. The method according to any one of claims 1 to 15, wherein the first image region corresponds to the user's expected field of view of the XR display device.

17. The method according to any one of claims 1 to 16, wherein the image is a stereoscopic image having a left image portion and a right image portion.

18. The method of claim 17, wherein the second image region includes the boundary between the left image portion and the right image portion.

19. The method according to claim 17 or 18, wherein the first image region comprises a first sub-region and a second sub-region, and wherein the first sub-region and the second sub-region do not overlap.

20. The method of claim 19, wherein the first sub-region is centered on one of the left image portion and the right image portion, and wherein the second sub-region is centered on the other of the left image portion and the right image portion.

21. The method according to claim 19 or 20, wherein the first sub-region and / or the second sub-region has a rectangular shape, and wherein the height of the rectangular shape of the first sub-region and / or the second sub-region is less than the width of the rectangular shape of the first sub-region and / or the second sub-region.

22. The method according to any one of claims 1 to 21, wherein suppressing the encoding of the enhancement layer for acquiring the second image region comprises: The enhancement layer mask is then applied to the second image region.

23. The method according to any one of claims 1 to 22, wherein the second image region includes at least a portion of the edge of the image.

24. The method according to any one of claims 1 to 23, comprising: A third enhancement layer process is performed on a third image region of the image, wherein performing the third enhancement layer process includes obtaining the enhancement layer code of the third image region according to different third encoding priorities.

25. The method of claim 24, wherein the third image region is between the first image region and the second image region, and the third encoding priority is between the first encoding priority and the second encoding priority.

26. The method according to any one of claims 1 to 25, wherein the enhancement layer comprises residuals generated by upsampler-downsampler asymmetry and / or residuals generated by encoder-decoder errors.

27. The method according to any one of claims 1 to 26, wherein the base layer encoding of the image comprises: The first base layer encoding portion corresponding to the first image region; and The second base layer encoding portion corresponding to the second image region. The first base layer coding portion has a first coding quality, and the second base layer coding portion has a second lower coding quality.

28. The method according to any one of claims 1 to 27, wherein the XR content includes XR pixel streaming content.

29. The method according to any one of claims 1 to 28, wherein the enhancement layer coding of the first image region and / or the enhancement layer coding of the second image region are enhancement layer coding derived from a multilayer scalable codec.

30. The method according to any one of claims 1 to 29, comprising: Output user data for one or more regions of the image, where one or more post-processing operations will be applied.

31. The method of claim 30, wherein the one or more regions of the image are different from the first image region and / or the second image region of the image.

32. The method of claim 30 or 31, wherein the one or more post-processing operations include a de-ringing post-processing operation.

33. The method according to any one of claims 1 to 32, comprising: Obtain the base layer encoding of the depth map, which indicates depth information associated with the image; performing a first enhancement layer processing for a first region of the depth map, wherein performing a first enhancement layer processing for the first region of the depth map comprises obtaining an enhancement layer coding of the first region of the depth map, the enhancement layer coding of the first region of the depth map being in accordance with a first depth map coding priority; as well as Performing a second enhancement layer process on a second region of the depth map, wherein performing the second enhancement layer process on the second region of the depth map includes: Suppress the enhancement layer encoding of the second region used to acquire the depth map; or The enhancement layer code of the second region of the depth map is obtained, and the enhancement layer code of the second region of the depth map is obtained according to a second lower depth map coding priority.

34. The method of claim 33, wherein the first region of the depth map corresponds to the first image region of the image.

35. A method of processing an image, the image including extended reality XR content and for use in an XR display device, the method comprising: Obtain the base layer encoding of the image; The enhancement layer encoding of a first image region of the image is obtained, and the enhancement layer encoding of the first image region is performed according to a first encoding priority; as well as Any one of the following: Obtain the enhancement layer code of the second image region, wherein the enhancement layer code of the second image region is obtained according to a second lower coding priority; or The enhancement layer encoding of the second image region is not acquired. The method further includes: The acquired encoding is used to reconstruct the image; as well as The reconstructed image is displayed on the XR display device.

36. The method of claim 35, wherein the method is performed during video game streaming.

37. The method of claim 35, wherein the method is performed during a six-DOF narrative having a pre-rendered field of view.

38. The method of claim 35, wherein the method is performed during the design review of the digital twin and / or digital avatar.

39. An apparatus configured to perform the method according to any one of claims 1 to 38.

40. A bitstream comprising: The base layer encoding of the image, which includes extended reality XR content and is intended for XR display devices; and The enhancement layer encoding of the first image region of the image, wherein the enhancement layer encoding of the first image region has been generated according to a first encoding priority. Any one of the following: The bitstream includes an enhancement layer encoding of a second image region, which has been generated according to a second, lower encoding priority; or The bitstream does not include the enhancement layer encoding of the second image region.

Images