Encoding and decoding immersive video

Abstract

A bitstream, an encoder and method for encoding the bitstream, and a decoder and method for decoding the bitstream. The bitstream includes an encoded volumetric representation of a scene and an encoded light field (i.e., view-dependent) representation of view-dependent regions of the scene.

Description

Technical Field

[0001] This invention relates to the field of immersive video. Background Technology

[0002] Six degrees of freedom (6 DoF) immersive video allows viewing a scene from different positions and orientations. The creation of 6 DoF immersive video typically uses multiple cameras to acquire image frames of the scene from different viewpoints. The acquired image frames are then used to generate image frames at the new viewpoints.

[0003] Information from the scene, including acquired image frames that can be used to generate image frames at the target viewpoint, can be encoded in an atlas-based format. Typically, this involves processing the acquired image frames to identify redundant regions; then discarding the redundant regions and packing the remaining regions into an atlas. The atlas, along with associated metadata, is then encoded into a bitstream. A device receiving the encoded atlas and metadata can generate an image frame at the target viewpoint by reconstructing the image regions into the image frame at the target viewpoint using the metadata.

[0004] This process relies on the assumption that objects in the scene have Lambertian (i.e., diffuse) reflective properties. Surfaces with non-Lambertian reflective properties (such as glossy surfaces) cannot be rendered to different viewpoints using an image region of the surface from a single viewpoint because the photometric appearance of such surfaces depends on the viewpoint. Summary of the Invention

[0005] This invention is defined by the claims.

[0006] According to one aspect of the invention, a bitstream is provided, comprising: an encoded volumetric representation of a scene; and an encoded light field representation of one or more regions of the scene, wherein the light field representation includes residual signals for generating video data of at least a subset of views of the volumetric representation, each residual signal defining a difference between the corresponding view and a reprojection of the volumetric representation to the corresponding view.

[0007] Volumetric representation and light field representation each provide three-dimensional information about the scene. In the volumetric representation of the scene, viewpoint-dependent effects (such as those caused by glossy surfaces) have been removed or averaged. The light field representation of the scene can be a residual light field representation; that is, a representation that defines the differences between the volumetric representation and the video data used to generate the volumetric representation; these differences are due to viewpoint-dependent effects.

[0008] Specifically, the scene's light field representation represents the light field in regions containing glossy surfaces or other view-dependent surfaces. By combining the volume representation and the light field representation in the bitstream, the bitstream contains enough information to ensure that view-dependent surfaces in the scene can be accurately rendered at various viewpoints.

[0009] The inventors have recognized that including only a portion of the light field representation for the scene reduces the bit rate and pixel rate of the bitstream, as well as the complexity of rendering an image at the target viewpoint using the bitstream, while still including view-related aspects of the scene.

[0010] In some examples, the encoded light field representation has a smaller scale compared to the encoded volume representation.

[0011] The inventors have recognized that view-dependent lighting properties do not require the same high resolution as other visual elements. Therefore, the light field representation can be computed or scaled down at a smaller scale than the volume representation (if the volume representation is scaled down, the light field representation is scaled down by a larger scaling factor), while still allowing regions with view-dependent aspects to be reconstructed to the desired quality. In this way, the bit rate and pixel rate of the light field representation can be reduced.

[0012] In some examples, the bitstream also includes an indication of whether using a light field representation during rendering is required or optional.

[0013] The instruction that the light field representation is optional means that the volume representation itself can be used to render an image of the scene at the target viewpoint (although without any view-dependent lighting effects). This allows the light field representation to be discarded or ignored during rendering if needed.

[0014] In some examples, the bitstream also includes instructions on how to combine volume representation and light field representation at render time.

[0015] In some examples, the bitstream also includes instructions on how to interpolate different parts of the light field representation.

[0016] According to an example of another aspect of the invention, a method for encoding immersive video is provided, the method comprising: obtaining video data including a plurality of views of a scene; processing the video data to generate a volumetric representation of the scene; processing the video data and the volumetric representation to identify any differences between the video data and the volumetric representation, wherein any differences between the video data and the volumetric representation are identified by performing the following operations for each view of the video data: reprojecting the volumetric representation onto the view; and calculating a residual signal defining the difference between the view and the corresponding reprojection; processing the video data for any region of the scene containing the identified differences to define a light field representation of one or more regions, wherein the light field representation includes the residual signal and corresponding depth information for at least a subset of views of the video data; and encoding the volumetric representation and the light field representation into a bitstream.

[0017] A method for decoding immersive video is also provided, the method comprising: receiving a bitstream including an encoded volumetric representation of a scene and an encoded light field representation of one or more regions of the scene, wherein the light field representation includes residual signals for generating video data of at least a subset of views of the volumetric representation, each residual signal defining a difference between a corresponding view and a reprojection of the volumetric representation to the corresponding view; decoding the volumetric representation and the light field representation; defining a target viewpoint; processing the light field representation to define the position of one or more first regions in an image frame at the target viewpoint, the one or more first regions corresponding to the one or more regions of the scene represented by the light field representation; processing the light field representation for each pixel in the one or more first regions to determine a first pixel value for the pixel; processing the volumetric representation to define the position of one or more second regions in the image frame at the target viewpoint; processing the volumetric representation for at least a subset of pixels in the one or more second regions to determine a second pixel value for each pixel; and processing the first pixel value and the second pixel value for each pixel having both the first pixel value and the second pixel value to determine a final pixel value for the pixel.

[0018] In this way, compared to rendering the entire image frame using a full light field representation, the image frame at the target viewpoint can be rendered with less complexity to accurately capture the photometric appearance of the scene at the target viewpoint.

[0019] One or more first regions are regions of the target image that contain one or more objects in the scene having a view-dependent appearance. One or more second regions may overlap with one or more first regions.

[0020] In some examples, the step of processing the first pixel value and the second pixel value to determine the final pixel value of the pixel includes mixing the first pixel value and the second pixel value.

[0021] In some examples, the volume representation is processed to determine the second pixel value only for pixels that are not fully visible in the light field representation.

[0022] In other words, since the first pixel value of at least a region of the image frame at the target viewpoint has been determined, the second pixel value can be uncertain for pixels with 0% transparency for the first pixel value. This reduces the occurrence of unnatural transients.

[0023] In some examples, the step of processing the first pixel value and the second pixel value to determine the final pixel value of the pixel includes adding the first pixel value and the second pixel value together.

[0024] In some examples, the method further includes: obtaining repair data for one or more regions of the image frame; and processing the repair data to determine a third pixel value for any pixel that has neither a first pixel value nor a second pixel value.

[0025] In some examples, each first pixel value is determined using a first rendering technique; and each second pixel value is determined using a different second rendering technique.

[0026] For example, a simple rendering technique (such as z-buffering) can be used to determine the second pixel value, while a more complex rendering technique can be used to determine the first pixel value. This provides high-quality rendering of view-dependent effects in an image frame without unduly increasing rendering complexity (and thus enabling more efficient rendering of image frames).

[0027] A computer program product including computer program code units is also provided, which, when executed on a computing device having a processing system, cause the processing system to perform all the steps of any of the methods described above.

[0028] According to another aspect of the invention, an encoder configured to encode immersive video is provided, the encoder being configured to: acquire video data comprising a plurality of views of a scene; process the video data to generate a volumetric representation of the scene; process the video data and the volumetric representation to identify any differences between the video data and the volumetric representation, wherein any differences between the video data and the volumetric representation are identified by performing the following operations for each view of the video data: reprojecting the volumetric representation onto the view; and calculating a residual signal defining the difference between the view and the corresponding reprojection; processing the video data for any region of the scene containing the identified differences to define a light field representation of the region, wherein the light field representation includes the residual signal for at least a subset of views of the video data and corresponding depth information; and encoding the volumetric representation and the light field representation into a bitstream.

[0029] A decoder configured to decode immersive video is also provided, the decoder being configured to: receive a bitstream including an encoded volumetric representation of a scene and an encoded light field representation of one or more regions of the scene, wherein the light field representation includes residual signals for generating video data of at least a subset of views of the volumetric representation, each residual signal defining a difference between a corresponding view and a reprojection of the volumetric representation to the corresponding view; decode the volumetric representation and the light field representation; define a target viewpoint; process the light field representation to define the position of one or more first regions in an image frame at the target viewpoint, the one or more first regions corresponding to the one or more regions of the scene represented by the light field representation; process the light field representation for each pixel in the one or more first regions to determine a first pixel value for the pixel; process the volumetric representation to define the position of one or more second regions in the image frame at the target viewpoint; process the volumetric representation for at least a subset of pixels in the one or more second regions to determine a second pixel value for each pixel; and process the first pixel value and the second pixel value for each pixel having both the first pixel value and the second pixel value to determine a final pixel value for the pixel.

[0030] These and other aspects of the invention will become apparent from the embodiments described below, and these and other aspects of the invention will be illustrated with reference to the embodiments described below. Attached Figure Description

[0031] To better understand the invention and to more clearly illustrate how the invention can be implemented, reference will now be made to the accompanying drawings by way of example only, in which:

[0032] Figure 1 An immersive video system according to an embodiment of the present invention is shown;

[0033] Figure 2 A method for encoding immersive video according to an embodiment of the present invention is shown; and

[0034] Figure 3 A method for decoding immersive video according to an embodiment of the present invention is shown. Detailed Implementation

[0035] The invention will be described with reference to the accompanying drawings.

[0036] It should be understood that while the detailed descriptions and specific examples indicate exemplary embodiments of the apparatuses, systems, and methods, they are intended for illustrative purposes only and not to limit the scope of the invention. These and other features, aspects, and advantages of the apparatuses, systems, and methods of the present invention will be better understood from the following description, the appended claims, and the accompanying drawings. It should be understood that the drawings are merely schematic and not drawn to scale. It should also be understood that the same reference numerals are used throughout the drawings to indicate the same or similar parts.

[0037] This invention provides a bitstream, an encoder and method for encoding the bitstream, and a decoder and method for decoding the bitstream. The bitstream includes an encoded volumetric representation of a scene and an encoded light field (i.e., view-dependent) representation of view-dependent regions of the scene.

[0038] Figure 1 An immersive video system 100 according to an embodiment of the present invention is shown. The immersive video system includes an encoder 110 and a decoder 120. The encoder is configured to output a bitstream 115 received by the decoder. Figure 1 In this process, bitstream 115 is sent directly from the encoder to the decoder; however, as will be readily understood by those skilled in the art, the bitstream may alternatively be stored on a suitable storage medium before being sent to the decoder.

[0039] Bitstream 115 itself is an embodiment of the present invention. The bitstream includes an encoded volumetric representation of the scene and an encoded light field representation of one or more regions of the scene. The encoded light field representation is a view-dependent representation of the scene, while the encoded volumetric representation is not view-dependent. For example, the encoded volumetric representation may provide the color value of each of a plurality of points in the scene, while the encoded light field representation may provide the color and / or transparency value of each of a plurality of rays in one or more regions of the scene. One or more regions of the scene are one or more regions that include view-dependent surfaces (such as glossy surfaces and surfaces with specular highlights). In some examples, one or more regions of the scene may be represented by both a volumetric representation and a light field representation; in other examples, some or all of the regions represented by the light field representation may be excluded from the volumetric representation.

[0040] In some examples, the encoded light field representation can fully describe the light field of one or more regions of the scene; in other examples, the encoded light field representation may only contain the view-related aspects of the light field and can only fully describe the light field when combined with the scene's volume representation. Light field representation and volume representation are described in more detail below.

[0041] Various alternatives are envisioned for arranging the encoded volumetric representation and the encoded optical field representation within the bitstream 115. For example, the encoded volumetric representation and the encoded optical field representation can be provided in different atlases of the bitstream, or a single atlas can contain both patches of volumetric representation and patches of optical field representation. In some examples, patches of volumetric representation and patches of optical field representation can be arranged within an atlas such that the patches of volumetric representation and patches of optical field representation are contained in different tiles of the atlas. The bitstream may also include metadata indicating which atlas(s), patches, and / or patches (as the case) contain a portion of the volumetric representation and which atlas(s), patches, and / or patches contain a portion of the optical field representation.

[0042] In some examples, the encoded light field representation has a smaller scale compared to the encoded volumetric representation. The inventors have recognized that separate representations of view-related aspects of one or more regions of a scene (i.e., separate from representations of other visual elements of the scene) can be scaled down by a large scaling factor (i.e., larger than the scaling factor typically used to scale the representation of a scene in immersive video applications), or computed at a smaller scale compared to the representation of a scene typically used in immersive video applications, without significantly affecting the reconstruction quality of view-related effects. In some examples, different portions of the light field representation can be scaled down in the bitstream by different scaling factors. The bitstream may also include metadata providing the scaling factor by which the light field representation (or a specific portion of the light field representation) has been scaled down; alternatively, the scaling factor may be provided by an external device.

[0043] In some examples, bitstream 115 may also include an indication of whether the use of a light field representation during rendering is necessary or optional (e.g., provided in the bitstream's metadata). The bitstream may include an indication that the light field representation is necessary for rendering one or more areas of the scene that are not represented by a volume representation. Conversely, the bitstream may include an indication that the light field representation is optional when the volume representation represents the entire scene (i.e., any area of ​​the scene represented by the light field representation is also represented by the volume representation). If the bitstream includes an indication that the light field representation is optional, any scaling down or thinning of the bitstream can be performed by scaling down or removing the light field representation, since the volume representation itself provides sufficient information to allow rendering a full view of the scene at the target viewpoint (albeit at a lower quality than if a light field representation is also used).

[0044] In some examples, bitstream 115 may also include instructions on how to combine volume representations and light field representations during rendering (e.g., provided in the bitstream's metadata). For example, if the light field representation itself does not fully describe the light field of one or more regions, the bitstream may include instructions on using additive techniques. If the light field representation fully describes the light field of one or more regions, the bitstream may include instructions on using blending techniques. Techniques for combining volume representations and light field representations during rendering are described in more detail below.

[0045] In some examples, bitstream 115 may include instructions on how to interpolate different portions of the light field representation (e.g., provided in the bitstream's metadata). For example, the bitstream may include instructions on how to blend different overlapping portions of the light field representation at the target viewpoint when rendering image frames of the scene at the target viewpoint. In some examples, this instruction may include a definition of the (non-linear) transients of the light field representation from one view to another.

[0046] Figure 2 A method 200 for encoding immersive video according to an embodiment of the present invention is shown. Figure 1 The encoder 110 can use method 200 to generate bit stream 115.

[0047] Method 200 begins at step 210, where video data comprising multiple views of the scene is obtained. The video data may include multiple sequences of video frames, each sequence having been acquired by cameras with different positions and / or orientations.

[0048] At step 220, the video data is processed to generate a volumetric representation of the scene. The volumetric representation is a non-view-dependent representation of the scene (i.e., a representation that does not describe how the color of a part of the scene changes depending on the viewpoint).

[0049] For example, a volumetric representation can be generated by performing the following operation for each pixel in at least one image frame of video data: identifying corresponding pixels in image frames of other views of the scene (e.g., using depth information of the scene and the position / orientation of the camera used to acquire the video data) to define a set of pixels for each of a plurality of points in the scene. Abnormal pixels (e.g., due to occlusion) can be removed from each pixel set, and the remaining pixels in that set can be processed to determine the color of the point in the scene in the volumetric representation (e.g., the color of the point can be the average color of the set or the minimum (i.e., the darkest) color). The determination of the color of the point in the volumetric representation affects how the volumetric representation and the lighting field representation are combined when rendering an image frame at the target viewpoint. For example, if the minimum color in each set is selected, an additive combination involving only positive color values ​​can be used at rendering time. If the average color of each set is used in the volumetric representation, an additive combination involving differential color values ​​can be used at rendering time.

[0050] A scene's volumetric representation can include multiple texture blocks and multiple depth blocks, as is commonly used when encoding immersive video according to the MPEG Immersive Video (MIV) standard. Texture blocks provide texture information (e.g., color) of regions of the scene captured by video data, while depth blocks provide depth information of regions of the scene. Each depth block can correspond to a texture block.

[0051] At step 230, the video data and volume representation are processed to identify any differences between the video data and the volume representation.

[0052] At step 240, video data of any region of the scene containing the identified differences is processed to define a light field representation for each region containing the identified differences.

[0053] In some examples, the difference between video data and volumetric representation can be identified by reprojecting the volumetric representation onto the source view for each view of the video data (i.e., each "source view"). A residual signal defining the difference between the source view and the corresponding reprojection can be computed for each source view. The computation of the residual signal can depend on how the volumetric representation and the light field representation are combined when rendering image frames at the target viewpoint. For example, when using an additive combination technique, the residual signal may include only color values ​​(e.g., positive color values ​​if the volumetric representation provides the minimum color for each point in the scene, or differential color values ​​if the volumetric representation provides the average color for each point in the scene). When using a blending technique, the residual signal may include color values ​​premultiplied by transparency values ​​(e.g., [RA, GA, BA, A] or equivalents in another color space) and transparency values ​​(e.g., RGBA values) or color values.

[0054] The light field representation may include residual signals for at least a subset of the source views and corresponding depth information. In some examples, residual signals from all source views may be processed to select a subset of residual signals that allows the view dependencies of the scene to be modeled to the desired quality, and the light field representation may include the selected subset and corresponding depth information. As those skilled in the art will understand, the number of residual signals in such a subset will depend on the reflective properties of the surfaces in the scene, the complexity of the lighting in the scene, and the desired quality.

[0055] In some examples, the difference between the volumetric representation and the video data can be identified only for regions of the scene that have been identified as containing one or more view-dependent surfaces (e.g., surfaces with non-Lambertian reflective properties, such as glossy surfaces and surfaces with specular highlights). Regions containing view-dependent surfaces can be identified by reprojecting each source view in the video data onto at least one other source view and comparing the depth information of the source views to identify regions where points with the same 3D location in the scene have different color values ​​in different source views (i.e., the difference in color values ​​exceeds a predetermined difference threshold or a threshold adjusted according to available resources such as pixel rate).

[0056] Therefore, in some examples where the light field representation includes residual signals and corresponding depth information, the residual signals can be identified as the difference between the region of each custom source view that contains one or more view-related surfaces and the corresponding region of the volume representation reprojected onto the corresponding source view.

[0057] In other examples, the difference between the volumetric representation and the video data can be identified for the entire source view to calculate the residual signal. Regions with residual signals below a predetermined signal threshold can then be identified as regions with little or no view relevance; the residual signals for these regions can be omitted from the light field representation.

[0058] At step 250, the volume representation and light field representation are encoded into a bitstream. Metadata that enables the encoded volume and light field representations to be decoded and used to render image frames at the target viewpoint can also be encoded into a bitstream. In some examples, the volume and light field representations can be encoded into one or more video sub-bitstreams, and the metadata can be encoded into a metadata sub-bitstream; the sub-bitstreams can then be multiplexed to form a bitstream. Any suitable codec can be used to encode the volume representation, light field representation, and metadata.

[0059] As referenced above Figure 1As described in bitstream 115, the metadata may include information identifying the positions of the volume representation and the light field representation in the bitstream (e.g., information identifying which atlas(s), slices, and / or blocks contain a portion of the volume representation and which atlas(s), slices, and / or blocks contain a portion of the light field representation), and information on the position and / or orientation of each camera used to obtain video data of the scene. In some examples, the metadata may include instructions on how to combine the volume representation and the light field representation at rendering time, as described above. In some examples, the metadata may include instructions on how to interpolate different portions of the light field representation, as described above.

[0060] In some examples, the light field representation can be encoded at a smaller scale compared to the volume representation. For example, step 250, which encodes the volume representation and the light field representation into a bitstream, may include a sub-step of downscaling the light field representation. Alternatively, the light field representation can be computed at a smaller scale (e.g., by downscaling the video data before computing the light field representation, or at least downscaling the area of ​​the scene to be represented by the light field representation). The scaling factor of the light field representation can be included in the metadata of the bitstream.

[0061] In some examples, different portions of the light field representation can be scaled down using different scaling factors. For instance, where the light field representation includes residual signals and corresponding depth information, a desired scaling factor can be determined for each residual signal and its corresponding depth information. The desired scaling factor for each residual signal and its corresponding depth information can be determined such that the residual signals and depth information of a given source view (or a region of the source view, as appropriate), scaled down by the desired scaling factor, allow the source view to be reconstructed with the desired quality. In some examples, the desired scaling factor can be determined based on an error metric, such as the sum of the differences between the original light field representation and the light field representation obtained by (color-adaptive) upscaling of the scaled-down light field representation.

[0062] The residual signal and its corresponding depth information can then be scaled down using a desired scaling factor for each residual signal. The desired scaling factor and information identifying the residual signal and depth information associated with each desired scaling factor can be included in the metadata.

[0063] In some examples, the volume representation encoded in the bitstream can be a volume representation of only a portion of the scene. For example, the region of the volume representation that corresponds to the area of ​​the scene fully represented in the light field representation (e.g., a highly specular region captured by all views of the light field representation) can be removed from the volume representation, and the remaining volume representation can be encoded into the bitstream. In other examples, the volume representation can represent all regions of the scene (i.e., including all regions represented by the light field representation).

[0064] Metadata in the bitstream can include indications about whether using a light field representation during rendering is necessary (i.e., in cases where a volume representation does not represent the entire scene) or optional (i.e., in cases where the area represented by the light field representation is also represented by the volume representation).

[0065] Figure 3 A method 300 for decoding immersive video according to an embodiment of the present invention is shown. Figure 1 Decoder 120 can use method 300 to decode bitstream 115.

[0066] Method 300 begins at step 310, where a bitstream is received. The bitstream includes an encoded volumetric representation of the scene and encoded light field representations of one or more regions of the scene. The bitstream may also include associated metadata (e.g., metadata in the bitstream identifying the encoded volumetric and light field representations, information defining the position and / or orientation of each camera used to obtain video data of the scene from which the volumetric and light field representations are generated, etc.). For example, the received bitstream may be as described above. Figure 1 The described bitstream is 115.

[0067] At step 320, the volume representation and the light field representation are decoded. Any suitable codec can be used to decode the volume representation and the light field representation.

[0068] In step 330, the target viewpoint is defined. The target viewpoint is the viewpoint of the image frame where the scene is to be rendered.

[0069] At step 340, the light field representation is processed to define the position of one or more first regions in the image frame at the target viewpoint. The one or more first regions correspond to one or more regions of the scene represented by the light field representation. In other words, the light field representation is distorted to the target viewpoint. In some examples, where more than one first region is defined, the first regions may overlap each other (i.e., different portions of the light field representation overlap at the target viewpoint).

[0070] For example, as described above, the light field representation may include a residual signal defining the difference between the volume representation and the source view used to generate the volume representation, along with corresponding depth information. One or more first regions may be defined by warping at least a subset of the residual signal to the target view using the corresponding depth information (e.g., warping at least the residual signal of the source view having the viewpoint closest to the target viewpoint). For example, each warped residual may define a first region.

[0071] In some examples, the light field representation may already be encoded at a smaller scale in the bitstream, as described above. The scaling factor used for the light field representation may be provided in the metadata of the bitstream or by an external device. In these examples, step 340, which defines the position of one or more first regions in an image frame at the target viewpoint, may include a sub-step of rescaling the light field representation using a scaling factor (i.e., magnifying each portion of the light field representation by the relevant scaling factor). For example, if the light field representation (or a portion thereof) is scaled down by a factor of 16, the light field representation (or the relevant portion, as appropriate) may be rescaled by magnifying it by a factor of 16.

[0072] At step 350, the light field representation is processed to determine a first pixel value for each pixel in one or more first regions. The determined pixel value may be the color value of the pixel. In some examples, the pixel value may include transparency.

[0073] In some examples, the metadata in the bitstream may include instructions on how to interpolate different portions of the light field representation; if such instructions are provided, the first pixel value of any pixel belonging to more than one first region can be determined by interpolating the different portions of the light field representation as indicated by the instructions. If no such instructions are provided, a predefined default interpolation technique can be used.

[0074] For example, in a light field representation that includes the residual signal and corresponding depth information as described above, and where each first region is defined by the warped residual signal, the first pixel value of each pixel belonging to a single first region is determined as the value of the pixel's associated warped residual signal. For pixels belonging to more than one first region, each warped residual signal defining the first region to which the pixel belongs can be used to determine the first pixel value: for example, the values ​​of each of the pixel's warped residual signals can be weighted based on the distance from the target viewpoint.

[0075] At step 360, the volume representation is processed to define the position of one or more second regions in the image frame at the target viewpoint. The one or more second regions may overlap with one or more first regions (e.g., in the case where the volume representation represents the entire scene, the one or more second regions may together occupy the entire image frame).

[0076] At step 370, the volume representation is processed to determine a second pixel value for each pixel in at least a subset of pixels within one or more second regions. As described below, whether the second pixel value is determined for each pixel in one or more second regions or only for a subset will depend on the technique used to determine the final pixel value.

[0077] It is well known to use volume representation to determine the pixel value of a pixel in an image frame at a target viewpoint. For example, in the case where the volume representation includes multiple texture blocks and multiple depth blocks, each texture block can be warped to the target viewpoint using the corresponding depth blocks to define one or more second regions, and the value of the warped texture block to which the pixel in the second region belongs can be used to determine the second pixel value of the pixel. To improve rendering efficiency, the rendering technique used to determine the second pixel value can be a simple rendering technique, such as z-buffering, because the more complex rendering techniques used to determine the first pixel value are not necessary when using volume representation to determine the pixel value.

[0078] At step 380, for each pixel having both a first pixel value and a second pixel value, the first pixel value and the second pixel value are processed to determine the final pixel value of the pixel. The final pixel value of each pixel is the pixel value used in the image frame at the target viewpoint. For a pixel having only a first pixel value, the first pixel value is the pixel value used in the image frame at the target viewpoint; similarly, for a pixel having only a second pixel value, the second pixel value is the pixel value used in the image frame at the target viewpoint.

[0079] Consider various techniques for processing the first and second pixel values ​​of a pixel to determine the final pixel value. In some examples, metadata in the bitstream may include instructions on how to combine the volume representation and the light field representation at render time; the final pixel value can then be determined as specified in the metadata. Where the bitstream does not provide instructions on how to combine the volume representation and the light field representation, a predefined default technique can be used to determine the final pixel value (or the value provided for the light field representation can be used to determine which technique to use; for example, if the light field representation includes transparency values, a blending technique can be used).

[0080] In some examples, step 380 of determining the final pixel value may include: for each pixel having both a first pixel value and a second pixel value, blending the first pixel value and the second pixel value to determine the final pixel value (e.g., using the transparency of the first pixel value to alpha-blend the first pixel value on the second pixel value). Blending techniques can be used when the light field representation fully describes the light field of one or more regions of the scene.

[0081] In some examples, when using blending techniques to determine the final pixel value, the second pixel value can be determined only for pixels in one or more second regions that are not fully visible in the light field representation. In other words, the volume representation can be processed to determine the second pixel value for each pixel in a subset of pixels within one or more second regions, where this subset does not include pixels with a first pixel value having 0% transparency (or transparency below a predefined transparency threshold). Culling techniques can be used to determine which regions of the volume representation are visible to generate a subset of pixels in one or more second regions.

[0082] In some examples, step 380 of determining the final pixel value may include: for each pixel having both a first pixel value and a second pixel value, adding the first pixel value and the second pixel value of the pixel. This addition technique can be used where the light field representation only fully describes the light field when combined with the volume representation.

[0083] For example, in some cases (e.g., where the volume representation provides the minimum color for each point in the scene), each first pixel value and each second pixel value can be positive color values, and the final pixel value is the sum of the first and second pixel values. In other cases (e.g., where the volume representation provides the average color for each point in the scene), each first pixel value can be a differential color value that defines the difference between the color of a relevant point in the scene in the volume representation and the color of a relevant point from the target view. A positive differential color value indicates that the color of a relevant point from the target view has a higher value than the corresponding value in the volume representation, while a negative differential color value indicates that the color of a relevant point from the target view has a lower value than the corresponding value in the volume representation. A differential color value of zero indicates that the color of a relevant point in the scene in the volume representation is the same as the color of a relevant point from the target view.

[0084] It should be understood that Figure 3 Method 300 steps may not be followed Figure 3 The steps are executed in the order shown. For example, in some examples, steps 360 and 370 may be executed before or in parallel with steps 340 and 350. In some examples, the sub-steps of steps 340 to 380 may be interleaved: for example, the final pixel value of a region of the image frame at the target viewpoint may be determined before the first and second pixel values ​​of another region of the image frame are determined.

[0085] In some cases, volumetric and light field representations may not provide enough information to allow rendering a view of the scene at all viewpoints (e.g., some areas of the scene may not be visible in any view included in the video data). For some target viewpoints, this will result in image frames where some pixels have neither a first pixel value nor a second pixel value.

[0086] In some examples, repair data can be obtained for one or more regions of an image frame at the target viewpoint, and the repair data can be processed to determine a third pixel value for any pixel that has neither a first pixel value nor a second pixel value. For such pixels, the third pixel value is the pixel value used in the image frame at the target viewpoint (it is the only pixel value available for the pixel).

[0087] In some examples, the bitstream, which includes volumetric and light field representations, may also include scene restoration data (i.e., restoration data for one or more regions of an image frame can be obtained by receiving and decoding the bitstream and warping the restoration data to a target viewpoint). For example, the bitstream may include encoded blocks of restoration data and metadata identifying the blocks of restoration data in the bitstream, as described in International Patent Application WO2022069325.

[0088] Any method described herein can be implemented by a computer, wherein computer implementation means that the steps of the method are performed by one or more computers, and wherein a computer is defined as a device adapted to process data. A computer may be adapted to process data according to prescribed instructions.

[0089] It should be understood that, since the disclosed method can be a computer-implemented method, the concept of a computer program product is also proposed, which includes code units for implementing any described method when the program is run on a processing system.

[0090] Those skilled in the art will be able to readily develop processors for performing any of the methods described herein. Therefore, each step of the flowchart can represent a different action performed by the processor and can be executed by the corresponding module of the processor.

[0091] One or more steps of any method described herein may be performed by one or more processors. A processor includes electronic circuitry suitable for processing data.

[0092] As described above, the system utilizes a processor to perform data processing. A processor can be implemented in various ways using software and / or hardware to perform a variety of required functions. A processor typically employs one or more microprocessors that can be programmed using software (e.g., microcode) to perform desired functions. A processor can be implemented as a combination of dedicated hardware for performing some functions and one or more programmable microprocessors and associated circuitry for performing other functions.

[0093] Examples of circuit systems that may be employed in various embodiments of this disclosure include, but are not limited to, conventional microprocessors, application-specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs).

[0094] In various implementations, the processor may be associated with one or more storage media, such as volatile and non-volatile computer memories, such as RAM, PROM, EPROM, and EEPROM. The storage media may be encoded with one or more programs that perform the required functions when executed on one or more processors and / or controllers. The various storage media may be fixed within the processor or controller, or may be transferable, such that one or more programs stored thereon can be loaded into the processor.

[0095] A single processor or other unit can perform the functions of several items recounted in the claims.

[0096] By studying the accompanying drawings, the disclosure, and the appended claims, those skilled in the art can understand and implement variations of the disclosed embodiments in practicing the claimed invention. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite articles "a" or "an" do not exclude a plurality.

[0097] The functions implemented by a processor can be implemented by a single processor or by multiple separate processing units, which together can be considered to constitute a "processor". In some cases, such processing units can be located far apart from each other and communicate with each other via wired or wireless means.

[0098] The fact that certain measures are described in mutually different dependent claims does not imply that combinations of these measures cannot be used advantageously.

[0099] Computer programs can be stored / distributed on suitable media, such as optical storage media or solid-state media provided together with or as part of other hardware, but can also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems.

[0100] If the term “suitable” is used in the claims or description, it should be noted that the term “suitable” is intended to be equivalent to the term “configured as.” If the term “arranged” is used in the claims or description, it should be noted that the term “arranged” is intended to be equivalent to the term “system,” and vice versa.

[0101] Any reference numerals in the claims should not be construed as limiting the scope.

Claims

1. A bitstream (115), comprising: The encoded volume representation of the scene; as well as An encoded light field representation of one or more regions of the scene, wherein the light field representation includes residual signals for generating video data of at least a subset of views, each residual signal defining the difference between the corresponding view and the reprojection of the volume representation to the corresponding view.

2. The bitstream (115) according to claim 1, wherein, The encoded light field representation has a smaller scale compared to the encoded volume representation.

3. The bitstream (115) according to claim 1 or 2 further includes an indication of whether the use of the light field representation during rendering is necessary or optional.

4. The bitstream (115) according to any one of claims 1 to 3, further comprising instructions on how to combine the volume representation and the light field representation during rendering.

5. The bitstream (115) according to any one of claims 1 to 4 further includes instructions on how to interpolate different portions of the optical field representation.

6. A method (200) for encoding immersive video, the method comprising: Obtain video data including multiple views of the scene; The video data is processed to generate a volumetric representation of the scene; The video data and the volume representation are processed to identify any differences between the video data and the volume representation, wherein any differences between the video data and the volume representation are identified by performing the following operations for each view of the video data: Reproject the volume representation onto the view; and Calculate the residual signal that defines the difference between the view and the corresponding reprojection; The video data is processed for any region of the scene containing the identified differences to define a light field representation for one or more regions, wherein the light field representation includes the residual signal and corresponding depth information for at least a subset of views of the video data; and The volume representation and the optical field representation are encoded into a bit stream (115).

7. A method (300) for decoding immersive video, the method comprising: Receive bitstream (115), the bitstream (115) including an encoded volumetric representation of a scene and an encoded light field representation of one or more regions of the scene, wherein the light field representation includes residual signals for generating video data of at least a subset of views of the volumetric representation, each residual signal defining the difference between the corresponding view and the reprojection of the volumetric representation to the corresponding view; Decode the volume representation and the light field representation; Define the target viewpoint; The light field representation is processed to define the position of one or more first regions in an image frame at the target viewpoint, the one or more first regions corresponding to the one or more regions of the scene represented by the light field representation; For each pixel in the one or more first regions, the light field representation is processed to determine a first pixel value for that pixel; The volume representation is processed to define the position of one or more second regions in the image frame at the target viewpoint; For at least a subset of pixels in the one or more second regions, the volume representation is processed to determine a second pixel value for each pixel; and For each pixel having both a first pixel value and a second pixel value, the first pixel value and the second pixel value are processed to determine the final pixel value of the pixel.

8. The method (300) according to claim 7, wherein, The step of processing the first pixel value and the second pixel value to determine the final pixel value of the pixel includes mixing the first pixel value and the second pixel value.

9. The method (300) according to claim 7 or 8, wherein, The volume representation is processed to determine a second pixel value only for pixels that are not fully visible in the light field representation.

10. The method (300) according to claim 7, wherein, The step of processing the first pixel value and the second pixel value to determine the final pixel value of the pixel includes adding the first pixel value and the second pixel value together.

11. The method (300) according to any one of claims 7 to 10, further comprising: Obtain repair data for one or more regions of the image frame; as well as For any pixel that has neither a first pixel value nor a second pixel value, the repair data is processed to determine a third pixel value for that pixel.

12. The method (300) according to any one of claims 7 to 11, wherein: Each first pixel value is determined using a first rendering technique; and Each second pixel value is determined using a different second rendering technique.

13. A computer program product comprising computer program code units, which, when executed on a computing device having a processing system, cause the processing system to perform all the steps of the method (200, 300) according to any one of claims 6 to 11.

14. An encoder (110) configured to encode immersive video, the encoder being configured to: Obtain video data including multiple views of the scene; The video data is processed to generate a volumetric representation of the scene; The video data and the volume representation are processed to identify any differences between the video data and the volume representation, wherein any differences between the video data and the volume representation are identified by reprojecting the volume representation onto the view for each view of the video data; The video data is processed for any region of the scene containing the identified differences to define a light field representation of the region, wherein the light field representation includes the residual signal and corresponding depth information for at least a subset of views of the video data; and The volume representation and the optical field representation are encoded into a bit stream (115).

15. A decoder (120) configured to decode immersive video, the decoder being configured to: Receive bitstream (115), the bitstream (115) comprising an encoded volume representation of the scene and an encoded light field representation of one or more regions of the scene, wherein, The light field representation includes residual signals for generating video data of at least a subset of views, each residual signal defining the difference between the corresponding view and the reprojection of the volume representation to the corresponding view; Decode the volume representation and the light field representation; Define the target viewpoint; The light field representation is processed to define the position of one or more first regions in an image frame at the target viewpoint, the one or more first regions corresponding to the one or more regions of the scene represented by the light field representation; For each pixel in the one or more first regions, the light field representation is processed to determine a first pixel value for that pixel; The volume representation is processed to define the position of one or more second regions in the image frame at the target viewpoint; For at least a subset of pixels in the one or more second regions, the volume representation is processed to determine a second pixel value for each pixel; as well as For each pixel having both a first pixel value and a second pixel value, the first pixel value and the second pixel value are processed to determine the final pixel value of the pixel.

Images