Method and device for coding and decoding multiview video sequences

Abstract

To provide a method for decoding a data stream representing a multi-view video, and a method for encoding the data stream.SOLUTION: A method for decoding a data stream includes at least coded data representing at least one sub-image, and includes texture data and depth data associated with the texture data, and the sub-image includes at least one zone. An indicator is decoded, indicating whether the sub-image is coded according to a first method or a second method. When the indicator indicates that the sub-image is coded according to the first method, a binary map is decoded, indicating, for at least one pixel of the sub-image, whether or not the pixel belongs to a useful zone. When the indicator indicates that the sub-image is coded according to the second method, the depth data of the sub-image for the pixels located outside the useful zone includes a decoded depth value.SELECTED DRAWING: None

Description

[Technical Field]

[0001] The present invention relates to immersive video representing scenes captured by one or more cameras, and more particularly to the coding and decoding of such video. [Background technology]

[0002] In the context of immersive video, i.e., where the viewer feels immersed in the scene, the scene is typically captured by a set of cameras, as shown in Figure 1. These cameras may be of type 2D (cameras C1, C2, C3, C4 in Figure 1) or of type 360, i.e., capturing the entire scene in 360 degrees around the camera (camera C5 in Figure 1).

[0003] All of these captured views are conventionally encoded and then decoded by the viewer's terminal. However, to provide a sufficient quality experience, and therefore a good sense of visual quality and immersion in the scene displayed to the viewer, displaying only the captured views is insufficient.

[0004] To improve the sense of immersion in the scene, one or more views, called intermediate views, are usually computed from the decoded views.

[0005] These intermediate views can be calculated by a view "synthesis" algorithm. An example of a synthesis algorithm is shown in Figure 2, which mainly shows the inputs / outputs of the view synthesis algorithm. According to this example, from n views (here 1 to 4), the synthesis algorithm can synthesize a viewpoint "v" located in a given volume called the viewing volume, depending on the camera position. The n views and the synthesized view are then combined with texture data (T1 to T4, T v ) and depth data (D1~D4, D v) Depth data is typically represented in the form of a depth map, i.e. an image that indicates, at each point in the image, the depth associated with the object represented at that point.

[0006] These depth maps are either captured by a specific camera or computed from texture views. Due to the large amount of data involved, it is difficult to compress the captured views to represent the scene.

[0007] Furthermore, since the captured views are needed to synthesize intermediate views, a compromise between view coding efficiency and the quality of intermediate view synthesis is required.

[0008] MV-HEVC and 3D-HEVC encoders are two extensions of the 2D HEVC encoder adapted for coding multiview video ("High Efficiency Video Coding, Coding Tools and Specification", Matthias Wien, Signals and Communication Technology, 2015).

[0009] Such an encoder takes as input a set of views and optionally associated depth maps. MV-HEVC encoders do not apply any specific coding tools at the block level. When coding an image of the current view, they simply use images of other views as reference images in addition to previously coded / decoded images of the current view.

[0010] The 3D-HEVC encoder adds specific tools to depth map coding, uses more advanced inter-view and inter-component redundancy, and the depth map coding leverages texture coding.

[0011] However, for 360-degree multiview content, the redundancy between views does not seem to be properly exploited by these two codecs. Therefore, their coding efficiency is far from optimal. Furthermore, nothing is done to preserve zones that may lead to occlusions, which affect the quality of the synthesized view image.

[0012] FIG. 3 shows another method for coding multiview video and synthesizing intermediate views at the decoder side.

[0013] According to this method, one or more base views (T in Figure 3) b , D b ) is coded with a 2D encoder, for example, an HEVC encoder, or a multiview encoder.

[0014] Other Views (T s , D s ) are processed to extract specific zones from each of these views.

[0015] If not all captured views are sent to the decoder, or if the cameras are naturally far apart, occlusion zones increase. These zones correspond to objects that are hidden in the views sent to the decoder but are required to reconstruct the image from any viewpoint within the viewing volume. Thus, the extracted zones can correspond to occlusion zones that are hidden in the base views but are visible in views that are not sent to the decoder.

[0016] Subsampling the views can introduce significant flaws in view synthesis, especially when objects are highly distorted depending on the viewing position, i.e., the synthesized viewpoint.

[0017] According to the method shown in Figure 3, the extracted zones, hereinafter also referred to as patches, are coded, for example, by a conventional 2D video encoder. Then, at the decoder side, a synthesis algorithm can generate one or more intermediate views from the base view and the decoded patches.

[0018] Overall, the patch can transmit information to the decoder related to the same zone that can be seen from the perspective of the base view or from another perspective, thus taking into account changes in lighting between the two perspectives or a decrease in projection quality related to the object itself, for example in the case of tilted objects (over a certain depth range).

[0019] In order to code the patches representing the extracted zones, it is necessary to code the depth associated with the texture of these patches. To simplify the coding and representation of these patches, which may have different shapes, the patches are generally stored in the form of rectangular images, or at least in the form of quadrilaterals, also called sub-images.

[0020] Expressed in the form of sub-images, the sub-images appear to correspond to extracted zones and comprise useful data that is likely to be used for the synthesis of intermediate views, as well as non-useful data that does not initially belong to the extracted patch.

[0021] For efficient coding of patches, the coding of non-useful data needs to have minimal impact on the coding of useful data.

[0022] Furthermore, at the decoder side, it is necessary to send to the synthesis algorithm an item of information that helps it to distinguish between useful and unuseful zones so that it does not use unuseful zones during synthesis.

[0023] Therefore, there is a need to improve upon the prior art. [Prior art documents] [Non-patent literature]

[0024] [Non-Patent Document 1] "High Efficiency Video Coding, Coding Tools and Specification", Matthias Wien, Signals and Communication Technology, 2015 Summary of the Invention [Means for solving the problem]

[0025] The present invention improves upon the prior art. To this end, the invention relates to a method for decoding a data stream representing a multi-view video, the data stream comprising at least coded data representing at least one sub-image, the coded data comprising texture data and depth data associated with said texture data, the sub-image comprising at least one zone, called useful zone, intended to be used to generate at least one image of at least one view.

[0026] Such a decoding method is - decoding from said at least one data stream an indicator indicating whether a sub-image is coded according to a first method or according to a second method; - if the indicator indicates that the sub-image is coded according to the first method, decoding into the sub-image a binary map indicating, for at least one pixel of the sub-image, whether said pixel belongs to a useful zone or not; - decoding texture data and depth data of the sub-image, wherein if the indicator indicates that the sub-image is coded according to the second method, the depth data of said sub-image for pixels located outside the useful zone comprise decoded depth values.

[0027] Thus, according to the present invention, it is possible to code and decode sub-image coding zones that are useful for synthesis while limiting the coding cost of these sub-images. Specifically, depending on the value of the indicators coded in the stream, the useful zones of the sub-images representing the patches can be determined by the synthesis algorithm when reconstructing the view from the depth information coded in the stream of sub-images or from the binary map coded and decoded from the stream.

[0028] Therefore, unlike conventional techniques, the present invention does not require systematic coding of binary validity maps for subimages to identify useful zones. For example, a competition can be implemented in the encoder to determine the optimal rate / distortion coding of the depth information for a patch. Such a competition compares the performance of a first method that codes a binary map indicating which pixels belong to the useful zone and are sent to the synthesis algorithm when synthesizing the view, with the performance of a second method that assigns specific values ​​to the depth values ​​of pixels located outside the useful zone.

[0029] When a sub-image is coded according to the first method, only the depth values ​​of pixels that belong to the useful zone are coded and decoded. On the other hand, when a sub-image is coded according to the second method, the depth values ​​of all pixels of the sub-image are coded and decoded, regardless of whether these pixels belong to the useful zone. Pixels of the sub-image that do not belong to the useful zone have a specific decoded depth value determined at the time of coding.

[0030] Such a specific value ensures that the compositing algorithm does not use pixels that do not belong to the useful zone when compositing views.

[0031] The sub-images may comprise texture and depth data related to occlusion zones that are not displayed in the views coded in the data stream, and coding such sub-images therefore improves the quality of synthesis of views that are not coded in the data stream.

[0032] In another embodiment variant, a sub-image may comprise texture and depth data relating to a zone that is visible from a view coded in the data stream but that introduces significant deficiencies into the synthesis of the view, and coding such a sub-image improves the quality of the synthesis of this view by providing texture and depth refinements to the zone.

[0033] The proposed invention allows for better adaptation to view content and occlusion type. Specifically, the first method preserves the sharp contours of occlusion zones during synthesis, while the second method saves coding costs by avoiding the transmission of binary maps. However, coding according to the second method does not preserve the sharp contours of occlusion zones, since no distinction is made between useful and non-useful zones in the sub-image.

[0034] The choice in the encoder between the first and second method can be made, for example, via rate / distortion optimization, where the distortion is measured in the encoder from a partial synthesis of the reconstructed zone by the useful zone.

[0035] The present invention also provides a method for coding a data stream representing a multi-view video, comprising the steps of: - coding at least one sub-image comprising texture data and depth data associated with said texture data, said sub-image comprising a zone, called useful zone, intended to be used to generate at least one image of at least one view, The step of encoding at least one sub-image comprises: - determining a coding method for coding the sub-image between a first method and a second method; - coding an indicator that indicates whether the sub-image is coded according to the first method or according to the second method; - if the indicator indicates that the sub-image is coded according to the first method, coding in the sub-image a binary map indicating, for at least one pixel of the sub-image, whether said pixel belongs to a useful zone or not; - if the indicator shows that the sub-image is coded according to the second method, - determining an allocation depth value; - for pixels of a sub-image located outside the useful zone, assigning an assigned depth value to the depth data of said sub-image; - coding the texture and depth data of the sub-image.

[0036] According to a particular embodiment of the invention, the assigned depth value is determined over at least the useful zone of the sub-image and is determined from a minimum depth value reduced by a predetermined value.

[0037] The minimum depth value is, for example, the depth value of the furthest (depth-wise) point of the useful zone in the sub-image. The minimum depth value is reduced by a predetermined value to obtain a specific depth value different from the depth values ​​of the points of the useful zone. The reduction by a predetermined value therefore makes it possible to take into account quantization errors when encoding the depth data.

[0038] Furthermore, using specific depth values ​​that remain close to the depth values ​​of the points of the useful zone limits the impact of the depth coding cost of the sub-images, in fact limiting the occurrence of strong discontinuities in the depth information coded for the sub-images.

[0039] Furthermore, such assigned depth values ​​can be used advantageously by the compositing algorithm. Indeed, if a conflict occurs during the compositing of images, the compositing algorithm will naturally prioritize objects located in the foreground, and therefore will have a larger depth value than objects located in the background. Thus, using a depth value for pixels of a sub-image located in a non-useful zone that is lower than the depth values ​​of pixels of the useful zone indicates to the compositing algorithm that the pixels with the smallest depth value do not belong to the useful zone, and therefore will not be prioritized for consideration by the compositing algorithm in the event of a conflict when reconstructing the composite image.

[0040] According to another particular embodiment of the invention, if the indicator indicates that the sub-image has been coded according to the second method, the decoding method, respectively the coding method, comprises the step of decoding, respectively coding, items of information representing assigned depth values, the assigned depth values ​​corresponding to the decoded, respectively coded, depth values ​​for pixels located outside the useful zone.

[0041] Therefore, according to this particular embodiment of the present invention, a binary map indicating which pixels belong to the useful zone can be created for a subimage coded according to the second method, since the decoder knows the assigned depth values ​​assigned to pixels located outside the useful zone. Therefore, it is possible to identify, from the depth values ​​of the decoded pixels, which pixels belong to the useful zone and which pixels are located outside the useful zone. The binary map thus constructed can be sent to a compositing algorithm. Therefore, the information provided as input to the compositing algorithm is the same regardless of whether the subimage was coded according to the first method or the second method.

[0042] According to another particular embodiment of the invention, the depth data of a sub-image coded into the data stream comprises, for at least one pixel of the sub-image, a residual determined by the difference between the depth value of said pixel and said assigned depth value.

[0043] This particular embodiment of the invention reduces the coding cost of the depth data of a sub-image when the sub-image is coded according to the second method: indeed, when a sub-image only comprises background information, the dynamics of the depth data is low and it is more advantageous to code this data as a residual.

[0044] According to another particular embodiment of the invention, an indicator indicating whether depth data is coded by a residual determined by the difference between the depth value of the pixel and the assigned depth value is decoded or coded for the at least one sub-image.

[0045] This particular embodiment of the invention allows to signal whether depth data in the case of coding by the second method are coded by residuals. Indeed, depending on the type of information carried by the sub-image, a residual-based coding mode may or may not be advantageous. Residual-based coding may be advantageous when the sub-image carries information related to occlusion zones, but may be less attractive for sub-images with refinement information.

[0046] According to another particular embodiment of the invention, the decoding method comprises the steps of: - if the indicator indicates that the sub-image is coded according to the second method, from the reconstructed depth data and the items of information representing the decoded assigned depth values, - if the indicator shows that the sub-image is coded according to the first method, then from the decoded binary map, - determining a useful zone; - constructing a depth map of a sub-image in which pixels located outside the useful zone are assigned the value of infinity; - transmitting the depth map to a synthesis module.

[0047] According to this particular embodiment of the present invention, the value of infinity assigned to the depth value allows the compositing algorithm to simply indicate which pixels belong to the useful and non-useful zones, avoiding the need to send additional data to the compositing module, such as a binary map indicating the useful and non-useful zones.

[0048] According to another particular embodiment of the invention, if the indicator indicates that the sub-image is coded according to the second method, the decoding method comprises: - determining useful zones from the reconstructed depth data and from the items of information representing the decoded assigned depth values; - constructing, for at least one pixel of the sub-image, a binary map indicating whether the pixel belongs to a useful zone or not; - sending the binary map to a synthesis module.

[0049] According to another particular embodiment of the invention, at least two sub-images intended to be used to generate said at least one image of said at least one view are coded into a data stream, and a minimum depth value is determined from each useful zone of said at least two sub-images.

[0050] According to another particular embodiment of the invention, the data stream comprises coded data of at least one view of a video, said at least one view corresponding to a view determined in the coding from at least one source view.

[0051] A source view is defined here as the original view of the scene represented by the multiview video, in other words, it is the sequence of images resulting from the capture of the scene by a camera.

[0052] According to this particular embodiment of the invention, at least one base view transmitted to the decoder is a view determined from one or more source views, which determination can be performed at coding time or before coding.

[0053] For example, it may be a view synthesized from multiple source views, or a view computed / estimated from the perspective of multiple sources.

[0054] For example, the determined view may correspond to a sequence of images, each image being larger than a camera image and comprising all images of the same moment from all cameras in a 360-degree capture camera array.

[0055] According to another particular embodiment of the invention, the decoding method further comprises a step of receiving texture data and depth data of said sub-image, said sub-image resulting from a synthesized intermediate view of a viewer's position and viewing direction within the viewing volume.

[0056] According to this particular embodiment of the invention, the data for the sub-images is provided to the decoder as a function of the position in the viewing volume and the viewing direction. Such a particular embodiment of the invention allows for the data for the sub-images to be provided on demand as the viewer navigates through the multi-view video. Thus, the data for the corresponding refinement or occlusion zones is better adapted to the viewer's position and viewing direction.

[0057] The invention also relates to a device for decoding at least one data stream representing a multiview video, said at least one data stream comprising at least coded data representing at least one sub-image, said coded data comprising texture data and depth data associated to said texture data, said sub-image comprising a zone, called useful zone, intended to be used to generate at least one image of at least one view, said decoding device comprising: - decoding from said at least one data stream an indicator indicating whether a sub-image is coded according to a first method or according to a second method; - if the indicator indicates that the subimage is coded according to the first method, decoding into the subimage a binary map indicating, for at least one pixel of the subimage, whether said pixel belongs to a useful zone; - decoding texture data and depth data of the sub-image, where if the indicator indicates that the sub-image is coded according to the second method, depth data of said sub-image for pixels located outside the useful zone comprises decoded depth values.

[0058] According to a particular embodiment of the invention, such a decoding device is provided in a terminal or in a server.

[0059] The present invention also provides - encoding at least one sub-image comprising texture data and depth data associated with said texture data, said sub-image comprising a useful zone intended to be used to generate at least one image of at least one view; - determining a coding method for coding the sub-image between a first method and a second method; - encoding an indicator that indicates whether the sub-image is coded according to the first method or according to the second method; - if the indicator indicates that the sub-image is coded according to the first method, coding in the sub-image a binary map indicating, for at least one pixel of the sub-image, whether said pixel belongs to a useful zone or not; - if the indicator shows that the sub-image is coded according to the second method, - determining an allocation depth value; - assigning an assigned depth value to the depth data of said sub-image for pixels of said sub-image located outside the useful zone; encoding the - coding texture data and depth data of sub-images.

[0060] According to a particular embodiment of the invention, such a coding device is provided in a terminal or in a server.

[0061] The invention also relates to a data stream representing a multiview video, comprising at least coded data representing at least one sub-image, the coded data comprising texture data and depth data associated with said texture data, said sub-image comprising zones, called useful zones, intended to be used to generate at least one image of at least one view, the data stream comprising: - an indicator showing whether the sub-image is coded according to the first method or according to the second method; - if the indicator indicates that the sub-image is coded according to the first method, coded data representing a binary map indicating, for at least one pixel of the sub-image, whether said pixel belongs to a useful zone or not; - depth data of said sub-image for pixels located outside the useful zone, comprising depth values ​​if the indicator indicates that the sub-image is coded according to the second method.

[0062] According to a particular embodiment of the invention, the data stream further comprises an item of information representative of the assigned depth value if the indicator shows that the sub-image is coded according to the second method.

[0063] According to another particular embodiment of the invention, the data stream comprises an indicator of said at least one sub-image indicating whether depth data is coded by obtaining a residual determined by the difference between the depth value of a pixel and said assigned depth value.

[0064] According to another particular embodiment of the invention, the data stream further comprises coded data of at least one view of the video, said at least one view corresponding to a view synthesized in the coding from at least one source view.

[0065] The decoding method, respectively the encoding method, according to the invention can be implemented in various ways, in particular in wired or software form. According to a particular embodiment of the invention, the decoding method, respectively the encoding method, is implemented by a computer program. The invention also relates to a computer program comprising instructions for implementing the decoding method or the encoding method according to any one of the particular embodiments when said program is executed by a processor. Such a program can use any programming language and can be downloaded from a communication network and / or recorded on a computer-readable medium.

[0066] The program may use any programming language and may be in the form of source code, object code, or any intermediate code between source code and object code, such as a partially compiled form or any other desired form.

[0067] The present invention also relates to a computer-readable storage medium or data medium comprising instructions for a computer program as described above. The recording medium can be any entity or device capable of storing a program. For example, the medium can comprise a storage means such as a ROM, e.g., a CD-ROM or a microelectronic circuit ROM, a USB flash drive, or a magnetic recording means, e.g., a hard drive. Alternatively, the recording medium can correspond to a transmissible medium, such as an electrical or optical signal, which can be conveyed via an electrical or optical cable, by radio, or other means. The program according to the present invention can be downloaded, in particular, over an Internet-type network.

[0068] Alternatively, the recording medium may correspond to an integrated circuit in which the program is embedded, the circuitry being adapted for performing, or for use in the performance of, the method in question.

[0069] Other features and advantages of the invention will become more clearly apparent on reading the following description of particular embodiments, given as a simple illustrative and non-limiting example, and the accompanying drawings, in which: [Brief explanation of the drawings]

[0070] [Figure 1] FIG. 1 is a diagram illustrating an example of a multi-view scene capture system. [Figure 2] FIG. 10 illustrates example inputs / outputs of a view synthesis module. [Figure 3] FIG. 1 shows a schematic diagram of an example of a multi-view encoder based on patch coding. [Figure 4] FIG. 2 illustrates steps of an encoding method according to a particular embodiment of the invention. [Figure 5] FIG. 10 shows an example of an image grouping sub-images to be coded. [Figure 6]FIG. 6 shows an example of an image with sub-images and a corresponding binary map showing useful and non-useful zones of the sub-images of the image shown in FIG. 5. [Figure 7] 10A-10C illustrate examples of filling in non-useful zones of a patch's texture, according to certain embodiments of the present invention. [Figure 8] 10A-10C illustrate examples of filling in non-useful zones of patch depth according to certain embodiments of the present invention. [Figure 9] FIG. 3 illustrates steps of a decoding method according to a particular embodiment of the invention. [Figure 10] FIG. 4 illustrates steps of a decoding method according to another particular embodiment of the invention. [Figure 11] FIG. 4 illustrates steps of a decoding method according to another particular embodiment of the invention. [Figure 12] FIG. 2 illustrates an example of an architecture of a coding device according to a particular embodiment of the present invention. [Figure 13] 1 illustrates an example of an architecture of a decoding device according to a particular embodiment of the present invention. [Figure 14] FIG. 2 illustrates an example data stream in accordance with certain embodiments of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0071] FIG. 4 illustrates steps of a method for coding at least one data stream representing a multiview video according to a particular embodiment of the invention.

[0072] According to the present invention, multi-view video is coded according to a coding scheme such as that shown in relation to Figure 3, in which one or more base views are coded into a data stream, and sub-images or patches comprising texture and depth data are also coded into the data stream, allowing a decoder to synthesize other views of the scene not coded into the data stream. The steps of such a coding scheme relevant to particular embodiments of the present invention are described below.

[0073] Here, for example, a scene is captured according to four viewpoints V0 to V3, and depth maps d0 to d3 associated with each of the images of the views are considered to have been previously estimated. Here, a view is defined as a sequence of images representing a scene captured from a viewpoint. Each view is therefore a video, i.e., a sequence of images, and a depth map is determined for each image of the view.

[0074] In step E40, the views V0 to V3 and the associated depth maps d0 to d3 are coded and decoded, for example by an HEVC encoder or any other suitable 2D or multiview encoder. These views are considered as base views and are transmitted to the decoder, where they can be used to synthesize intermediate views. As a variant, not all views V0 to V3 may be coded, but for example only one view, or two or three views may be coded and transmitted to the decoder.

[0075] According to a particular embodiment of the invention, the base view coded in the data stream in step E40 corresponds to a view synthesized from one or more views V0 to V3.

[0076] In step E41, an intermediate view Vi is synthesized using the coded views transmitted to the decoder. Typically, such an intermediate view Vi comprises texture data and depth data associated with the texture data.

[0077] In step E42, from this intermediate view Vi and the coded / decoded base view, one or more zones are extracted from the intermediate view Vi. The method is presented here for the case of a synthesized intermediate view Vi. Of course, the method applies equally if several intermediate views are synthesized and one or more zones are extracted from these intermediate views.

[0078] The purpose of the present invention is not to determine the zones to be extracted, and therefore methods for extracting these zones will not be developed further.

[0079] The extracted zones can correspond to occlusion zones, i.e., zones that are not displayed in the base view but are displayed in the intermediate view, or zones that are already present in the base view but provide insufficient synthesis quality in the synthesized view, for example due to distortions that appear during synthesis. In the latter case, the synthesized intermediate view corresponds to a view captured by a camera. The intermediate view from which the zones are extracted can be a view captured by a camera but not coded into the data stream. It can therefore be synthesized at the decoder side and refined by sub-images coded into the data stream.

[0080] The extracted zones are represented in the form of patches of arbitrary shape, hereafter referred to as sub-images, which comprise texture data, depth data, position within each view, and size.

[0081] It is understood that for an intermediate view, one or more sub-images can be extracted for the same image of the intermediate view. Similarly, sub-images representing the same zone extracted from the intermediate view can be extracted for different images of the intermediate view. In this way, the texture and depth data of the sub-images change over time to form a sequence of sub-images.

[0082] An example of sub-images P0 to P5 extracted from an image of an intermediate view Vi is shown in Figure 5. In the example of Figure 5, the sub-images are grouped into a rectangular image 50 to facilitate their coding. It will be appreciated that, depending on the variants of the embodiment of such an image, image 50 may comprise sub-images extracted from different images of the intermediate view, or from a single image, or from sub-images extracted from the intermediate view Vi, and also from other intermediate views.

[0083] Similarly, sub-images can be coded without being grouped into rectangular images, and each sub-image is then treated as an image or sequence of images as the sub-images change over time.

[0084] In Figure 5, if the zones extracted from the intermediate view are represented in a form suitable for coding (rectangles or at least quadrilaterals), the sub-images appear to comprise useful data (51, the heavily hatched zones in Figure 5) that correspond to the extracted zones and are likely to be used for synthesizing the intermediate view or other views, as well as non-useful data (52, the finely hatched zones in Figure 5) with zero texture and depth data.

[0085] The useful data of a sub-image may be comprised in one zone of the sub-image or in several disjoint zones of the sub-image, so that the sub-image comprises several zones that are useful for compositing.

[0086] The coding method according to the invention aims to reduce the coding costs of non-useful zones.

[0087] Below, the invention will be described when the sub-images are considered to be grouped into larger images 50. However, the general principles of the invention apply in a similar manner when the sub-images are not grouped into larger images but are processed individually.

[0088] In step E42, the sub-images are coded. For this purpose, according to the invention, two methods for coding the sub-images can be used. A competition between the two coding methods is implemented in order to select the coding method that offers the best rate / distortion compromise for a given sub-image. An indicator is coded into the stream to indicate which coding method is used for a sub-image.

[0089] Two encoding methods are described below.

[0090] According to a first coding method, the sub-image is coded using a binary map that indicates, for a given pixel of the sub-image, whether the pixel belongs to a useful zone (e.g., the pixel takes the value 1 in the binary map) or does not belong to a useful zone (e.g., the pixel takes the value 0 in the binary map).

[0091] FIG. 6 shows a sub-image and a useful (51) of the sub-image of the image 50 shown in FIG. 5, a texture image 50 text In the hatched zone) zone and not useful (52, texture image 50 text The white zones in the corresponding binary map (50 bin ) with images (50 text ) shows an example of a first coding method where only the texture and depth data of pixels located in the useful zone are coded and transmitted to the decoder, and the associated binary map is coded for all pixels of the sub-image.

[0092] It should be noted here that predefined values, such as 0, which specify that the pixel in question is not useful, cannot be used as references in the texture data. Indeed, after encoding and decoding the texture, the pixel value is not guaranteed and may not be zero. Furthermore, other null values ​​may appear in pixels that belong to the useful zone.

[0093] According to the second coding method, the sub-image is coded by filling in the non-useful zones of both the texture and depth data, and then the sub-image is coded in the standard way by coding all pixels of the sub-image.

[0094] For this purpose, in step E420, the coding method to be used to code the sub-image is determined between the first and second methods described above. For example, a coding competition between the two methods is implemented and the method offering the best rate / distortion compromise is selected. In known methods, such a competition is performed by simulating the coding according to each of the two methods and calculating the rate used by each method and the associated distortion calculated between the original and reconstructed values ​​of the texture and / or depth data.

[0095] In step E421, an indicator is coded into the data stream indicating whether the sub-image has been coded according to the first method or according to the second method. If the indicator indicates that the sub-image has been coded according to the first method, the method proceeds to step E422. If the indicator indicates that the sub-image has been coded according to the second method, the method proceeds to step E424.

[0096] In step E422, the binary map of the sub-image is coded, for example using entropy coding.

[0097] In step E423, texture and depth data are coded for the sub-image, taking into account only pixels located in the useful zone of the sub-image. These pixels are identified using a binary map. The texture and depth data are coded in a standard manner, for example by an HEVC encoder or any other 2D or 3D encoder.

[0098] In step E424, if the indicator indicates that the sub-image is coded according to the second method, the assigned depth value d * is determined.

[0099] For example, the allocation depth value d * is the minimum depth value d determined over at least the useful zone of the sub-image min and reduced by a predetermined value delta. min corresponds to the minimum depth value of the pixels of the sub-image that belong to the useful zone, i.e., the depth of the pixels of the sub-image that belong to the useful zone, and the furthest value is d min =min(d i ), where d i corresponds to the depth value of the pixels in the useful zone, and min corresponds to the function returning the minimum value.

[0100] This minimum value d min is reduced by a value delta to account for the quantization error introduced in encoding the depth value. For example, the assigned value d * is d * =d min Obtained by -delta.

[0101] Various values ​​of delta are possible, for example delta equals 10.

[0102] According to certain embodiments of the present invention, the minimum depth value is determined from the useful zones of several sub-images. For example, if the sub-images are grouped into a larger image, the useful zones of all sub-images are scanned to determine the minimum depth value among all sub-images of the larger image. Such certain embodiments of the present invention allow for more consistent assigned depth values ​​for all sub-images of the larger image.

[0103] In step E425, the assigned depth value d *is assigned as a depth value to pixels located outside the useful zone. Such a step is illustrated in FIG. 8, which represents the depth data of a sub-image. Pixels in the useful zone (51) are provided with a depth value estimated for these pixels when extracting the useful zone from the intermediate view. Pixels in the non-useful zone (52) are provided with an assigned depth value d * Equipped with.

[0104] In step E426, texture values ​​are also assigned to pixels located outside the useful zone.

[0105] For example, non-useful zones are filled by mirroring. As shown in Figure 7, the texture values ​​of pixels outside the useful zone are calculated to best reproduce the data present in the useful zone. In Figure 7, a sub-image is shown. The data of the useful zone (51) is propagated to the non-useful zone (52) symmetrically with respect to the boundary (53) between the useful and non-useful zones.

[0106] According to another example, the texture value of the pixels located outside the useful zone (52) corresponds to the average of the texture values ​​of the pixels of the useful zone (51). In this example, the pixels of the non-useful zone are filled with the same value.

[0107] In step E427, the texture and depth data of the sub-image are coded taking into account all pixels of the sub-image. In other words, according to this second coding method, the pixels of the useful zone and the pixels of the non-useful zone are coded. The texture and depth data are coded in a standard manner, for example by an HEVC encoder or any other 2D or 3D encoder.

[0108] According to a particular embodiment of the present invention, the depth data is represented by an assigned depth value d *According to this particular embodiment of the invention, the prediction residual is then res(x,y)=prof(x,y)−d * where prof(x,y) is the depth value of pixel (x,y) in the sub-image and res(x,y) is the residual value of pixel (x,y) in the sub-image.

[0109] Therefore, when coding the depth data of a sub-image, it is the residual as described above that is coded.

[0110] According to this particular embodiment of the invention, in order for the decoder to reconstruct the depth data of the sub-image from the residual, the decoder assigns a depth value d * You need to be aware of this.

[0111] Thus, according to this particular embodiment of the invention, in step E428, the allocated depth value d * is encoded into the data stream.

[0112] According to a variant of this particular embodiment of the invention, a differential coding of the depth data is placed in competition with a conventional coding of the depth data. Thus, according to this variant, in step E420, when testing the sub-image coding according to the second method, it is also determined whether the depth data according to this second coding method is coded by residuals. In order to select the coding mode that offers the best rate / distortion compromise, a residual-based coding of the depth data and a conventional coding of the depth data are simulated.

[0113] According to a variant of this embodiment of the invention, an item of information indicating which coding mode of the depth data has been selected needs to be coded into the data stream. Therefore, in step E429, a depth data coding indicator is coded into the data stream of the sub-image. Such an indicator indicates whether the depth data has been coded by the residual.

[0114] 9 illustrates steps of a method for decoding a data stream representing a multi-view video according to a particular embodiment of the invention, for example generated by the coding method described in relation to FIG.

[0115] According to the present invention, the data stream comprises at least coded data related to the coding of one or more base views of the multiview video, and coded data representing sub-images or patches comprising texture and depth data. When decoded, these sub-images allow, for example, a decoder to synthesize other views of the scene not coded in the data stream. Steps of such a decoding scheme relevant to particular embodiments of the present invention are described below.

[0116] In step E90, the base view and the associated depth map are decoded from the data stream.

[0117] According to certain embodiments of the invention, the base view decoded from the data stream corresponds to a view synthesized at the encoder from one or more captured views.

[0118] The sub-images coded in the data stream are then decoded. A sub-image is defined here as a sub-image representing a useful zone extracted at the encoder from an intermediate view. This sub-image can be grouped with other sub-images in a larger image.

[0119] In step E91, an indicator is decoded which indicates whether the sub-image has been coded according to the first or second method described above.

[0120] If the indicator shows that the sub-image has been coded according to the first method, the method continues to step E92. If the indicator shows that the sub-image has been coded according to the second method, the method continues to step E94.

[0121] In step E92, a binary map is decoded for the sub-image, such a binary map indicating, for a given pixel of the sub-image, whether or not the pixel belongs to a useful zone.

[0122] In step E93, the texture and depth data of the sub-image are decoded and reconstructed taking into account only the pixels located in the useful zone of the sub-image, which are identified using a binary map.

[0123] At the end of the reconstruction, the pixels of the sub-image lying outside the useful zone are provided with a texture and depth value of 0.

[0124] If the indicator shows that the sub-image is coded according to the second method, then in step E94 the texture and depth data of the sub-image are decoded as follows.

[0125] In step E940, the texture and depth data of the sub-image are decoded and reconstructed in the standard way, taking into account all pixels of the sub-image. At the end of the reconstruction, all pixels of the sub-image are provided with texture and depth values. Pixels of the sub-image located outside the useful zone are provided with texture and depth values ​​determined during coding.

[0126] According to a particular embodiment of the invention, if the indicator indicates that the sub-image is coded according to the second method, the decoder does not distinguish between useful and non-useful zones of the sub-image.

[0127] However, the decoded depth values ​​of pixels in the non-useful zones of the sub-images prevent the synthesis algorithm from using these pixels when synthesizing intermediate views. Indeed, according to a particular embodiment of the invention, such pixels are assigned a depth value determined from the minimum depth value of the zones of the sub-images, reduced by a value that allows to take into account quantization errors. In this way, these pixels are considered by the synthesis algorithm as the furthest points.

[0128] According to another particular embodiment of the invention, in step E941, the assigned depth value d assigned at the time of coding to the pixels located outside the useful zone of the sub-image is * An item of information representing is decoded from the data stream.

[0129] According to this particular embodiment of the invention, a binary map can therefore be constructed for the sub-images to identify pixels in useful and non-useful zones, for example, if such a map is required for a compositing algorithm.

[0130] In a variation of this particular embodiment, the depth data of the sub-images is assigned depth values ​​d * were differentially coded.

[0131] Thus, according to a variant of this embodiment, in step E943, the depth data of the sub-image is calculated by assigning the depth value d decoded to the depth residual value of the sub-image decoded in step E940. * is reconstructed by adding

[0132] According to another variant, in step E942, a depth data coding indicator of the sub-image is decoded from the data stream, such coding indicator indicating whether the depth data is differentially coded or not.

[0133] According to a particular embodiment of the invention, the decoder operates in a client-server mode, with the server providing data for the multi-view video that the viewer is navigating.

[0134] According to this particular embodiment of the invention, the decoding method comprises a preliminary step E901 for receiving texture and depth data of a sub-image, for example following a request issued by the decoder for a position and viewing direction corresponding to or close to the viewer's position and viewing direction in the viewing volume.

[0135] 10 illustrates the steps of a decoding method according to another particular embodiment of the invention. According to this particular embodiment of the invention, the depth values ​​of pixels of the reconstructed sub-images that do not belong to useful zones are assigned a value of infinity, so that the synthesis algorithm can distinguish between useful and non-useful zones.

[0136] For this purpose, in step E100, the useful zone of the reconstructed sub-image is determined. If the indicator decoded in step E91 indicates that the sub-image has been coded according to the second method, the useful zone is determined by the reconstructed depth data and the decoded assigned depth value d * If the indicator decoded in step E91 indicates that the sub-image is coded according to the first method, the useful zone is determined from the decoded binary map in step E92.

[0137] In step E101, a depth map of the sub-image is constructed, in which the pixels of the useful zone take their decoded depth value and the pixels located outside the useful zone are assigned the value of infinity.

[0138] In step E102, the depth map is sent to a synthesis module.

[0139] FIG. 11 illustrates the steps of a decoding method according to another particular embodiment of the invention.

[0140] If the indicator decoded in step E91 indicates that the sub-image is coded according to the second method, then in step E110 the pixels of the useful zone are coded according to the reconstructed depth data and the decoded assigned depth value d * is determined from

[0141] In step E111, a binary map is constructed for the sub-image. Such a binary map indicates, for a given pixel of the sub-image, whether the pixel belongs to a useful zone or not. Such a binary map is similar to the binary map that would have been decoded if the sub-image had been coded according to the first method.

[0142] In step E112, the binary map is sent to a synthesis module.

[0143] FIG. 12 shows a simplified structure of a coding device COD adapted to implement a coding method according to any one of the particular embodiments of the invention.

[0144] According to a particular embodiment of the invention, the steps of the coding method are implemented by computer program instructions. To this end, the coding device COD has a standard architecture of a computer, in particular a processing unit UT comprising a memory MEM, for example a processor PROC, and driven by a computer program PG stored in the memory MEM. The computer program PG comprises instructions for implementing the steps of such a coding method when the program is executed by the processor PROC.

[0145] At initialization, the code instructions of the computer program PG are loaded, for example, into a RAM memory (not shown) before being executed by the processor PROC. In particular, the processor PROC of the processing unit UT implements the steps of the coding method described above according to the instructions of the computer program PG.

[0146] FIG. 13 shows a simplified structure of a decoding device DEC adapted to implement the decoding method according to any one of the particular embodiments of the invention.

[0147] According to a particular embodiment of the invention, the decoding device DEC has a standard architecture of a computer and in particular comprises a memory MEM0, a processor PROC0, and a processing unit UT0 driven by a computer program PG0 stored in the memory MEM0, the computer program PG0 comprising instructions for implementing the steps of the decoding method as described above when the program is executed by the processor PROC0.

[0148] At initialization, the code instructions of the computer program PG0 are loaded into a RAM memory (not shown) before being executed by the processor PROC0, for example. In particular, the processor PROC0 of the processing unit UT0 implements the steps of the above-described decoding method according to the instructions of the computer program PG0.

[0149] 14 shows an example of a data stream STR according to a particular embodiment of the invention, for example, the data stream STR is generated by the encoding method described in connection with FIG.

[0150] Referring to FIG. 14, the data stream STR represents a multi-view video, specifically a coded data V representing one or more base views. band coded data representing at least one sub-image comprising texture data (text0, text1) and depth data (dpth0, dpth1) associated with said texture data.

[0151] According to a particular embodiment of the invention, the base views coded in the data stream correspond to views synthesized in the coding from at least one source view of the scene.

[0152] The data stream STR shown in Figure 14 is an illustrative example, presented here as comprising two sub-images (Im0, Im1). The sub-images comprised in the data stream comprise useful zones intended to be used to generate at least one image of at least one view of a scene captured by multi-view video.

[0153] According to various specific embodiments of the invention, the data stream STR comprises the following data for a sub-image: - indicator (Id_meth0, Id_meth1) of whether the subimage is coded according to the first or second method described above, - if the indicator indicates that the subimage is coded according to the first method (Id_meth1 in the example described here), coded data representing a binary map (valid1) indicating, for a given pixel of the subimage, whether said pixel belongs to a useful zone or not.

[0154] According to a particular embodiment of the invention, if the indicator indicates that the sub-image is coded according to the second method (Id_meth0 in the example described here), the data stream STR is coded according to the assigned depth value d * It has information items that represent the following:

[0155] According to another particular embodiment of the invention, if the indicator indicates that the sub-image is coded according to the second method, the data stream STR includes the depth values ​​of the pixels and the assigned depth values ​​d * and a sub-image coding indicator (ld_res) that indicates whether the depth data is coded by taking a residual determined by the difference between

[0156] The above examples are merely illustrative and non-limiting. The coded data described above may be provided in the same data stream or in multiple data streams or sub-streams of the same data stream. Also, some of the data described above, and other data not described, may be provided in metadata associated with one or more data streams.

[0157] For example, texture and depth data can be coded into separate streams.

[0158] According to another example, the coded data of the base view may be provided in a video data stream and the coded data relating to the sub-image may be provided in metadata associated with the video data stream. [Explanation of symbols]

[0159] 50 images 51 Useful Data, Useful Zones 52 Unusable data, unusable zones 53 Boundary

Claims

1. 1. A method for decoding at least one data stream representing a multiview video, wherein the at least one data stream comprises at least coded data representing at least one view, called a base view, of the multiview video and at least one sub-image, wherein the coded data of the at least one base view comprises texture data and depth data associated with the texture data, the sub-image comprises at least one zone, called a useful zone, and one zone, called a non-useful zone, wherein the useful zone corresponds to a zone extracted from a view of the multiview video, called an intermediate view, that is not coded in the data stream, and the coded data of the at least one sub-image comprises texture data and depth data associated with the texture data, and the useful zone is intended to be used to generate at least one image of at least another view, the method for decoding comprising: - decoding from said at least one data stream an indicator indicating whether said sub-image is coded according to a first method or according to a second method; - if the indicator indicates that the sub-image is coded according to the first method, decoding for the sub-image a binary map indicating, for at least one pixel of the sub-image, whether the pixel belongs to the useful zone or not, and if the pixel belongs to the useful zone, decoding, for the sub-image, texture data and depth data associated with the texture data; - decoding the texture data and the depth data of the sub-image, where if the indicator indicates that the sub-image is coded according to the second method, the texture data of the sub-image for the pixels located outside the useful zone comprise assigned texture values ​​and the depth data of the sub-image for the pixels located outside the useful zone comprise assigned depth values; 11. A method for decoding, comprising:

2. 2. The method for decoding according to claim 1, further comprising the step of decoding an item of information representing the assigned depth value if the indicator indicates that the sub-image is coded according to the second method, wherein the assigned depth value corresponds to the decoded depth value for the pixel located outside the useful zone.

3. 3. The method for decoding of claim 2, wherein the depth data of the sub-image coded into the data stream comprises, for at least one pixel of the sub-image, a residual determined by the difference between the depth value of the pixel and the assigned depth value.

4. 4. The method for decoding according to claim 3, comprising the step of decoding, in the data stream, an indicator indicating whether, for the at least one sub-image, the depth data is coded by a residual determined by a difference between the depth value of the pixel and the assigned depth value.

5. - determining the useful zone from reconstructed depth data and the item of information representing the decoded assigned depth values ​​if the indicator indicates that the subimage is coded according to the second method, or from a decoded binary map if the indicator indicates that the subimage is coded according to the first method; - constructing a depth map of said sub-image in which said pixels located outside said useful zone are assigned a value of infinity; - transmitting the depth map to a synthesis module; The method for decoding according to any one of claims 2 to 4, further comprising:

6. If the indicator indicates that the sub-image is coded according to the second method, - determining said useful zone from the reconstructed depth data and said items of information representing said decoded assigned depth values; - constructing, for at least one pixel of said sub-image, a binary map indicating whether said pixel belongs to said useful zone; - sending said binary map to a synthesis module; The method for decoding according to any one of claims 2 to 4, further comprising:

7. 7. A method for decoding according to claim 1, wherein the at least one base view of the multiview video corresponds to a view determined in coding from at least one source view.

8. 8. A method for decoding as claimed in any one of claims 1 to 7, further comprising a step of receiving the texture data and the depth data of the sub-image, the sub-image resulting from a synthesized intermediate view of a viewer's position and viewing direction within a viewing volume.

9. 1. A method for coding a data stream representing a multiview video, comprising the step of coding at least one view, called a base view, of a multiview video and at least one sub-image, wherein the data stream comprises coded data of the at least one base view and of the sub-image, the coded data of the base view comprising texture data and depth data associated with the texture data, the at least one sub-image comprising a zone, called a useful zone, and a zone, called a non-useful zone, the useful zone corresponding to a zone extracted from a view of the multiview video, called an intermediate view, not coded in the data stream, the coded data of the at least one sub-image comprising texture data and depth data associated with the texture data, the useful zone being intended to be used to generate at least one image of at least another view, wherein coding the at least one sub-image comprises: - determining a coding method for coding said sub-image between a first method and a second method; - encoding an indicator that indicates whether the sub-image is coded according to the first method or according to the second method; - if the indicator indicates that the sub-image has been coded according to the first method, coding for the sub-image a binary map indicating, for at least one pixel of the sub-image, whether the pixel belongs to the useful zone or not, and if the pixel belongs to the useful zone, coding for the sub-image texture data and depth data associated with the texture data; - if the indicator indicates that the subimage is coded according to the second method, determining assigned texture values ​​and assigning the assigned texture values ​​to the texture data of the subimage for the pixels of the subimage located outside the useful zone, and determining assigned depth values ​​and assigning the assigned depth values ​​to the depth data of the subimage for the pixels of the subimage located outside the useful zone; - encoding the texture data and the depth data of the sub-image. A method for coding.

10. 10. A method for coding according to claim 9, wherein the assigned depth value is determined over at least the useful zone of the sub-image and is determined from a minimum depth value reduced by a predetermined value.

11. 11. A method for encoding as claimed in claim 9 or 10, further comprising the step of coding an item of information representative of the assigned depth value if the indicator shows that the sub-image has been coded according to the second method, the assigned depth value corresponding to the coded depth value for the pixels located outside the useful zone.

12. 12. The method for encoding of claim 11, wherein the depth data of the sub-image coded into the data stream comprises, for at least one pixel of the sub-image, a residual determined by the difference between the depth value of the pixel and the assigned depth value.

13. 13. The method for encoding according to claim 12, comprising the step of encoding in the data stream an indicator indicating, for the at least one sub-image, whether the depth data is coded by a residual determined by a difference between the depth value of the pixel and the assigned depth value.

14. 11. The method for encoding according to claim 10, wherein at least two useful zones belonging to at least two sub-images respectively are coded into the data stream, the at least two useful zones being intended to be used to generate the at least one image of another view, and the minimum depth value is determined from the at least two useful zones.

15. 1. A device for decoding at least one data stream representing a multiview video, said at least one data stream comprising at least coded data representing at least one view, called a base view, of a multiview video and at least one sub-image, said coded data of said at least one base view comprising texture data and depth data associated to said texture data, said sub-image comprising zones, called useful zones, and zones, called non-useful zones, said useful zones corresponding to zones extracted from views of the multiview video, called intermediate views, not coded in said data stream, said coded data of said at least one sub-image comprising texture data and depth data associated to said texture data, said useful zones being intended to be used to generate at least one image of at least another view, said device for decoding comprising: - decoding from said at least one data stream an indicator indicating whether said sub-image is coded according to a first method or according to a second method; - if the indicator indicates that the sub-image is coded according to the first method, decoding for the sub-image a binary map indicating, for at least one pixel of the sub-image, whether the pixel belongs to the useful zone or not, and if the pixel belongs to the useful zone, decoding for the sub-image texture data and depth data associated with the texture data; - decoding the texture data and the depth data of the sub-image, such that if the indicator indicates that the sub-image is coded according to the second method, the texture data of the sub-image for pixels located outside the useful zone comprise assigned texture values ​​and the depth data of the sub-image for pixels located outside the useful zone comprise assigned depth values; A device configured to:

16. 1. A device for coding a data stream representing a multiview video, the device being configured to code at least one view, termed a base view, of a multiview video and at least one sub-image, the data stream comprising coded data of the at least one base view and of the sub-image, the coded data of the base view comprising texture data and depth data associated with the texture data, the sub-image comprising zones, termed useful zones, and zones, termed non-useful zones, the useful zones corresponding to zones extracted from views of the multiview video, termed intermediate views, not coded in the data stream, the coded data of the at least one sub-image comprising texture data and depth data associated with the texture data, the useful zones being intended to be used to generate at least one image of at least another view, - determining a coding method for coding said sub-image between a first method and a second method; - encoding an indicator that indicates whether the sub-image is coded according to the first method or according to the second method; - if the indicator indicates that the sub-image has been coded according to the first method, coding for the sub-image a binary map indicating, for at least one pixel of the sub-image, whether the pixel belongs to the useful zone or not, and if the pixel belongs to the useful zone, coding for the sub-image texture data and depth data associated with the texture data; - if the indicator indicates that the subimage is coded according to the second method, determining assigned texture values ​​and assigning the assigned texture values ​​to the texture data of the subimage for the pixels of the subimage located outside the useful zone; determining assigned depth values ​​and assigning the assigned depth values ​​to the depth data of the subimage for the pixels of the subimage located outside the useful zone; - encoding the texture data and the depth data of the sub-image; A device comprising:

17. 15. A computer program comprising instructions for implementing the method for decoding according to any one of claims 1 to 8 and / or the method for encoding according to any one of claims 9 to 14, wherein the computer program is executed by a processor.

Images