Method and device for encoding and decoding multi-view video sequence

Abstract

To provide a method for decoding a data stream representing a multi-view video, and a method for encoding the data stream.SOLUTION: A method for decoding a data stream includes at least coded data representing at least one sub-image, and includes texture data and depth data associated with the texture data, and the sub-image includes at least one zone. An indicator is decoded, indicating whether the sub-image is coded according to a first method or a second method. When the indicator indicates that the sub-image is coded according to the first method, a binary map is decoded, indicating, for at least one pixel of the sub-image, whether or not the pixel belongs to a useful zone. When the indicator indicates that the sub-image is coded according to the second method, the depth data of the sub-image for the pixels located outside the useful zone includes a decoded depth value.SELECTED DRAWING: None

Description

[Technical field]

[0001] The present invention relates to immersive video representing scenes captured by one or more cameras, and more particularly to the coding and decoding of such video. [Background technology]

[0002] In the context of immersive video, i.e. where the viewer feels immersed in the scene, the scene is usually captured by a set of cameras, as shown in Figure 1. These cameras can be of type 2D (cameras C1, C2, C3, C4 in Figure 1) or of type 360, i.e. capturing the entire scene in 360 degrees around the camera (camera C5 in Figure 1).

[0003] All these captured views are conventionally encoded and then decoded by the viewer's terminal. However, to provide a sufficient quality experience, and thus a good sense of visual quality and immersion in the scene displayed to the viewer, displaying only the captured views is insufficient.

[0004] To improve the sense of immersion in the scene, one or more views, called intermediate views, are usually computed from the decoded views.

[0005] These intermediate views can be computed by a view "synthesis" algorithm. An example of a synthesis algorithm is shown in Figure 2, which mainly shows the inputs / outputs of the view synthesis algorithm. According to this example, from n views (here 1 to 4), the synthesis algorithm can synthesize a viewpoint "v" located in a given volume, called the viewing volume, depending on the camera position. The n views and the synthesized view are computed using texture data (T 1 ~T 4 , T v ) and depth data (D1 ~D 4 , D v ) Depth data is typically represented in the form of a depth map, i.e. an image that indicates, at each point in the image, the depth associated with the object represented at that point.

[0006] These depth maps are either captured by a particular camera or computed from texture views. Due to the large amount of data involved, it is difficult to compress the captured views to represent the scene.

[0007] Furthermore, since the captured views are needed to synthesize the intermediate views, a compromise is required between view coding efficiency and the quality of the intermediate view synthesis.

[0008] MV-HEVC and 3D-HEVC encoders are two extensions of the 2D HEVC encoder adapted for coding multiview video ("High Efficiency Video Coding, Coding Tools and Specification", Matthias Wien, Signals and Communication Technology, 2015).

[0009] Such an encoder takes as input a set of views and optionally associated depth maps. MV-HEVC encoders do not apply any specific coding tools at the block level. When coding an image of the current view, they simply use images of another view as reference images in addition to previously coded / decoded images of the current view.

[0010] The 3D-HEVC encoder adds specific tools to the depth map coding, uses more advanced inter-view and inter-component redundancy, and the depth map coding leverages texture coding.

[0011] However, for 360-degree multiview content, the redundancy between views does not seem to be properly exploited by these two codecs. Thus, their coding efficiency is far from optimal. Moreover, nothing is done to preserve zones that may lead to occlusions that affect the quality of the image of the synthesized view.

[0012] FIG. 3 shows another method for coding multiview video and synthesizing intermediate views at the decoder side.

[0013] According to this method, one or more base views (T in FIG. 3) are b , D b ) is coded with a 2D encoder, e.g., an HEVC encoder, or a multiview encoder.

[0014] Other Views (T s , D s ) are processed to extract specific zones from each of these views.

[0015] If not all captured views are sent to the decoder, or if the cameras are naturally far apart, occlusion zones increase. These zones correspond to objects that are hidden in the views sent to the decoder, but are needed to reconstruct the image from any viewpoint within the viewing volume. Thus, the extracted zones can correspond to occlusion zones that are hidden in the base views, but are visible in views that are not sent to the decoder.

[0016] Subsampling the views can introduce significant artifacts into view synthesis, especially when objects are highly distorted depending on the viewing position, i.e., the synthesized viewpoint.

[0017] According to the method shown in Fig. 3, the extracted zones, also called patches in the following, are coded, for example, by a conventional 2D video encoder. Then, at the decoder side, a synthesis algorithm can generate one or more intermediate views from the base view and the decoded patches.

[0018] Overall, the patch can transmit information to the decoder related to the same zone that can be seen from the perspective of the base view or from another perspective, thus taking into account changes in lighting between the two viewpoints or, for example, reductions in projection quality related to the object itself, for example in the case of tilted objects (over a certain depth range).

[0019] In order to code the patches representing the extracted zones, it is necessary to code the depth associated with the texture of these patches. To simplify the coding and representation of these patches, which may have different shapes, the patches are generally stored in the form of rectangular images, or at least in the form of quadrilaterals, also called sub-images.

[0020] Expressed in the form of a sub-image, the sub-image corresponds to an extracted zone and appears to comprise useful data that is likely to be used for the synthesis of an intermediate view, and non-useful data that does not belong to the patch originally extracted.

[0021] For efficient coding of patches, the impact of the coding of non-useful data on the coding of useful data needs to be minimized.

[0022] Furthermore, at the decoder side, an item of information needs to be transmitted to the synthesis algorithm that helps it to distinguish between useful and non-useful zones, so as not to use non-useful zones during synthesis.

[0023] Therefore, there is a need to improve upon the prior art. [Prior art documents] [Non-patent literature]

[0024] [Non-Patent Document 1] "High Efficiency Video Coding, Coding Tools and Specification", Matthias Wien, Signals and Communication Technology, 2015 Summary of the Invention [Means for solving the problem]

[0025] The present invention improves upon the prior art. To this end, the invention concerns a method for decoding a data stream representing a multi-view video, the data stream comprising at least coded data representing at least one sub-image, the coded data comprising texture data and depth data associated with said texture data, the sub-image comprising at least one zone, called useful zone, intended to be used for generating at least one image of at least one view.

[0026] Such a decoding method is - decoding from said at least one data stream an indicator of whether a sub-image is coded according to a first method or according to a second method; - if the indicator indicates that the subimage is coded according to the first method, decoding into the subimage a binary map indicating, for at least one pixel of the subimage, whether said pixel belongs to a useful zone or not; - decoding texture and depth data of the sub-image, where if the indicator indicates that the sub-image is coded according to a second method, depth data of said sub-image for pixels located outside the useful zone comprises decoded depth values.

[0027] Thus, according to the invention, it is possible to code and decode sub-image coding zones that are useful for synthesis while limiting the coding cost of these sub-images. In particular, depending on the value of an indicator coded in the stream, the useful zones of a sub-image representing a patch can be determined by the synthesis algorithm when reconstructing the view from the depth information coded in the stream of sub-images or from a binary map coded and decoded from the stream.

[0028] Thus, according to the present invention, unlike the prior art, there is no need to systematically code a validity binary map for identifying the useful zone of a sub-image for the sub-image. For example, in the encoder, a competition can be implemented to determine the optimal coding in terms of rate / distortion of the depth information of the patch. Such a competition compares the performance of a first method of coding a binary map indicating which pixels belong to the useful zone and are sent to the synthesis algorithm when synthesizing the view with the performance of a second method of assigning a specific value to the depth values ​​of pixels located outside the useful zone.

[0029] When a sub-image is coded according to the first method, only the depth values ​​of the pixels that belong to the useful zone are coded and decoded, whereas when a sub-image is coded according to the second method, the depth values ​​of all pixels of the sub-image are coded and decoded, regardless of whether these pixels belong to the useful zone or not. Pixels of the sub-image that do not belong to the useful zone have a specific decoded depth value that was determined at the time of coding.

[0030] Such a particular value ensures that the synthesis algorithm does not use pixels that do not belong to the useful zone when synthesizing the view.

[0031] A sub-image may comprise texture and depth data related to occlusion zones that are not displayed in the views coded in the data stream, and coding such a sub-image thus improves the quality of synthesis of views that are not coded in the data stream.

[0032] In another embodiment variant, a sub-image may comprise texture and depth data relating to a zone that is visible from a view coded in the data stream but that introduces significant deficiencies into the synthesis of the view, and coding such a sub-image improves the quality of the synthesis of this view by providing texture and depth refinements to the zone.

[0033] The proposed invention allows for better adaptation to the view content and occlusion type. Specifically, the first method preserves the sharp contours of occlusion zones during synthesis, while the second method saves coding costs by avoiding the transmission of binary maps. However, coding according to the second method does not preserve the sharpness of the contours of occlusion zones, since no distinction is made between useful and non-useful zones in the sub-images.

[0034] The selection in the encoder between the first and second method can be done, for example, via rate / distortion optimization, where the distortion is measured in the encoder from a partial synthesis of the zone reconstructed by the useful zone.

[0035] The invention also relates to a method for coding a data stream representing a multiview video, comprising the steps of: - coding at least one sub-image comprising texture data and depth data associated to said texture data, said sub-image comprising a zone, called useful zone, intended to be used for generating at least one image of at least one view, The step of encoding at least one sub-image comprises: - determining a coding method for coding the sub-images between a first method and a second method; - coding an indicator of whether the sub-image has been coded according to the first method or according to the second method; - if the indicator indicates that the subimage is coded according to the first method, coding in the subimage a binary map indicating, for at least one pixel of the subimage, whether said pixel belongs to a useful zone or not; - if the indicator shows that the sub-picture is coded according to the second method, - determining an allocation depth value; - for pixels of a sub-image located outside a useful zone, assigning an assigned depth value to the depth data of said sub-image; - coding the texture and depth data of the sub-image.

[0036] According to a particular embodiment of the invention, the assigned depth value is determined at least over the useful zone of the sub-image and is determined from a minimum depth value reduced by a predetermined value.

[0037] The minimum depth value is for example the depth value of the furthest (depth-wise) point of the useful zone in the sub-image. The minimum depth value is reduced by a predetermined value in order to obtain a specific depth value different from the depth values ​​of the points of the useful zone. The reduction by a predetermined value thus makes it possible to take into account quantization errors when encoding the depth data.

[0038] Furthermore, using specific depth values ​​that remain close to the depth values ​​of the points of the useful zone limits the impact of the depth coding cost of the sub-images: in fact, the occurrence of strong discontinuities in the depth information coded for the sub-images is limited.

[0039] Moreover, such assigned depth values ​​can be used advantageously by the compositing algorithm. Indeed, in case of conflicts during the compositing of images, the compositing algorithm will naturally give preference to objects located in the foreground, which will therefore have a greater depth value than objects located in the background. Thus, using a lower depth value for pixels of the sub-images located in the non-useful zone than the depth values ​​of the pixels of the useful zone indicates to the compositing algorithm that the pixels with the smallest depth value do not belong to the useful zone, and therefore will not be given priority for their consideration by the compositing algorithm in case of conflicts when reconstructing the composite image.

[0040] According to another particular embodiment of the invention, if the indicator indicates that the sub-image has been coded according to the second method, the decoding method, respectively the coding method, comprises the step of decoding, respectively coding, an item of information representative of an assigned depth value, the assigned depth value corresponding to the decoded, respectively coded, depth value for pixels located outside the useful zone.

[0041] Thus, according to this particular embodiment of the invention, a binary map indicating the pixels belonging to the useful zone can be created for a sub-image coded according to the second method, since the decoder knows the assigned depth values ​​assigned to the pixels located outside the useful zone. It is therefore possible to identify from the depth values ​​of the decoded pixels which pixels belong to the useful zone and which pixels are located outside the useful zone. The binary map thus constructed can be sent to a compositing algorithm. The information provided as input to the compositing algorithm is therefore identical regardless of whether the sub-image was coded according to the first method or according to the second method.

[0042] According to another particular embodiment of the invention, the depth data of a sub-image coded in the data stream comprises, for at least one pixel of the sub-image, a residual determined by the difference between the depth value of said pixel and the assigned depth value.

[0043] This particular embodiment of the invention reduces the coding cost of the depth data of a sub-image when the sub-image is coded according to the second method: indeed, when a sub-image only comprises background information, the dynamics of the depth data is low and it is more advantageous to code this data as a residual.

[0044] According to another particular embodiment of the invention, an indicator indicating whether depth data is coded by a residual determined by the difference between the depth value of the pixel and the assigned depth value is decoded or coded for the at least one sub-image.

[0045] This particular embodiment of the invention makes it possible to signal whether the depth data in the case of coding according to the second method are coded by residuals or not. Indeed, depending on the type of information carried by the sub-image, a residual-based coding mode may or may not be advantageous. Residual-based coding may be advantageous when the sub-image carries information related to the occlusion zone, but may be less attractive for sub-images equipped with refinement information.

[0046] According to another particular embodiment of the invention, the decoding method comprises the steps of: - if the indicator indicates that the sub-image is coded according to a second method, from the reconstructed depth data and from the items of information representative of the decoded assigned depth values, - if the indicator shows that the sub-image is coded according to the first method, then from the decoded binary map, - determining a useful zone; - constructing a depth map of a sub-image in which pixels located outside the useful zone are assigned the value of infinity; - transmitting said depth map to a synthesis module.

[0047] According to this particular embodiment of the invention, the infinite value assigned to the depth value allows for a simple indication to the compositing algorithm of the pixels that belong to the useful and non-useful zones, avoiding the need to send additional data to the compositing module, such as a binary map indicating the useful and non-useful zones.

[0048] According to another particular embodiment of the invention, if the indicator indicates that the sub-image is coded according to the second method, the decoding method comprises: - determining a useful zone from the reconstructed depth data and from the items of information representative of the decoded assigned depth values; - constructing, for at least one pixel of the sub-image, a binary map indicating whether the pixel belongs to a useful zone or not; - transmitting said binary map to a synthesis module.

[0049] According to another particular embodiment of the invention, at least two sub-images intended to be used for generating said at least one image of said at least one view are coded in a data stream, and a minimum depth value is determined from each useful zone of said at least two sub-images.

[0050] According to another particular embodiment of the invention, the data stream comprises coded data of at least one view of a video, said at least one view corresponding to a view determined in the coding from at least one source view.

[0051] A source view is defined here as the original view of the scene represented by the multiview video: in other words, it is the sequence of images resulting from the capture of the scene by a camera.

[0052] According to this particular embodiment of the invention, at least one base view transmitted to the decoder is a view determined from one or more source views, this determination can be performed at coding time or before coding.

[0053] For example, it may be a view synthesized from multiple source views, or a view computed / estimated from the perspective of multiple sources.

[0054] For example, the determined view may correspond to a sequence of images, each image being larger than a camera image, comprising all images of the same instant in time from all cameras in a 360 degree capture camera array.

[0055] According to another particular embodiment of the invention, the decoding method further comprises a step of receiving texture and depth data of said sub-image, said sub-image resulting from a synthesized intermediate view of a viewer's position and viewing direction within the viewing volume.

[0056] According to this particular embodiment of the invention, the data of the sub-images is provided to the decoder as a function of the position in the viewing volume and the viewing direction. Such a particular embodiment of the invention allows providing the data of the sub-images on demand as the viewer navigates through the multi-view video. Thus, the data of the corresponding refinement or occlusion zones is better adapted to the viewer's position and viewing direction.

[0057] The invention also relates to a device for decoding at least one data stream representing a multiview video, said at least one data stream comprising at least coded data representing at least one sub-image, said coded data comprising texture data and depth data associated to said texture data, said sub-image comprising a zone, called useful zone, intended to be used for generating at least one image of at least one view, said decoding device comprising: - decoding from said at least one data stream an indicator indicating whether a sub-image is coded according to a first method or according to a second method; - if the indicator indicates that the subimage is coded according to the first method, decoding into the subimage a binary map indicating, for at least one pixel of the subimage, whether said pixel belongs to a useful zone or not; - decoding texture data and depth data of the sub-image, where if the indicator indicates that the sub-image is coded according to a second method, depth data of said sub-image for pixels located outside the useful zone comprises decoded depth values.

[0058] According to a particular embodiment of the invention, such a decoding device is comprised in a terminal or in a server.

[0059] The present invention also provides - encoding at least one sub-image comprising texture data and depth data associated with said texture data, said sub-image comprising a useful zone intended to be used for generating at least one image of at least one view; - determining a coding method for coding the sub-image between a first method and a second method; - encoding an indicator that indicates whether the sub-image is coded according to the first method or according to the second method; - if the indicator indicates that the subimage is coded according to the first method, coding in the subimage a binary map indicating, for at least one pixel of the subimage, whether said pixel belongs to a useful zone or not; - if the indicator shows that the sub-picture is coded according to the second method, - determining an allocation depth value; - assigning an assigned depth value to the depth data of said sub-image for pixels of said sub-image located outside the useful zone; and - coding texture data and depth data of a sub-image.

[0060] According to a particular embodiment of the invention, such a coding device is comprised in a terminal or in a server.

[0061] The invention also relates to a data stream representing a multiview video, comprising at least coded data representing at least one sub-image, the coded data comprising texture data and depth data associated to said texture data, said sub-image comprising zones, called useful zones, intended to be used to generate at least one image of at least one view, the data stream comprising: - an indicator of whether the sub-image is coded according to the first method or according to the second method; - if the indicator indicates that the sub-image is coded according to a first method, coded data representing a binary map for at least one pixel of the sub-image indicating whether said pixel belongs to a useful zone or not; - if the indicator indicates that the sub-image is coded according to the second method, depth data of said sub-image for pixels located outside the useful zone comprising depth values.

[0062] According to a particular embodiment of the invention, the data stream further comprises an item of information representative of an assigned depth value if the indicator shows that the sub-image has been coded according to the second method.

[0063] According to another particular embodiment of the invention, the data stream comprises an indicator of the at least one sub-image indicating whether depth data is coded by obtaining a residual determined by the difference between a depth value of a pixel and the assigned depth value.

[0064] According to another particular embodiment of the invention, the data stream further comprises coded data of at least one view of the video, said at least one view corresponding to a view synthesized in the coding from at least one source view.

[0065] The decoding method, respectively the coding method according to the invention can be implemented in various ways, in particular in wired or software form. According to a particular embodiment of the invention, the decoding method, respectively the coding method is implemented by a computer program. The invention also relates to a computer program comprising instructions for implementing the decoding method or the coding method according to any one of the particular embodiments described above, when said program is executed by a processor. Such a program can use any programming language. It can be downloaded from a communication network and / or recorded on a computer-readable medium.

[0066] The program may use any programming language and may be in the form of source code, object code, or any intermediate code between source code and object code, such as a partially compiled form or any other desired form.

[0067] The invention also relates to a computer-readable storage medium or data medium comprising instructions of a computer program as described above. Said recording medium can be any entity or device capable of storing a program. For example, the medium can comprise a storage means such as a ROM, for example a CD-ROM or a microelectronic circuit ROM, a USB flash drive, or a magnetic recording means, for example a hard drive. On the other hand, the recording medium can correspond to a transmissible medium, such as an electric or optical signal, which can be conveyed via an electric or optical cable, by radio or other means. The program according to the invention can in particular be downloaded onto an Internet-type network.

[0068] Alternatively, the recording medium may correspond to an integrated circuit in which the program is embedded, the circuit being adapted to perform, or to be used in the performance of, the method in question.

[0069] Other characteristics and advantages of the invention will become more clearly apparent on reading the following description of particular embodiments, given simply as illustrative and non-limiting examples, and the accompanying drawings, in which: [Brief description of the drawings]

[0070] [Figure 1] FIG. 1 diagrammatically illustrates an example of a multi-view scene capture system. [Diagram 2] FIG. 13 illustrates example inputs / outputs of a view synthesis module. [Diagram 3] FIG. 1 diagrammatically illustrates an example of a multiview encoder based on patch coding. [Figure 4] FIG. 3 illustrates steps of an encoding method according to a particular embodiment of the invention. [Diagram 5] FIG. 1 shows an example of an image grouping sub-images to be coded. [Figure 6]FIG. 6 shows an example of an image with sub-images and a corresponding binary map showing useful and non-useful zones of the sub-image of the image shown in FIG. 5. [Figure 7] FIG. 13 illustrates an example of filling in non-useful zones of a patch's texture, according to certain embodiments of the present invention. [Figure 8] FIG. 13 illustrates an example of filling in non-useful zones of a patch depth according to certain embodiments of the present invention. [Figure 9] FIG. 3 illustrates steps of a decoding method according to a particular embodiment of the invention. [Figure 10] FIG. 4 illustrates steps of a decoding method according to another particular embodiment of the invention. [Figure 11] FIG. 4 illustrates steps of a decoding method according to another particular embodiment of the invention. [Figure 12] FIG. 2 illustrates an example of an architecture of a coding device according to a particular embodiment of the invention. [Figure 13] FIG. 2 illustrates an example of an architecture of a decoding device according to a particular embodiment of the invention. [Figure 14] FIG. 2 illustrates an example of a data stream according to certain embodiments of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0071] FIG. 4 illustrates steps of a method for coding at least one data stream representing a multiview video according to a particular embodiment of the invention.

[0072] According to the invention, multi-view video is coded according to a coding scheme as shown in relation to Figure 3, where one or more base views are coded into the data stream, and where sub-images or patches with texture and depth data are also coded into the data stream, allowing a decoder to synthesize other views of the scene not coded into the data stream. The steps of such a coding scheme relevant to particular embodiments of the invention are described below.

[0073] Here, for example, it is assumed that a scene is captured according to four viewpoints V0 to V3, and a depth map d0 to d3 associated with each of the images of the views has been previously estimated. A view is here defined as a sequence of images representing a scene captured from a viewpoint. Each view is thus a video, i.e. a sequence of images, and a depth map is determined for each image of a view.

[0074] In step E40, the views V0 to V3 and the associated depth maps d0 to d3 are coded and decoded, for example by an HEVC encoder, or any other suitable 2D or multiview encoder. These views are considered as base views and are transmitted to the decoder. They can be used to synthesize intermediate views at the decoder side. As a variant, not all views V0 to V3 are coded, but for example only one view, or two or three views can be coded and transmitted to the decoder.

[0075] According to a particular embodiment of the invention, the base view coded in the data stream in step E40 corresponds to a view synthesized from one or more views V0 to V3.

[0076] In step E41, an intermediate view Vi is synthesized using the coded views transmitted to the decoder. Typically, such an intermediate view Vi comprises texture data and depth data associated with the texture data.

[0077] In step E42, from this intermediate view Vi and the coded / decoded base view, one or more zones are extracted from the intermediate view Vi. The method is presented here for the case of a synthesized intermediate view Vi. Of course, the method applies in the same way if several intermediate views are synthesized and one or more zones are extracted from these intermediate views.

[0078] It is not the object of the present invention to determine the zones to be extracted, therefore methods for extracting these zones are not developed further.

[0079] The extracted zones may correspond to occlusion zones, i.e. zones that are not displayed in the base view and are displayed in the intermediate view, or zones that are already present in the base view but provide insufficient synthesis quality in the synthesized view, for example due to distortions that appear during synthesis. In the latter case, the synthesized intermediate view corresponds to a view captured by the camera. The intermediate view from which the zones are extracted may be a view captured by the camera but not coded in the data stream. It can therefore be synthesized at the decoder side and refined by sub-images coded in the data stream.

[0080] The extracted zones are represented in the form of patches of arbitrary shape, hereafter referred to as sub-images, which comprise texture data, depth data, position within each view, and size.

[0081] It is understood that for an intermediate view, one or more sub-images can be extracted for the same image of the intermediate view. Similarly, sub-images representing the same zone extracted from the intermediate view can be extracted for different images of the intermediate view. In this way, the texture and depth data of the sub-images change over time to form a sequence of sub-images.

[0082] A sub-image P is extracted from an intermediate view Vi. 0 ~P 5 An example of is shown in Fig. 5. In the example of Fig. 5, the sub-images are grouped into a rectangular image 50 to facilitate their coding. It will be understood that, depending on the variants of the embodiment of such an image, the image 50 can comprise sub-images extracted from different images of the intermediate view, or from a single image, or from sub-images extracted from the intermediate view Vi, and also from other intermediate views.

[0083] Similarly, sub-images can be coded without grouping them into rectangular images, and each sub-image is then treated as an image or sequence of images as the sub-images change over time.

[0084] In FIG. 5, if we represent the zones extracted from the intermediate view in a form suitable for coding (rectangles or at least quadrilaterals), the sub-images appear to comprise useful data (51, the heavily hatched zone in FIG. 5) that correspond to the extracted zones and are likely to be used for synthesis of the intermediate view or other views, and non-useful data (52, the finely hatched zone in FIG. 5) with zero texture and depth data.

[0085] The useful data of a sub-image may be comprised in one zone of the sub-image or in several disjoint zones of the sub-image, so that the sub-image comprises several zones that are useful for compositing.

[0086] The coding method according to the invention aims at reducing the coding costs of non-useful zones.

[0087] Below, the invention is described where the sub-images are considered to be grouped into larger images 50. However, the general principles of the invention apply in a similar manner when the sub-images are not grouped into larger images but are processed individually.

[0088] In step E42 the sub-pictures are coded. For this purpose, according to the invention, two methods for coding the sub-pictures can be used. A competition between the two coding methods is implemented in order to select, for a given sub-picture, the coding method that offers the best rate / distortion compromise. An indicator is coded into the stream to indicate which coding method is used for a sub-picture.

[0089] Two encoding methods are described below.

[0090] According to a first coding method, the sub-image is coded using a binary map which indicates, for a given pixel of the sub-image, whether the pixel belongs to a useful zone (e.g., the pixel takes the value 1 in the binary map) or does not belong to a useful zone (e.g., the pixel takes the value 0 in the binary map).

[0091] FIG. 6 shows a sub-image and a useful (51, texture image 50) of the image 50 shown in FIG. text In the hatched zone) zone and not useful (52, texture image 50 text The white zones in the corresponding binary map (50 bin ) with images (50 text According to this first coding method, only the texture and depth data of the pixels located in the useful zone are coded and transmitted to the decoder, and the associated binary map is coded for all pixels of the sub-image.

[0092] It should be noted here that predefined values, for example 0, specifying that the pixel in question is not useful cannot be used as references in the texture data. In fact, after encoding and decoding the texture, the pixel value is not guaranteed and may not be zero. Moreover, other null values ​​may appear in pixels that belong to the useful zone.

[0093] According to a second coding method, the sub-image is coded by filling in the non-useful zones of both the texture and depth data, then the sub-image is coded in the standard way by coding all the pixels of the sub-image.

[0094] For this purpose, in step E420, the coding method to be used for coding the sub-image is determined between the first and second methods mentioned above. For example, a coding competition between the two methods is implemented and the method offering the best rate / distortion compromise is selected. In known methods, such a competition is performed by simulating the coding according to the two methods respectively and calculating the rates used by each method and the associated distortions calculated between the original and reconstructed values ​​of the texture and / or depth data.

[0095] In step E421 an indicator is coded into the data stream indicating whether the sub-image has been coded according to the first method or according to the second method. If the indicator indicates that the sub-image has been coded according to the first method, the method proceeds to step E422. If the indicator indicates that the sub-image has been coded according to the second method, the method proceeds to step E424.

[0096] In step E422, the binary map of the sub-image is coded, for example using entropy coding.

[0097] In step E423, texture and depth data are coded for the sub-image, considering only the pixels located in the useful zone of the sub-image. These pixels are identified using a binary map. The texture and depth data are coded in a standard manner, for example by an HEVC encoder or any other 2D or 3D encoder.

[0098] If, in step E424, the indicator indicates that the subimage is coded according to the second method, the assigned depth value d * is determined.

[0099] For example, the allocation depth value d * is the minimum depth value d determined over at least the useful zone of the sub-image min and is decreased by a predetermined value delta. min corresponds to the minimum depth value of the pixels of the sub-image that belong to the useful zone, i.e. the depth of the pixels of the sub-image that belong to the useful zone, and the furthest value is d min =min(d i ), where d i corresponds to the depth value of a pixel in the useful zone and min corresponds to a function returning the minimum value.

[0100] This minimum value d min is decreased by a value delta to account for the quantization error introduced in encoding the depth value. For example, the assigned value d * d * =d min Obtained by -delta.

[0101] Various values ​​of delta are possible, for example delta equal to 10.

[0102] According to certain embodiments of the invention, the minimum depth value is determined from the useful zones of several sub-images. For example, if the sub-images are grouped into a larger image, the useful zones of all sub-images are scanned to determine the minimum depth value among all sub-images of the larger image. Such certain embodiments of the invention allow for more consistent assigned depth values ​​for all sub-images of the larger image.

[0103] In step E425, the assigned depth value d *are assigned as depth values ​​to pixels located outside the useful zone. Such a step is illustrated in FIG. 8, which represents the depth data of a sub-image. Pixels in the useful zone (51) are provided with depth values ​​estimated for these pixels when extracting the useful zone from the intermediate view. Pixels in the non-useful zone (52) are provided with assigned depth values ​​d * Equipped with.

[0104] In step E426, texture values ​​are also assigned to pixels lying outside the useful zone.

[0105] For example, non-useful zones are filled by mirroring. As shown in Fig. 7, the texture values ​​of the pixels outside the useful zone are calculated to best reproduce the data present in the useful zone. In Fig. 7, a sub-image is shown. The data of the useful zone (51) is propagated to the non-useful zone (52) symmetrically with respect to the boundary (53) between the useful and non-useful zones.

[0106] According to another example, the texture value of a pixel located outside the useful zone (52) corresponds to the average of the texture values ​​of the pixels of the useful zone (51). In this example, the pixels of the non-useful zone are filled with the same value.

[0107] In step E427, the texture and depth data of the sub-image are coded taking into account all the pixels of the sub-image. In other words, according to this second coding method, the pixels of the useful zone and the pixels of the non-useful zone are coded. The texture and depth data are coded in a standard manner, for example by an HEVC encoder or any other 2D or 3D encoder.

[0108] According to a particular embodiment of the invention, the depth data is represented by an assigned depth value d *According to this particular embodiment of the invention, the prediction residual is then res(x,y)=prof(x,y)-d * where prof(x,y) is the depth value of the sub-image pixel (x,y) and res(x,y) is the residual value of the sub-image pixel (x,y).

[0109] Therefore, when coding the depth data of a sub-image, it is the residual as described above that is coded.

[0110] According to this particular embodiment of the invention, in order for the decoder to reconstruct the depth data of the sub-image from the residual, the decoder assigns a depth value d * You need to be aware of this.

[0111] Thus, according to this particular embodiment of the invention, in step E428, an assigned depth value d * is encoded into the data stream.

[0112] According to a variant of this particular embodiment of the invention, a differential coding of the depth data is placed in competition with a conventional coding of the depth data. Thus, according to this variant, in step E420, when testing the sub-image coding according to the second method, it is also determined whether the depth data according to this second coding method is coded by residuals. In order to select the coding mode that offers the best rate / distortion compromise, a residual-based coding of the depth data and a conventional coding of the depth data are simulated.

[0113] According to a variant of this embodiment of the invention, an item of information indicating which coding mode of the depth data has been selected needs to be coded into the data stream. Thus, in step E429, a depth data coding indicator is coded into the data stream of the sub-image. Such an indicator indicates whether the depth data has been coded by the residual or not.

[0114] 9 illustrates steps of a method for decoding a data stream representing a multi-view video according to a particular embodiment of the invention, for example generated by the coding method described in relation to FIG.

[0115] According to the invention, the data stream comprises at least coded data related to the coding of one or more base views of the multiview video, and coded data representing sub-images or patches comprising texture and depth data. When decoded, these sub-images for example enable the decoder to synthesize other views of the scene not coded in the data stream. The steps of such a decoding scheme relevant to a particular embodiment of the invention are described below.

[0116] In step E90, the base view and the associated depth map are decoded from the data stream.

[0117] According to a particular embodiment of the invention, a base view decoded from a data stream corresponds to a view synthesized at the encoder from one or more captured views.

[0118] The sub-image coded in the data stream is then decoded. A sub-image is defined here as a sub-image that represents a useful zone extracted at the encoder from an intermediate view. This sub-image can be grouped with other sub-images in a larger image.

[0119] In step E91, an indicator is decoded which indicates whether the sub-image has been coded according to the first method or according to the second method described above.

[0120] If the indicator indicates that the sub-image has been coded according to the first method, the method proceeds to step E92. If the indicator indicates that the sub-image has been coded according to the second method, the method proceeds to step E94.

[0121] In step E92, a binary map is decoded for the subimage: such a binary map indicates, for a given pixel of the subimage, whether or not this pixel belongs to a useful zone.

[0122] In step E93, the texture and depth data of the sub-image are decoded and reconstructed taking into account only the pixels located in the useful zone of the sub-image: these pixels are identified using a binary map.

[0123] At the end of the reconstruction, the pixels of the sub-image lying outside the useful zone are provided with a texture and depth value of zero.

[0124] If the indicator shows that the sub-image has been coded according to the second method, then in step E94 the texture and depth data of the sub-image are decoded as follows.

[0125] In step E940, the texture and depth data of the sub-image are decoded and reconstructed in the standard way taking into account all pixels of the sub-image. At the end of the reconstruction, all pixels of the sub-image are provided with texture and depth values. The pixels of the sub-image located outside the useful zone are provided with texture and depth values ​​determined during coding.

[0126] According to a particular embodiment of the invention, if the indicator indicates that the subimage is coded according to the second method, the decoder does not distinguish between useful and non-useful zones of the subimage.

[0127] However, the decoded depth values ​​of pixels of non-useful zones of the sub-images cause the synthesis algorithm to not use these pixels when synthesizing intermediate views. Indeed, according to a particular embodiment of the invention, such pixels are assigned a depth value determined from the minimum depth value of the zone of the sub-image, reduced by a value that allows to take into account quantization errors. In this way, these pixels are considered by the synthesis algorithm as the furthest points.

[0128] According to another particular embodiment of the invention, in step E941, the assigned depth value d assigned at the time of coding to the pixels located outside the useful zone of the sub-image is * An item of information representing is decoded from the data stream.

[0129] According to this particular embodiment of the invention, a binary map can therefore be constructed for the sub-images to identify pixels in useful and non-useful zones, for example if such a map is required for a compositing algorithm.

[0130] In a variation of this particular embodiment, the depth data of the sub-images is assigned a depth value d * were differentially coded.

[0131] Thus, according to a variant of this embodiment, in step E943, the depth data of the sub-image is decoded by the assigned depth value d decoded to the depth residual value of the sub-image decoded in step E940. * is reconstructed by adding

[0132] According to another variant, in step E942, a depth data coding indicator of the sub-image is decoded from the data stream, such coding indicator indicating whether the depth data is differentially coded or not.

[0133] According to a particular embodiment of the invention, the decoder operates in a client-server mode, with the server providing data for the multi-view video that the viewer is navigating.

[0134] According to this particular embodiment of the invention, the decoding method comprises a preliminary step E901 for receiving texture and depth data of a sub-image, for example following a request issued by the decoder for a position and viewing direction corresponding to or close to the viewer's position and viewing direction in the viewing volume.

[0135] Figure 10 illustrates the steps of a decoding method according to another particular embodiment of the invention. According to this particular embodiment of the invention, the depth values ​​of the pixels of the reconstructed sub-images that do not belong to the useful zones are assigned a value of infinity, so that the synthesis algorithm can distinguish between useful and non-useful zones.

[0136] For this purpose, in step E100, a useful zone of the reconstructed sub-image is determined. If the indicator decoded in step E91 indicates that the sub-image has been coded according to the second method, the useful zone is determined based on the reconstructed depth data and the decoded assigned depth value d * If the indicator decoded in step E91 indicates that the subimage has been coded according to the first method, the useful zone is determined from the decoded binary map in step E92.

[0137] In step E101, a depth map of the subimage is constructed, in which the pixels of the useful zone take their decoded depth value and the pixels located outside the useful zone are assigned the value of infinity.

[0138] In step E102, the depth map is transmitted to a synthesis module.

[0139] FIG. 11 illustrates the steps of a decoding method according to another particular embodiment of the invention.

[0140] If the indicator decoded in step E91 indicates that the subimage is coded according to the second method, then in step E110 the pixels of the useful zone are coded according to the reconstructed depth data and the decoded assigned depth value d * is determined from.

[0141] In step E111, a binary map is constructed for the subimage. Such a binary map indicates, for a given pixel of the subimage, whether the pixel belongs to a useful zone or not. Such a binary map is similar to the binary map that would have been decoded if the subimage had been coded according to the first method.

[0142] In step E112, the binary map is transmitted to a synthesis module.

[0143] FIG. 12 shows a simplified structure of a coding device COD adapted to implement the coding method according to any one of the particular embodiments of the invention.

[0144] According to a particular embodiment of the invention, the steps of the coding method are implemented by computer program instructions. For this purpose, the coding device COD has a standard architecture of a computer and in particular comprises a processing unit UT comprising a memory MEM, for example a processor PROC and driven by a computer program PG stored in the memory MEM. The computer program PG comprises instructions for implementing the steps of the coding method as described above, when the program is executed by the processor PROC.

[0145] At initialization, the code instructions of the computer program PG are loaded, for example, into a RAM memory (not shown) before being executed by the processor PROC. In particular, the processor PROC of the processing unit UT implements the steps of the coding method described above according to the instructions of the computer program PG.

[0146] FIG. 13 shows a simplified structure of a decoding device DEC adapted to implement the decoding method according to any one of the particular embodiments of the invention.

[0147] According to a particular embodiment of the invention, the decoding device DEC has a standard architecture of a computer and in particular comprises a memory MEM0, a processor PROC0, and a processing unit UT0 driven by a computer program PG0 stored in the memory MEM0. The computer program PG0 comprises instructions for implementing the steps of the decoding method as described above, when the program is executed by the processor PROC0.

[0148] At initialization, the code instructions of the computer program PG0 are loaded, for example, into a RAM memory (not shown) before being executed by the processor PROC0. In particular, the processor PROC0 of the processing unit UT0 implements the steps of the above-mentioned decoding method according to the instructions of the computer program PG0.

[0149] 14 shows an example of a data stream STR according to a particular embodiment of the invention, for example generated by the encoding method described in relation to FIG.

[0150] Referring to FIG. 14, the data stream STR represents a multi-view video, specifically, a coded data stream V representing one or more base views. band coded data representing at least one sub-image comprising texture data (text0, text1) and depth data (dpth0, dpth1) associated with said texture data.

[0151] According to a particular embodiment of the invention, the base views coded in the data stream correspond to views synthesized in the coding from at least one source view of the scene.

[0152] The data stream STR shown in Fig. 14 is an illustrative example. Here, it is presented as comprising two sub-images (Im0, Im1). The sub-images comprised in the data stream comprise a useful zone intended to be used for generating at least one image of at least one view of a scene captured by a multiview video.

[0153] According to various specific embodiments of the invention, the data stream STR comprises the following data for a sub-image: - an indicator (Id_meth0, Id_meth1) of whether the subimage is coded according to the first or second method described above, - if the indicator shows that the subimage is coded according to the first method (Id_meth1 in the example described here), coded data representing a binary map (valid1) indicating, for a given pixel of the subimage, whether said pixel belongs to a useful zone or not.

[0154] According to a particular embodiment of the invention, if the indicator indicates that the sub-image is coded according to the second method (Id_meth0 in the example described here), the data stream STR is coded according to the assigned depth value d * It has information items representing:

[0155] According to another particular embodiment of the invention, if the indicator indicates that the sub-image is coded according to the second method, the data stream STR is further coded according to the depth values ​​of the pixels and the assigned depth values ​​d * and a sub-image coding indicator (ld_res) that indicates whether the depth data is coded by taking a residual determined by the difference between

[0156] The above examples are merely illustrative and non-limiting. The coded data above may be provided in the same data stream, or in multiple data streams or sub-streams of the same data stream. Also, some of the above data, and other data not described, may be provided in metadata associated with one or more data streams.

[0157] For example, texture and depth data can be coded into separate streams.

[0158] According to another example, the coded data of the base view may be comprised in a video data stream and the coded data relating to the sub-image may be comprised in metadata associated with the video data stream. [Explanation of symbols]

[0159] 50 images 51 Useful Data, Useful Zones 52 Unusable data, unusable zones 53 Boundary

Claims

1. 1. A method for decoding at least one data stream representing a multiview video, said at least one data stream comprising at least coded data representing at least one sub-image, said coded data comprising texture data and depth data associated with said texture data, said sub-image comprising at least one zone, called useful zone, and one zone, called non-useful zone, said useful zone corresponding to a zone extracted from a view of said multiview video not coded in said data stream, said useful zone intended to be used for generating at least one image of at least one view, said decoding method comprising: - decoding from said at least one data stream an indicator of whether said sub-image is coded according to a first method or according to a second method; - if said indicator indicates that said subimage is coded according to said first method, decoding in said subimage a binary map indicating, for at least one pixel of said subimage, whether said pixel belongs to said useful zone; - decoding the texture data and the depth data of the sub-image, where if the indicator indicates that the sub-image is coded according to the second method, the depth data of the sub-image for the pixels located outside the useful zone comprises an assigned depth value; A method comprising:

2. 1. A method for coding a data stream representing a multiview video, comprising the step of coding at least one sub-image comprising texture data and depth data associated with said texture data, said sub-image comprising a zone, referred to as the useful zone, and a zone, referred to as the non-useful zone, said useful zone corresponding to a zone extracted from a view of said multiview video not coded in said data stream, said useful zone intended to be used for generating at least one image of at least one view, said step of coding at least one sub-image comprising: - determining a coding method for coding said sub-image between a first method and a second method; - encoding an indicator of whether said sub-image has been coded according to said first method or according to said second method; - if the indicator indicates that the sub-image is coded according to the first method, coding in the sub-image a binary map indicating, for at least one pixel of the sub-image, whether the pixel belongs to the useful zone or not; - if the indicator indicates that the subimage is coded according to the second method, determining an assigned depth value and assigning the assigned depth value to the depth data of the subimage for the pixels of the subimage that are located outside the useful zone; - coding the texture data and the depth data of the sub-image.

3. 3. Coding method according to claim 2, wherein the assigned depth value is determined over at least the useful zone of the sub-image from a minimum depth value reduced by a predetermined value.

4. A decoding method as claimed in claim 1 or an encoding method as claimed in claim 2 or 3, further comprising the step of decoding or encoding an item of information representative of the assigned depth value if the indicator indicates that the sub-image has been coded according to the second method, wherein the assigned depth value of 0 corresponds to the decoded or coded depth value for the pixels located outside the useful zone.

5. 5. The method of claim 4, wherein the depth data of the sub-image coded into the data stream comprises, for at least one pixel of the sub-image, a residual determined by a difference between the depth value of the pixel and the assigned depth value.

6. 6. The method of claim 5, comprising decoding or encoding, in the data stream, an indicator indicating for the at least one sub-image whether the depth data is coded by a residual determined by a difference between the depth value of the pixel and the assigned depth value.

7. - determining the useful zone from the reconstructed depth data and the item of information representing the decoded assigned depth values ​​if the indicator indicates that the subimage is coded according to the second method, or from the decoded binary map if the indicator indicates that the subimage is coded according to the first method; - constructing a depth map of said sub-image in which said pixels located outside said useful zone are assigned a value of infinity; - transmitting said depth map to a synthesis module; The method of decoding according to any one of claims 4 to 6, further comprising:

8. If the indicator indicates that the sub-image is coded according to the second method, - determining the useful zone from the reconstructed depth data and from the items of information representative of the decoded assigned depth values; - constructing, for at least one pixel of said sub-image, a binary map indicating whether said pixel belongs to said useful zone or not; - sending said binary map to a synthesis module; The method of decoding according to any one of claims 4 to 6, further comprising:

9. 4. The method of claim 3, wherein at least two useful zones each belong to at least two sub-images, the at least two useful zones intended to be used for generating the at least one image of the at least one view are coded into the data stream, and the minimum depth value is determined from each useful zone of the at least two useful zones.

10. 10. The method of claim 1, wherein the at least one data stream comprises coded data of at least one view of the video, the at least one view corresponding to a view determined in the coding from at least one source view.

11. 11. A method for decoding according to claim 1 or any one of claims 4 to 10, further comprising a step of receiving the texture data and the depth data of the sub-image, the sub-image resulting from a synthesized intermediate view of a viewer's position and viewing direction within a viewing volume.

12. A device for decoding at least one data stream representing a multiview video, said at least one data stream comprising at least coded data representing at least one sub-image, said coded data comprising texture data and depth data associated with said texture data, said sub-image comprising a zone, referred to as the useful zone, and a zone, referred to as the non-useful zone, said useful zone corresponding to a zone extracted from a view of said multiview video not coded in said data stream, said useful zone intended to be used for generating at least one image of at least one view, said decoding device comprising: - decoding from said at least one data stream an indicator indicating whether said sub-image is coded according to a first method or according to a second method; - if the indicator indicates that the subimage is coded according to the first method, decoding into the subimage a binary map indicating, for at least one pixel of the subimage, whether the pixel belongs to the useful zone or not; - decoding the texture data and the depth data of the sub-image, where if the indicator indicates that the sub-image is coded according to the second method, the depth data of the sub-image for the pixels located outside the useful zone comprises an assigned depth value. A device configured to:

13. A device for coding a data stream representing a multiview video, the device being configured to code at least one sub-image comprising texture data and depth data associated with said texture data, the sub-image comprising a zone, referred to as the useful zone, and a zone, referred to as the non-useful zone, the useful zone corresponding to a zone extracted from a view of the multiview video not coded in the data stream, the useful zone intended to be used to generate at least one image of at least one view, - determining a coding method for coding said sub-image between a first method and a second method; - encoding an indicator of whether said sub-image is coded according to said first method or according to said second method; - if the indicator indicates that the sub-image is coded according to the first method, coding in the sub-image a binary map indicating, for at least one pixel of the sub-image, whether the pixel belongs to the useful zone or not; - if the indicator indicates that the subimage is coded according to the second method, determining an assigned depth value and assigning the assigned depth value to the depth data of the subimage for the pixels of the subimage that are located outside the useful zone; - encoding the texture data and the depth data of the sub-image; A device comprising:

14. A data stream representing a multiview video, comprising at least coded data representing at least one sub-image, said coded data comprising texture data and depth data associated with said texture data, said sub-image comprising a zone, referred to as the useful zone, and a zone, referred to as the non-useful zone, said useful zone corresponding to a zone extracted from a view of said multiview video not coded in said data stream, said useful zone intended to be used for generating at least one image of at least one view, said data stream comprising: - an indicator of whether said sub-image has been coded according to a first method or according to a second method; - if said indicator indicates that said sub-image is coded according to said first method, coded data representing, for at least one pixel of said sub-image, a binary map indicating whether said pixel belongs to said useful zone or not; - if the indicator indicates that the subimage is coded according to the second method, the depth data of the subimage for the pixels located outside the useful zone comprising a depth value; A data stream comprising:

15. A computer program comprising instructions for implementing a decoding method according to any one of claims 1 or 4 to 10 and / or instructions for implementing an encoding method according to any one of claims 2 to 6 or 9 or 10, said program being executed by a processor.

Images