METHOD AND DEVICES FOR CODING AND DECODING MULTI-VIEW SEQUENCES

Description

1. Field of the invention

[0001] The invention relates to so-called immersive videos, representative of a scene captured by one or more cameras. More particularly, the invention relates to the coding and decoding of such videos. 2. Prior Art

[0002] In an immersive video context, that is to say where the viewer has the sensation of being immersed in the scene, the scene is classically captured by a set of cameras, as illustrated in figure 1 . These cameras can be of the 2D type (cameras C1, C2, C3, C4 of the figure 1 ), or 360 type, that is to say capturing the entire scene in 360 degrees around the camera (C5 camera of the figure 1 ).

[0003] All of these captured views are traditionally encoded and then decoded by a viewer's terminal. However, in order to provide sufficient quality of experience, and therefore visual quality and good immersion in the scene displayed to the viewer, displaying only the captured views is insufficient.

[0004] In order to improve the feeling of immersion in the scene, in general, one or more views, called intermediate views, are calculated from the decoded views.

[0005] The calculation of these intermediate views can be carried out by a so-called "view synthesis" algorithm. An example of a synthesis algorithm is illustrated in figure 2 showing mainly the inputs / outputs of a view synthesis algorithm. According to this example, from n views (here 1-4), the synthesis algorithm is able to synthesize a viewpoint "v" located in a given volume, called the viewing volume, depending on the positioning of the cameras. The n views and the synthesized view are composed of both texture data (T 1 -T 4 , T v ) and depth data (D 1 -D 4 , D v ) which make it possible to represent the distance of the elements of the scene in relation to the cameras. The depth data is classically represented in the form of a depth map, i.e. an image indicating at each point of the image the depth associated with the object represented at this point.

[0006] These depth maps are either captured by specific cameras or calculated from texture views. Compressing the captured views to represent the scene is challenging, due to the enormous amount of data involved.

[0007] Moreover, since captured views are required for intermediate view synthesis, there is a trade-off between view coding efficiency and intermediate view synthesis quality.

[0008] MV-HEVC and 3D-HEVC encoders are two extensions of the 2D HEVC encoder (" High Efficiency Video Coding, Coding Tools and Specification", Matthias Wien, Signals and Communication Technology, 2015 ) suitable for encoding multi-view videos.

[0009] Such encoders take as input a set of views and optionally the associated depth maps. The MV-HEVC encoder does not apply any specific block-level coding tool. To encode an image from a current view, it simply uses, in addition to the previously encoded / decoded images of the current view, an image from another view as a reference image.

[0010] The 3D-HEVC encoder, on the other hand, adds specific tools for depth map encoding and uses more advanced inter-view redundancies and inter-component redundancies, where depth map encoding takes advantage of texture encoding. However, it appears that for 360° multi-view content, inter-view redundancy is not properly exploited by these two codecs. Their coding efficiency is therefore far from optimal. In addition, nothing is done to preserve areas that potentially cause occlusions, which affects the quality of synthesized view images.

[0011] There figure 3 illustrates another method for encoding multi-view videos and for synthesizing intermediate views on the decoder side.

[0012] According to this method, one or more basic views (T b , D b on the figure 3 ) are encoded with a 2D encoder, for example an HEVC type encoder, or with a multi-view encoder.

[0013] The other views (T s , D s ) undergo processing which allows certain areas to be extracted from each of these views.

[0014] In the case where not all captured views are transmitted to the decoder, or when the cameras are naturally distant, the occlusion zones increase. These zones correspond to objects that are hidden in the views transmitted to the decoder but which are necessary to reconstruct an image from any viewpoint in the viewing volume. The extracted zones can thus correspond to the occlusion zones that are hidden in the base views but which are visible in the views not transmitted to the decoder.

[0015] Undersampling of views can also lead to significant defects in view synthesis, particularly when an object is very distorted depending on the viewing position, i.e. the synthesized viewpoint.

[0016] According to the method illustrated in figure 3 , the extracted areas, also called patches hereinafter, are coded for example by a classic 2D video coder. On the decoder side, the synthesis algorithm can then produce one or more intermediate views from the basic views and the decoded patches.

[0017] Overall, patches allow information to be transmitted to the decoder relating to the same area which can be seen from one point of view of a base view or from another point of view, so as to be able to take into account a change in illumination between the two points of view, or for example poor projection quality, linked to the object itself, for example in the case of an inclined object (which extends over a certain depth interval).

[0018] In order to encode the patches representing the extracted areas, it is necessary to encode the texture of these patches and the associated depth. To simplify the coding and representation of these patches which can have different shapes, in general, the patches are stored in the form of rectangular images or at least in the form of quadrilaterals, also called sub-images.

[0019] With a representation in the form of sub-images, it appears that the sub-images include so-called useful data corresponding to the extracted areas and likely to be used for the synthesis of intermediate views, and non-useful data not initially belonging to the extracted patches.

[0020] For efficient patch coding, it is necessary that the coding of non-useful data has minimal impact on the coding of useful data.

[0021] In addition, on the decoder side, it is necessary to transmit information to the synthesis algorithm allowing it to identify which areas are useful or not, so that it does not use unnecessary areas during synthesis.

[0022] RENAUD DORE (TECHNICOLOR) ET AL: "[MPEG-I Visual] RVS based 3DoF+ Evidence", no. m44588 October 9, 2018 (2018-10-09), XP030193168 discloses the generation of texture and depth sub-images, comprising useful and non-useful areas ("packed views"), which are then encoded with HEVC.

[0023] There is therefore a need to improve the state of the art. 3. Statement of the invention

[0024] The invention is defined by the claims. In the following, when the terms "embodiment" and "particular embodiment" refer to a combination of features not claimed, said terms should be understood as referring to examples of the present invention.

[0025] The invention improves the state of the art. To this end, it relates to a method for decoding at least one data stream representative of a multi-view video. The data stream comprises at least coded data representative of at least one sub-image, the coded data comprising texture data and depth data associated with said texture data, and the sub-image comprising at least one zone, called useful, intended to be used to generate at least one image of at least one view.

[0026] Such a decoding method includes: decoding from said at least one data stream, an indicator indicating whether the sub-image is coded according to a first method or according to a second method, when the indicator indicates that the sub-image is coded according to the first method, decoding for the sub-image a binary map indicating for at least one pixel of the sub-image whether said pixel belongs to the useful area or not, decoding the texture data and the depth data of the sub-image, when the indicator indicates that the sub-image is coded according to the second method, the depth data of said sub-image for the pixels located outside the useful area comprising a decoded depth value.

[0027] According to the invention, it is thus possible to code and decode sub-images coding areas useful for synthesis while limiting the cost of coding these sub-images. In particular, depending on the value of an indicator coded in the stream, the useful area in a sub-image representing a patch can be determined by a synthesis algorithm during the reconstruction of a view, either from the depth information coded in the stream for the sub-image, or from a binary map coded and decoded from the stream.

[0028] According to the invention, unlike the techniques of the prior art, it is thus not necessary to systematically code for the sub-image a validity binary map making it possible to identify the useful area of ​​the sub-image. For example, at the encoder, a competition can be implemented to determine the best coding in terms of flow / distortion for the depth information of the patch. Such a competition compares the performance of a first method coding a binary map indicating which pixels belong to the useful area and transmitted to the synthesis algorithm during the synthesis of a view, with the performance of a second method according to which the depth value of the pixels located outside the useful area is assigned a particular value. When the sub-image is coded according to the first method, only the depth values ​​of the pixels belonging to the useful area are coded and decoded.Whereas when the sub-image is encoded using the second method, the depth value of all pixels in the sub-image is encoded and decoded, regardless of whether these pixels belong to the useful area or not. The pixels in the sub-image that do not belong to the useful area carry a particular decoded depth value that was determined at encoding.

[0029] Such a special value allows the synthesis algorithm to not use pixels that do not belong to the useful area when synthesizing a view.

[0030] The sub-image may include texture and depth data relating to an occlusion area not visible in the views encoded in the data stream. Thus, encoding such a sub-image improves the quality of the synthesis of a view not encoded in the data stream.

[0031] In other embodiments, the sub-image may include texture and depth data relating to an area visible from views encoded in the data stream, but resulting in significant defects in the synthesis of a view. The coding of such a sub-image makes it possible to improve the quality of the synthesis of this view by providing texture and depth refinements to the area.

[0032] The proposed invention allows for better adaptation to the content of the views and to the types of occlusion. In particular, the first method allows for preserving clear contours of the occlusion zones during synthesis, while the second method allows for saving coding costs by avoiding the transmission of the binary map. However, since the distinction between useful area and non-useful area in the sub-image is no longer made during coding according to the second method, the sharpness of the contours of the occlusion zones is not preserved.

[0033] The choice at the encoder between the first method and the second method can be made for example via a flow / distortion optimization in which the distortion is measured at the encoder from a partial synthesis of the area reconstructed by the useful area.

[0034] The invention also relates to a method for encoding a data stream representative of a multi-view video, comprising: the coding of at least one sub-image comprising texture data and depth data associated with said texture data, said sub-image comprising a zone, called useful, intended to be used to generate at least one image of at least one view, the coding of said at least one sub-image comprises: determining a coding method for coding the sub-image from among a first method and a second method, coding an indicator indicating whether the sub-image is coded according to the first method or according to the second method, when the indicator indicates that the sub-image is coded according to the first method, coding for the sub-image a binary map indicating for at least one pixel of the sub-image whether said pixel belongs to the useful area or not, when the indicator indicates that the sub-image is coded according to the second method: determining an assignment depth value, assigning the assignment depth value to the depth data of said sub-image for the pixels of the sub-image located outside the useful area, coding the texture data and the depth data of the sub-image.

[0035] According to a particular embodiment of the invention, the assignment depth value is determined from a minimum depth value determined on at least the useful area of ​​the sub-image, reduced by a predetermined value.

[0036] A minimum depth value is, for example, the depth value of the most distant (in terms of depth) point in the useful area in the sub-image. The minimum depth value is reduced by a predetermined value so as to obtain a particular depth value that is distinct from the depth values ​​of the points in the useful area. The reduction by a predetermined value thus makes it possible to take into account quantization errors when encoding the depth data.

[0037] Furthermore, using a particular depth value that remains close to the depth values ​​of the points in the useful area makes it possible to limit the impact of the depth coding cost for the sub-image. Indeed, the appearance of strong discontinuities in the depth information to be coded for the sub-image is limited.

[0038] Furthermore, such an assignment depth value can be used by the synthesis algorithm in an advantageous manner. Indeed, in the event of a conflict during the synthesis of an image, a synthesis algorithm will naturally favor objects located in the foreground, therefore with a greater depth value, than objects located in the background. The use, for the pixels of the sub-image located in the non-useful zone, of a depth value lower than the depth value of the pixels in the useful zone thus makes it possible to indicate to the synthesis algorithm that the pixels with the lower depth value do not belong to the useful zone, and their consideration by the synthesis algorithm will not be favored in the event of conflicts during the reconstruction of a synthesized image.

[0039] According to another particular embodiment of the invention, when the indicator indicates that the sub-image is coded according to the second method, the decoding method, respectively the coding method, comprises the decoding, respectively the coding, of information representative of the assignment depth value, the assignment depth value corresponding to the decoded depth value, respectively coded for the pixels located outside the useful zone.

[0040] According to this particular embodiment of the invention, a binary map indicating the pixels belonging to the useful area can thus be created for the sub-images coded according to the second method, since the decoder is aware of the so-called assignment depth value which has been assigned to the pixels located outside the useful area. It can thus identify, by means of the decoded pixel depth value, which pixels belong to the useful area and which pixels are outside the useful area. The binary map thus constructed can be transmitted to the synthesis algorithm. The information provided as input to the synthesis algorithm is thus identical, whether the sub-image has been coded according to the first method or according to the second method.

[0041] According to another particular embodiment of the invention, the depth data of the sub-image coded in the data stream comprise, for at least one pixel of the sub-image, a residue determined by a difference between the depth value of said pixel and said assignment depth value.

[0042] This particular embodiment of the invention makes it possible to reduce the cost of coding the depth data for the sub-image when the sub-image is coded according to the second method. Indeed, when the sub-image only comprises background information, the dynamics of the depth data is low and the coding of this data in the form of a residue is more advantageous.

[0043] According to another particular embodiment of the invention, an indicator is decoded or coded for said at least one sub-image indicating whether the depth data is coded by a residue determined by a difference between the depth value of said pixel and said assignment depth value.

[0044] This particular embodiment of the invention makes it possible to signal whether the depth data in the case of coding by the second method are coded by a residue or not. Indeed, depending on the type of information carried by the sub-image, the residual coding mode may or may not be advantageous. Residual coding may be advantageous when the sub-image carries information relating to occlusion zones, whereas it may be less interesting for sub-images comprising refinement information.

[0045] According to another particular embodiment of the invention, the decoding method further comprises: determining the useful area: from the reconstructed depth data and the information representative of the decoded assignment depth value when the indicator indicates that the sub-image is coded according to the second method, from the decoded binary map when the indicator indicates that the sub-image is coded according to the first method, constructing a depth map for the sub-image in which an infinite value is assigned to the pixels located outside the useful area, transmitting said depth map to a synthesis module.

[0046] According to this particular embodiment of the invention, the infinite value assigned to the depth values ​​makes it possible to simply indicate to the synthesis algorithm the pixels belonging to the useful zone and to the non-useful zone and avoids the transmission to the synthesis module of additional data, such as a binary map indicating the useful zone and the non-useful zone.

[0047] According to another particular embodiment of the invention, the decoding method further comprises, when the indicator indicates that the sub-image is coded according to the second method: determining the useful area from the reconstructed depth data and the information representative of the decoded assignment depth value, constructing a binary map indicating, for at least one pixel of the sub-image, whether or not the pixel belongs to the useful area, transmitting said binary map to a synthesis module.

[0048] According to another particular embodiment of the invention, at least two sub-images intended to be used to generate said at least one image of said at least one view are coded in the data stream, the minimum depth value is determined from each useful area of ​​said at least two sub-images.

[0049] According to another particular embodiment of the invention, the data stream comprises coded data of at least one view of the video, said at least one view corresponding to a view determined during coding from at least one source view.

[0050] A source view here refers to an original view of the scene represented by the multi-view video. In other words, it is a sequence of images resulting from a capture of the scene by a camera.

[0051] According to this particular embodiment of the invention, at least one basic view transmitted to the decoder is a view determined from one or more source views. This determination can be carried out during coding or prior to coding.

[0052] For example, it could be a view synthesized from multiple source views or a view calculated / estimated from multiple source viewpoints.

[0053] For example, the determined view may correspond to a sequence of images in which each image is larger than an image from a camera and includes all images of the same instant from all cameras in a 360° camera array.

[0054] According to another particular embodiment of the invention, the decoding method further comprises receiving the texture data and the depth data of said sub-image, said sub-image coming from an intermediate view synthesized for a position and a viewing direction of an observer in a viewing volume.

[0055] According to this particular embodiment of the invention, the sub-image data is provided to the decoder, upon request for a position and a viewing direction in the viewing volume. Such a particular embodiment of the invention makes it possible to provide the sub-image data on demand, during the navigation of an observer in the multi-view video. The data of the corresponding refinement or occlusion zones are thus more adapted to the position and viewing direction of the observer.

[0056] The invention also relates to a device for decoding at least one data stream representative of a multi-view video, said at least one data stream comprising at least coded data representative of at least one sub-image, the coded data comprising texture data and depth data associated with said texture data, said sub-image comprising a zone, called useful, intended to be used to generate at least one image of at least one view, the decoding device is configured to: decoding from said at least one data stream, an indicator indicating whether the sub-image is coded according to a first method or according to a second method, when the indicator indicates that the sub-image is coded according to the first method, decoding for the sub-image a binary map indicating for at least one pixel of the sub-image whether said pixel belongs to the useful area or not, decoding the texture data and the depth data of the sub-image, when the indicator indicates that the sub-image is coded according to the second method, the depth data of said sub-image for the pixels located outside the useful area comprising a decoded depth value.

[0057] According to a particular embodiment of the invention, such a decoding device is included in a terminal or a server.

[0058] The invention also relates to a device for encoding a data stream representative of a multi-view video, configured to: encoding at least one sub-image comprising texture data and depth data associated with said texture data, said sub-image comprising an area, called a useful area, intended to be used to generate at least one image of at least one view, comprising: determining a coding method for coding the sub-image from among a first method and a second method, encoding an indicator indicating whether the sub-image is coded according to the first method or according to the second method, when the indicator indicates that the sub-image is coded according to the first method, encoding for the sub-image a binary map indicating for at least one pixel of the sub-image whether said pixel belongs to the useful area or not, when the indicator indicates that the sub-image is coded according to the second method: determining an assignment depth value,assigning the assignment depth value to the depth data of said sub-image for the pixels of the sub-image located outside the useful area, encoding the texture data and the depth data of the sub-image.

[0059] According to a particular embodiment of the invention, such a coding device is included in a terminal or a server.

[0060] The invention also relates to a data stream representative of a multi-view video, comprising at least coded data representative of at least one sub-image, the coded data comprising texture data and depth data associated with said texture data, said sub-image comprising a zone, called useful, intended to be used to generate at least one image of at least one view, the data stream comprises: an indicator indicating whether the sub-image is coded according to a first method or according to a second method, when the indicator indicates that the sub-image is coded according to the first method, coded data representative of a binary map indicating for at least one pixel of the sub-image whether said pixel belongs to the useful area or not, when the indicator indicates that the sub-image is coded according to the second method, the depth data of said sub-image for the pixels located outside the useful area comprising a depth value.

[0061] According to a particular embodiment of the invention, the data stream further comprises, when the indicator indicates that the sub-image is coded according to the second method, information representative of the assignment depth value.

[0062] According to another particular embodiment of the invention, the data stream comprises an indicator for said at least one sub-image indicating whether the depth data is encoded by obtaining a residue determined by a difference between the depth value of a pixel and said assignment depth value.

[0063] According to another particular embodiment of the invention, the data stream further comprises coded data of at least one view of the video, said at least one view corresponding to a view synthesized during coding from at least one source view.

[0064] The decoding method, respectively the coding method, according to the invention can be implemented in various ways, in particular in wired form or in software form. According to a particular embodiment of the invention, the decoding method, respectively the coding method, is implemented by a computer program. The invention also relates to a computer program comprising instructions for implementing the decoding method or the coding method according to any one of the particular embodiments described above, when said program is executed by a processor. Such a program can use any programming language. It can be downloaded from a communications network and / or recorded on a computer-readable medium.

[0065] This program may use any programming language, and may be in the form of source code, object code, or code intermediate between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0066] The invention also relates to a recording medium or information medium readable by a computer, and comprising instructions of a computer program as mentioned above. The recording media mentioned above can be any entity or device capable of storing the program. For example, the medium can comprise a storage means, such as a ROM, for example a CD ROM or a microelectronic circuit ROM, a USB key, or even a magnetic recording means, for example a hard disk. Furthermore, the recording media can correspond to a transmissible medium such as an electrical or optical signal, which can be conveyed via an electrical or optical cable, by radio or by other means. The program according to the invention can in particular be downloaded from a network such as the Internet.

[0067] Alternatively, the recording media may correspond to an integrated circuit in which the program is incorporated, the circuit being adapted to carry out or to be used in carrying out the method in question. 4. List of figures

[0068] Other characteristics and advantages of the invention will appear more clearly on reading the following description of a particular embodiment, given as a simple illustrative and non-limiting example, and the appended drawings, among which: [ Fig. 1 ] There figure 1 schematically illustrates an example of a multi-view capture system of a scene. Fig. 2 ] There figure 2 illustrates an example of inputs / outputs of a view synthesis module. Fig. 3 ] There figure 3 schematically illustrates an example of a multi-view encoder based on patch coding. [ Fig. 4 ] There figure 4 illustrates steps of a coding method according to a particular embodiment of the invention. Fig. 5 ] There figure 5 illustrates an example of an image grouping sub-images to be coded. [ Fig. 6 ] There figure 6 illustrates an example of an image comprising sub-images and a corresponding binary map indicating the useful and non-useful areas of the sub-images for the illustrated image in figure 5 . [ Fig. 7 ] There figure 7 illustrates an example of filling a non-useful area of ​​the texture of a patch according to a particular embodiment of the invention. Fig. 8 ] There figure 8 illustrates an example of filling a non-useful area of ​​the depth of a patch according to a particular embodiment of the invention. Fig. 9 ] There figure 9 illustrates steps of a decoding method according to a particular embodiment of the invention. Fig. 10 ] There figure 10 illustrates steps of a decoding method according to another particular embodiment of the invention. Fig. 11 ] There figure 11 illustrates steps of a decoding method according to another particular embodiment of the invention. Fig. 12 ] There figure 12 illustrates an example of architecture of a coding device according to a particular embodiment of the invention. Fig. 13 ] There figure 13 illustrates an example of architecture of a decoding device according to a particular embodiment of the invention. Fig. 14 ] There figure 14 illustrates an example of a data flow according to particular embodiments of the invention. 5. Description of an embodiment of the invention

[0069] There figure 4 illustrates steps of a method for coding at least one data stream representative of a multi-view video according to a particular embodiment of the invention.

[0070] According to the invention, the multi-view video is encoded according to a coding scheme as presented in relation to the figure 3 in which one or more so-called basic views are coded in the data stream and in which sub-images or patches comprising texture and depth data are also coded in the data stream and allow the decoder to synthesize other views of the scene which have not been coded in the data stream. The steps of such a coding scheme relating to a particular embodiment of the invention are described below.

[0071] For example, we consider here that the scene was captured from 4 viewpoints V0-V3 and that the depth maps d0-d3 associated with each of the view images were previously estimated. By view, we mean here a sequence of images representing the scene captured from a viewpoint. Each view is therefore a video, i.e., a sequence of images, and for each image of the view, a depth map is determined.

[0072] During a step E40, the views V0-V3 and the associated depth maps d0-d3 are coded and decoded, for example by a HEVC encoder, or any other suitable 2D or multi-view encoder. These views are considered as basic views and will be transmitted to the decoder. They may be used to synthesize intermediate views on the decoder side. Alternatively, not all the views V0-V3 may be coded, for example a single view, or 2 or 3 may be coded and transmitted to the decoder.

[0073] According to a particular embodiment of the invention, during step E40, a basic view coded in the data stream corresponds to a view synthesized from one or more views V0-V3.

[0074] In a step E41, an intermediate view Vi is synthesized using the coded views and which will be transmitted to the decoder. Conventionally, such an intermediate view Vi comprises texture data and depth data associated with the texture data.

[0075] In a step E42, from this intermediate view Vi and the coded / decoded basic views, one or more zones are extracted from the intermediate view Vi. The method is presented here in the case of a synthesized intermediate view Vi. Of course, the method applies in an identical manner if several intermediate views are synthesized, one or more zones being extracted from these intermediate views.

[0076] The object of the invention does not lie in determining the areas to be extracted. The method of extracting these areas will therefore not be developed further.

[0077] The extracted areas may correspond to occlusion areas, i.e. areas not visible in the basic views and visible in the intermediate view, or to areas already present in the basic views but providing insufficient synthesis quality in the synthesized view, for example because of deformations appearing in the synthesis.

[0078] In the latter case, the synthesized intermediate view then corresponds to a view captured by a camera. The intermediate view from which the zones are extracted can be a view captured by a camera but not encoded in the data stream. It can therefore be synthesized on the decoder side and refined by the sub-images encoded in the data stream.

[0079] The extracted areas are represented as patches of arbitrary shape. Subsequently, these patches are called sub-images. A sub-image includes texture data, depth data, a position in each view, and a size.

[0080] It is understood that for the intermediate view, one or more sub-images can be extracted for the same image of the intermediate view. Similarly, a sub-image representing the same area extracted from the intermediate view can be extracted for different images of the intermediate view. Thus the texture and depth data of the sub-image then vary over time and form a sequence of sub-images.

[0081] Examples of sub-images P 0 -P 5 extracted from an image of the intermediate view Vi are illustrated in figure 5 . In the example of the figure 5 , the sub-images are grouped into a rectangular image 50 in order to facilitate their coding. It is understood that according to the variants of embodiment of such an image, the image 50 can comprise sub-images extracted from different images of the intermediate view, or from a single image, or even sub-images extracted from the intermediate view Vi and also from other intermediate views.

[0082] Similarly, sub-images can be encoded without being grouped into a rectangular image. Each sub-image is then treated as an image or sequence of images when the sub-image varies over time.

[0083] On the figure 5 , by the representation of the areas extracted from the intermediate view in a form suitable for coding (rectangular or at least quadrilateral shape), it appears that the sub-images include so-called useful data (51, areas with large hatching on the figure 5 ) corresponding to the extracted areas which are likely to be used for the synthesis of the intermediate view or other views, and so-called non-useful data (52, areas with fine hatching on the figure 5 ) which include texture and depth data at 0.

[0084] The useful data of the sub-image can be included in one area of ​​the sub-image or in several disjoint areas of the sub-image, so that the sub-image then includes several areas said to be useful for synthesis.

[0085] The coding method according to the invention aims to reduce the cost of coding non-useful areas.

[0086] Subsequently, the invention is described in the case where it is considered that the sub-images are grouped into a large image 50. However, the general principle of the invention applies in a similar manner to the case where the sub-images are not grouped into a large image but processed individually.

[0087] During a step E42, the sub-image(s) are coded. For this, according to the invention, two methods of coding the sub-images can be used. For a sub-image considered, a competition between the two coding methods is implemented in order to select the coding method offering the best bit rate / distortion compromise. An indicator is coded in the stream to indicate, for the sub-image, which coding method is used.

[0088] The two coding methods are described below.

[0089] According to a first coding method, the sub-image is coded using a binary map indicating, for a considered pixel of the sub-image, whether the pixel belongs to the useful zone (for example in the binary map, the pixel takes the value 1) or not (for example in the binary map, the pixel takes the value 0).

[0090] There figure 6 illustrates an example of an image (50 text ) comprising sub-images and a corresponding binary map (50 bin ) indicating the useful areas (51, hatched areas in the texture image 50 text ) and non-useful areas (52, white areas in the texture image 50 text ) of the sub-images for the image 50 illustrated in figure 5 . According to this first coding method, only the texture and depth data of the pixels located in the useful area are coded and transmitted to the decoder, the associated binary map being coded for all the pixels of the sub-image.

[0091] It should be noted here that it is not possible to use a predefined value as a reference in the texture data, for example, such as 0, which would specify that the pixel in question is not useful. Indeed, after encoding and decoding the texture, the value of the pixel is not guaranteed, and may no longer be zero. In addition, other zero values ​​may appear for pixels belonging to the useful area.

[0092] In a second coding method, the sub-image is coded using a filling of the non-useful area, both for texture data and for depth data. The sub-image is then coded in a conventional way by coding all the pixels of the sub-image.

[0093] For this, during a step E420, the coding method to be used to code the sub-image is determined, from among the first method and the second method mentioned above. For example, a competition between the coding of the two methods is implemented and the method providing the best bitrate / distortion compromise is selected. In a known manner, such a competition is carried out by a simulation of the coding respectively according to the two methods, a calculation of the bitrate used by each method and the associated distortion calculated between the original values ​​and the reconstructed values ​​of the texture and / or depth data.

[0094] In a step E421, an indicator indicating whether the sub-image is coded according to the first method or according to the second method is coded in the data stream. When the indicator indicates that the sub-image is coded according to the first method, the method proceeds to step E422. When the indicator indicates that the sub-image is coded according to the second method, the method proceeds to step E424.

[0095] In step E422, the binary map of the sub-image is encoded. For example, entropy coding is used.

[0096] In a step E423, the texture data and the depth data are coded for the sub-image by considering only the pixels located in the useful area of ​​the sub-image. These pixels are identified using the binary map. The texture data and the depth data are coded in a conventional manner, for example by an HEVC encoder or any other 2D or 3D encoder.

[0097] In step E424, the indicator indicating that the sub-image is coded according to the second method, an assignment depth value d* is determined.

[0098] For example, the assignment depth value d* is determined from a minimum depth value d min determined on at least the useful area of ​​the sub-image, reduced by a predetermined value delta. The minimum depth value d min corresponds to the smallest depth value of the pixels of the sub-image belonging to the useful area, i.e. to the depth of the pixel of the sub-image belonging to the useful area, the furthest away: d min = min (di ), where di corresponds to the depth values ​​of the pixels of the useful area and min the function returning the smallest value.

[0099] This minimum value d min is reduced by a value delta to take into account the quantization error introduced when encoding depth values. For example, the assignment value d* is obtained by d* = d min - delta.

[0100] Different values ​​of delta are possible, for example delta is equal to 10.

[0101] According to a particular embodiment of the invention, the minimum depth value is determined from the useful area of ​​several sub-images. For example, when the sub-images are grouped in a large image, the useful areas of all the sub-images are scanned in order to determine the minimum depth value among all the sub-images of the large image. Such a particular embodiment of the invention makes it possible to have a more homogeneous assignment depth value for all the sub-images of a large image.

[0102] In a step E425, the assignment depth value d* is assigned as a depth value to the pixels located outside the useful area. Such a step is illustrated in figure 8 representing depth data of a sub-image. The pixels of the useful area (51) include the depth value that was estimated for these pixels when extracting the useful area from the intermediate view. The pixels of the non-useful area (52) include the assignment depth value d*.

[0103] In step E426, a texture value is also assigned to the pixels located outside the useful area.

[0104] For example, the filling of the non-useful area is done by mirroring. The texture values ​​of the pixels outside the useful area are calculated so as to best reproduce the data present inside the useful area, as shown in the figure 7 . On the figure 7 , a sub-image is represented. Data from the useful area (51) are propagated into the non-useful area (52) symmetrically with respect to the boundary (53) between the useful area and the non-useful area.

[0105] In another example, the texture values ​​of the pixels outside (52) the useful area correspond to the average of the texture values ​​of the pixels in the useful area (51). In this example, the pixels in the non-useful area are filled with the same value.

[0106] In a step E427, the texture data and the depth data for the sub-image are coded by considering all the pixels of the sub-image. In other words, according to this second coding method, the pixels of the useful area and the pixels of the non-useful area are coded. The texture data and the depth data are coded in a conventional manner, for example by an HEVC coder or any other 2D or 3D coder.

[0107] According to a particular embodiment of the invention, the depth data are differentially coded with respect to the assignment depth value d*. According to this particular embodiment of the invention, a prediction residual is then calculated by res(x,y)=prof(x,y)- d*, with prof(x,y) the depth value for the pixel (x,y) of the sub-image and res(x,y) the residual value for the pixel (x,y) of the sub-image.

[0108] When encoding the depth data for the sub-image, it is therefore the residual as mentioned above that is encoded.

[0109] According to this particular embodiment of the invention, in order for the decoder to be able to reconstruct depth data for the sub-image from the residual, the decoder must have knowledge of the assignment depth value d*.

[0110] Thus, according to this particular embodiment of the invention, during a step E428, the assignment depth value d* is coded in the data stream.

[0111] According to a variant of this particular embodiment of the invention, the differential coding of the depth data is put in competition with a conventional coding of the depth data. Thus, according to this variant, during step E420, when the coding of the sub-image according to the second method is tested, it is also determined whether the depth data according to this second coding method are coded by a residue or not. The coding of the depth data by residue and the coding of the depth data in a conventional manner are simulated to select the coding mode providing the best rate / distortion compromise.

[0112] According to this alternative embodiment of the invention, information indicating which depth data coding mode is selected must then be coded in the data stream. Thus, during a step E429, a depth data coding indicator is coded in the data stream for the sub-image. Such an indicator indicates whether the depth data is coded by a residue or not.

[0113] There figure 9 illustrates steps of a method for decoding a data stream representative of a multi-view video according to a particular embodiment of the invention. For example, the data stream was generated by the coding method described in relation to the figure 4 .

[0114] According to the invention, the data stream comprises at least coded data relating to the coding of one or more basic views of the multi-view video and coded data representative of sub-images or patches comprising texture and depth data. Once decoded, these sub-images allow the decoder to synthesize other views of the scene which have not been coded in the data stream for example. The steps of such a decoding scheme relating to a particular embodiment of the invention are described below.

[0115] In a step E90, the base views and associated depth maps are decoded from the data stream.

[0116] According to a particular embodiment of the invention, a basic view decoded from the data stream corresponds to a view synthesized at the encoder from one or more captured views.

[0117] Next, a coded sub-image is decoded in the data stream. A sub-image is a sub-image representing a useful area extracted at the encoder from an intermediate view. This sub-image can be grouped with other sub-images into a larger image.

[0118] In a step E91, an indicator indicating for the sub-image whether the sub-image is coded according to the first method or according to the second method which have been described above, is decoded.

[0119] When the indicator indicates that the sub-picture is coded according to the first method, the method proceeds to step E92. When the indicator indicates that the sub-picture is coded according to the second method, the method proceeds to step E94.

[0120] In step E92, a binary map is decoded for the sub-image. Such a binary map indicates for a considered pixel of the sub-image, whether the pixel belongs to the useful zone or not.

[0121] In a step E93, the texture data and the depth data of the sub-image are decoded and reconstructed by considering only the pixels located in the useful area of ​​the sub-image. These pixels are identified using the binary map.

[0122] After reconstruction, the pixels of the sub-image located outside the useful area include texture values ​​and depth values ​​at 0.

[0123] When the indicator indicates that the sub-picture is encoded according to the second method, in step E94, the texture data and the depth data of the sub-picture are decoded in the following manner.

[0124] In a step E940, the texture data and the depth data of the sub-image are decoded and reconstructed in a conventional manner by considering all the pixels of the sub-image. At the end of the reconstruction, all the pixels of the sub-image comprise a texture value and a depth value. The pixels of the sub-image located outside the useful area comprise a texture value and a depth value as determined during coding.

[0125] According to a particular embodiment of the invention, when the indicator indicates that the sub-image is coded according to the second method, the decoder does not distinguish between the useful area and the non-useful area of ​​the sub-image.

[0126] However, the decoded depth value for the pixels of the non-useful area of ​​the sub-image allows a synthesis algorithm not to use these pixels when synthesizing an intermediate view. Indeed, according to a particular embodiment of the invention, such pixels have been assigned a depth value determined from the minimum depth value of the area of ​​the sub-image, reduced by a value making it possible to take into account quantization errors. So that these pixels are considered by the synthesis algorithm as the most distant points.

[0127] According to another particular embodiment of the invention, during a step E941, information representative of the assignment depth value d* assigned to the coding of the pixels located outside the useful zone of the sub-image is decoded from the data stream.

[0128] According to this particular embodiment of the invention, a binary map can thus be constructed for the sub-image in order to identify the pixels of the useful area and the pixels of the non-useful area, in the case for example where such a map would be necessary for the synthesis algorithm.

[0129] As a variant of this particular embodiment, the depth data of the sub-image has been differentially encoded with respect to the assignment depth value d*.

[0130] Thus, according to this embodiment variant, during a step E943, the depth data of the sub-image are reconstructed by adding to the depth residue values ​​of the sub-image which were decoded in step E940, the decoded assignment depth value d*.

[0131] According to another variant, during a step E942, a depth data coding indicator for the sub-image is decoded from the data stream. Such a coding indicator indicates whether the depth data of the sub-image is differentially coded or not.

[0132] According to a particular embodiment of the invention, the decoder operates in client-server mode with a server providing the data of a multi-view video in which a viewer is navigating.

[0133] According to this particular embodiment of the invention, the decoding method comprises a prior step E901 of receiving the texture and depth data of the sub-image. For example, such sub-image data is received following a request sent by the decoder for a viewing position and direction corresponding to the viewing position and direction of an observer in the viewing volume, or for a viewing position and direction close to those of the observer.

[0134] There figure 10 illustrates steps of the decoding method according to another particular embodiment of the invention. According to this particular embodiment of the invention, an infinite value is assigned to the depth values ​​of the pixels of the reconstructed sub-image which do not belong to the useful area in order to allow the synthesis algorithm to identify the useful area and the non-useful area.

[0135] For this, during a step E100, the useful area of ​​the reconstructed sub-image is determined. When the indicator decoded in step E91 indicates that the sub-image is coded according to the second method, the useful area is determined from the reconstructed depth data and the decoded assignment depth value d*. When the indicator decoded in step E91 indicates that the sub-image is coded according to the first method, the useful area is determined from the binary map decoded in step E92.

[0136] In a step E101, a depth map for the sub-image is constructed, in which the pixels of the useful area take their decoded depth value and the pixels located outside the useful area are assigned an infinite value.

[0137] During a step E102, the depth map is transmitted to a synthesis module.

[0138] There figure 11 illustrates steps of the decoding method according to another particular embodiment of the invention.

[0139] When the indicator decoded in step E91 indicates that the sub-image is coded according to the second method, during a step E110, the pixels of the useful zone are determined from the reconstructed depth data and the decoded assignment depth value d*.

[0140] In a step E111, a binary map is constructed for the sub-image. Such a binary map indicates for a considered pixel of the sub-image, whether or not the pixel belongs to the useful zone. Such a binary map is similar to the binary map which would have been decoded in the case where the sub-image had been coded according to the first method.

[0141] During a step E112, the binary map is transmitted to a synthesis module.

[0142] There figure 12 presents the simplified structure of a COD coding device suitable for implementing the coding method according to any of the particular embodiments of the invention.

[0143] According to a particular embodiment of the invention, the steps of the coding method are implemented by computer program instructions. For this, the coding device COD has the conventional architecture of a computer and notably comprises a memory MEM, a processing unit UT, equipped for example with a processor PROC, and controlled by the computer program PG stored in memory MEM. The computer program PG comprises instructions for implementing the steps of the coding method as described above, when the program is executed by the processor PROC.

[0144] At initialization, the code instructions of the computer program PG are for example loaded into a RAM memory (not shown) before being executed by the processor PROC. The processor PROC of the processing unit UT implements in particular the steps of the coding method described above, according to the instructions of the computer program PG.

[0145] There figure 13 presents the simplified structure of a DEC decoding device suitable for implementing the decoding method according to any of the particular embodiments of the invention.

[0146] According to a particular embodiment of the invention, the decoding device DEC has the conventional architecture of a computer and notably comprises a memory MEM0, a processing unit UT0, equipped for example with a processor PROC0, and controlled by the computer program PG0 stored in memory MEM0. The computer program PG0 comprises instructions for implementing the steps of the decoding method as described above, when the program is executed by the processor PROC0.

[0147] At initialization, the code instructions of the computer program PG0 are for example loaded into a RAM memory (not shown) before being executed by the processor PROC0. The processor PROC0 of the processing unit UT0 implements in particular the steps of the decoding method described above, according to the instructions of the computer program PG0.

[0148] There figure 14 illustrates an example of an STR data stream according to particular embodiments of the invention. For example, the STR data stream is generated by the encoding method as described in connection with the figure 4 .

[0149] In reference to the figure 14 , the STR data stream is representative of a multi-view video, and notably comprises V b coded data representative of one or more basic views, and coded data representative of at least one sub-image comprising texture data (text0, text1) and depth data (dpth0, dpth1) associated with said texture data.

[0150] According to a particular embodiment of the invention, a basic view coded in the data stream corresponds to a view synthesized during coding from at least one source view of the scene.

[0151] The STR data flow illustrated in figure 14is an illustrative example. It is presented here comprising two sub-images (Im0, Im1). The sub-images included in the data stream comprise an area, called useful, intended to be used to generate at least one image of at least one view of the scene captured by the multi-view video.

[0152] Depending on the different particular embodiments of the invention, the STR data stream comprises the following data for the sub-images: an indicator (Id_meth0, Id_meth1) indicating whether the sub-image is coded according to the first method or according to the second method which have been described above, when the indicator indicates that the sub-image is coded according to the first method (ld_meth1 in the example described here), coded data representative of a binary map (valid1) indicating for a considered pixel of the sub-image, whether said pixel belongs to the useful zone or not.

[0153] According to a particular embodiment of the invention, when the indicator indicates that the sub-image is coded according to the second method (Id_meth0 in the example described here), the STR data stream comprises information representative of the assignment depth value d*.

[0154] According to another particular embodiment of the invention, when the indicator indicates that the sub-image is coded according to the second method, the STR data stream comprises a coding indicator (Id_res) for the sub-image indicating whether the depth data is coded by obtaining a residue determined by a difference between the depth value of a pixel and the assignment depth value d*.

[0155] The example described above is merely illustrative and not limiting. The encoded data described above may be included in a single data stream or multiple data streams or sub-streams of a single data stream. Also, some of the data described above and other data not described may be included in metadata associated with one or more data streams.

[0156] For example, texture data and depth data can be encoded in separate streams.

[0157] In another example, encoded data for base views may be included in a video data stream, and encoded data for subpictures may be included in metadata associated with the video data stream.

Claims

1. Method for decoding at least one data stream representative of a multi-view video, said at least one data stream comprising at least coded data representative of at least one view, referred to as a basic view, of the multi-view video and of at least one sub-image, the coded data of the at least one basic view comprising texture data and depth data associated with said texture data, said sub-image comprising a zone referred to as a useful zone and a zone referred to as a non-useful zone, said useful zone corresponding to a zone extracted from a view of the multi-view video that is not coded in the data stream, referred to as an intermediate view, the coded data of the at least one sub-image comprising texture data and depth data associated with said texture data, the useful zone being intended to be used to generate at least one image of at least one other view, the decoding method comprising: - decoding, from said at least one data stream, an indicator indicating whether the sub-image is coded according to a first method or according to a second method, - when the indicator indicates that the sub-image is coded according to the first method, decoding, for the sub-image, a binary map indicating, for at least one pixel of the sub-image, whether or not said pixel belongs to the useful zone and, if said pixel belongs to the useful zone, decoding, for the sub-image, the texture data and the depth data associated with said texture data, - decoding the texture data and the depth data of the sub-image, when the indicator indicates that the sub-image is coded according to the second method, the texture data, respectively depth data, of said sub-image, for pixels located outside the useful zone, comprising an assignment texture value, respectively an assignment depth value.

2. Method for coding a data stream representative of a multi-view video, comprising coding at least one view, referred to as a basic view, of the multi-view video and at least one sub-image, said data stream comprising coded data of the at least one basic view and coded data of the at least one sub-image, the coded data of the at least one basic view comprising texture data and depth data associated with said texture data, said at least one sub-image comprising a zone referred to as a useful zone and a zone referred to as a non-useful zone, said useful zone corresponding to a zone extracted from a view of the multi-view video that is not coded in the data stream, referred to as an intermediate view, the coded data of the at least one sub-image comprising texture data and depth data associated with said texture data, the useful zone being intended to be used to generate at least one image of at least one other view, the coding of said at least one sub-image comprising: - determining a coding method for coding the sub-image from a first method and a second method, - coding an indicator indicating whether the sub-image is coded according to the first method or according to the second method, - when the indicator indicates that the sub-image is coded according to the first method, coding, for the sub-image, a binary map indicating, for at least one pixel of the sub-image, whether or not said pixel belongs to the useful zone and, if said pixel belongs to the useful zone, coding the texture data and the depth data associated with said texture data, - when the indicator indicates that the sub-image is coded according to the second method, determining an assignment texture value, respectively assignment depth value, assigning the assignment texture value, respectively assignment depth value, to the depth data of said sub-image for pixels of the sub-image located outside the useful zone, and coding the texture data and the depth data of the sub-image.

3. Coding method according to Claim 2, wherein the assignment depth value is determined from a minimum depth value determined over at least the useful zone of the sub-image minus a predetermined value.

4. Decoding method according to Claim 1 or coding method according to either one of Claims 2 and 3, furthermore comprising, when the indicator indicates that the sub-image is coded according to the second method, decoding or coding information representative of the assignment depth value, the assignment depth value corresponding to the decoded or coded depth value for pixels located outside the useful zone.

5. Method according to Claim 4, wherein the depth data of the sub-image coded in the data stream comprise, for at least one pixel of the sub-image, a residue determined by a difference between the depth value of said pixel and said assignment depth value.

6. Method according to Claim 5, comprising decoding or coding, in the data stream, an indicator for said at least one sub-image indicating whether the depth data are coded by a residue determined by a difference between the depth value of said pixel and said assignment depth value.

7. Decoding method according to any one of Claims 4 to 6, furthermore comprising: - determining the useful zone from the reconstructed depth data and the information representative of the decoded assignment depth value when the indicator indicates that the sub-image is coded according to the second method, or from the decoded binary map when the indicator indicates that the sub-image is coded according to the first method, - constructing a depth map for the sub-image, in which an infinite value is assigned to pixels located outside the useful zone, - transmitting said depth map to a synthesis module.

8. Decoding method according to any one of Claims 4 to 6, furthermore comprising, when the indicator indicates that the sub-image is coded according to the second method: - determining the useful zone from the reconstructed depth data and the information representative of the decoded assignment depth value, - constructing a binary map indicating, for at least one pixel of the sub-image, whether or not the pixel belongs to the useful zone, - transmitting said binary map to a synthesis module.

9. Method according to Claim 3, wherein at least two useful zones respectively belonging to at least two subimages, said at least two useful zones being intended to be used to generate said at least one image of at least one other view, are coded in the data stream, the minimum depth value being determined from said at least two useful zones.

10. Method according to any one of Claims 1 to 9, wherein said at least one basic view of the multi-view video corresponds to a view determined at coding from at least one source view.

11. Decoding method according to any one of Claims 1 or 4 to 10, furthermore comprising receiving the texture data and the depth data of said sub-image, said sub-image originating from an intermediate view synthesized for a position and a viewing direction of a viewer in a viewing volume.

12. Device for decoding at least one data stream representative of a multi-view video, said at least one data stream comprising at least coded data representative of at least one view, referred to as a basic view, of the multi-view video and of at least one sub-image, the coded data of the at least one basic view comprising texture data and depth data associated with said texture data, said sub-image comprising a zone referred to as a useful zone and a zone referred to as a non-useful zone, said useful zone corresponding to a zone extracted from a view of the multi-view video that is not coded in the data stream, referred to as an intermediate view, the coded data of the at least one sub-image comprising texture data and depth data associated with said texture data, the useful zone being intended to be used to generate at least one image of at least one other view, the decoding device being configured to: - decode, from said at least one data stream, an indicator indicating whether the sub-image is coded according to a first method or according to a second method, - when the indicator indicates that the sub-image is coded according to the first method, decode, for the sub-image, a binary map indicating, for at least one pixel of the sub-image, whether or not said pixel belongs to the useful zone and, if said pixel belongs to the useful zone, decode, for the sub-image, the texture data and the depth data associated with said texture data, - decode the texture data and the depth data of the sub-image, when the indicator indicates that the sub-image is coded according to the second method, the texture data, respectively depth data, of said sub-image, for pixels located outside the useful zone, comprising an assignment texture value, respectively an assignment depth value.

13. Device for coding a data stream representative of a multi-view video, configured to code at least one view, referred to as a basic view, of the multi-view video and at least one sub-image, said data stream comprising coded data of the at least one basic view and coded data of the at least one sub-image, the coded data of the at least one basic view comprising texture data and depth data associated with said texture data, said sub-image comprising a zone referred to as a useful zone and a zone referred to as a non-useful zone, said useful zone corresponding to a zone extracted from a view of the multi-view video that is not coded in the data stream, referred to as an intermediate view, the coded data of the at least one sub-image comprising texture data and depth data associated with said texture data, the useful zone being intended to be used to generate at least one image of at least one other view, the coding device being configured to: - determine a coding method for coding the sub-image from a first method and a second method, - code an indicator indicating whether the sub-image is coded according to the first method or according to the second method, - when the indicator indicates that the sub-image is coded according to the first method, code, for the sub-image, a binary map indicating, for at least one pixel of the sub-image, whether or not said pixel belongs to the useful zone and, if said pixel belongs to the useful zone, code, for the sub-image, the texture data and the depth data associated with said texture data, - when the indicator indicates that the sub-image is coded according to the second method, determine an assignment texture value, respectively assignment depth value, assign the assignment texture value, respectively assignment depth value, to the texture data, respectively depth data, of said sub-image for pixels of the sub-image located outside the useful zone, and code the texture data and the depth data of the sub-image.

14. Data stream representative of a multi-view video, comprising at least coded data representative of at least one view, referred to as a basic view, of the multi-view video and of at least one sub-image, the coded data of the at least one basic view comprising texture data and depth data associated with said texture data, said sub-image comprising a zone referred to as a useful zone and a zone referred to as a non-useful zone, said useful zone corresponding to a zone extracted from a view of the multi-view video that is not coded in the data stream, referred to as an intermediate view, the coded data of the at least one sub-image comprising texture data and depth data associated with said texture data, the useful zone being intended to be used to generate at least one image of at least one other view, the data stream comprising: - an indicator indicating whether the sub-image is coded according to a first method or according to a second method, - when the indicator indicates that the sub-image is coded according to the first method, coded data representative of a binary map indicating, for at least one pixel of the sub-image, whether or not said pixel belongs to the useful zone and, if said pixel belongs to the useful zone, the coded texture data and the coded depth data associated with said texture data, - when the indicator indicates that the sub-image is coded according to the second method, the coded texture data, respectively depth data, of said sub-image, the pixels located outside the useful zone comprising an assignment texture value, respectively an assignment depth value.

15. Computer program comprising instructions allowing a processor to implement the decoding method according to any one of Claims 1 or 4 to 10 and / or instructions for implementing the coding method according to any one of Claims 2 to 6 or 9-10 when said program is executed by said processor.