Method and device for encoding and decoding multiview video sequences

Abstract

To improve prior arts.SOLUTION: A coded data stream includes coded data representative of an atlas, the atlas corresponding to an image including a patch. The patch corresponds to a set of pixels extracted from a component of a view of a multi-view video, the view not being coded in the coded data stream. A method includes: decoding, from the coded data stream, the atlas, including decoding the patch; determining, for the decoded patch, whether a transformation, and which transformation, must be applied to the decoded patch, the transformation belonging to a group of transformation including oversampling of the patch or a modification of pixel values of the patch; and applying the determined transformation to the decoded patch.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to immersive video representing a scene captured by one or more cameras. More specifically, the present invention relates to encoding and decoding such video. [Background technology]

[0002] In the context of immersive video, where viewers feel immersed in the scene, the scene is typically captured by a set of cameras as shown in Figure 1. These cameras can be Type 2D (cameras C1, C2, C3, and C4 in Figure 1) or Type 360, which captures the entire 360-degree scene around the camera (camera C5 in Figure 1).

[0003] All of these captured views are traditionally coded and then decoded by the viewer's device. However, simply displaying captured views is insufficient to provide sufficient immersive quality and therefore the visual quality of the scene displayed to the viewer and a good sense of immersion in that scene.

[0004] To enhance immersion in a scene, one or more views, typically called intermediate views, are calculated from the decoded views.

[0005] These intermediate views can be computed using a view synthesis algorithm.

[0006] Generally, in MIV (Metadata for Immersive Video) systems, which are currently being standardized, not all of the original view—that is, everything captured by the camera—is necessarily sent to the decoder. A selection process, also known as "pruning," is performed, from at least a portion of the original view to select the data that can be used to synthesize the intermediate view.

[0007] Figure 2 shows an example of an encoding-decoding system that uses such data selection from a multiview video to synthesize intermediate views on the decoder side.

[0008] According to this method, one or more basic views (T in Figure 2) b , D b ) are encoded by a 2D encoder, such as an HEVC encoder, or by a multiview encoder.

[0009] Remaining views (T s , D s The system processes each of these views to extract a specific zone. These extracted zones, hereafter also called patches, are collected in an image called an atlas. The atlas is coded, for example, by a conventional 2D video encoder, such as an HEVC encoder. On the decoder side, the atlas is decoded, and the decoded patches are then fed into a view compositing algorithm to generate an intermediate view from the base view and the decoded patches. Overall, the patches make it possible to transmit the same zone from different viewpoints. Specifically, the patches make it possible to transmit occlusion, that is, parts of the scene that are invisible from a given view.

[0010] The MIV system (MPEG-I Part 12), in its reference implementation (TMIV, which stands for "Test Model for Immersive Video"), generates an atlas formed by a set of patches.

[0011] Figure 3 shows an example of extracting patches (patch 2, patch 5, patch 8, patch 3, patch 7) from views (V0, V1, V2) to create associated atlases, for example, two atlases A0 and A1. These atlases A0 and A1 each contain texture images T0, T1 and corresponding depth maps D0, D1. Atlas A0 has texture T0 and depth D0, and atlas A1 has texture T1 and depth D1.

[0012] As illustrated in Figure 2, patches are collected within the image and encoded by a conventional 2D video encoder. To avoid the extra costs of signal transmission and encoding of the extracted patches, it is necessary to optimize the placement of patches within the atlas. Furthermore, if there is a large amount of information that the decoder must process to reconstruct the view of the multiview video, it is necessary not only to reduce the cost of compressing such patches but also to reduce the number of pixels that the decoder needs to process. In fact, in many application areas, devices for playing such videos have more limited resources than devices for encoding such videos. [Prior art documents] [Non-patent literature]

[0013] [Non-Patent Document 1] Basel Salahieh, Bart Kroon, Joel Jung, Marek Domanski, Test Model 4 for Immersive Video, ISO / IEC JTC 1 / SC 29 / WG 11 N19002, Brussels, BE - January 2020 [Overview of the Initiative] [Problems that the invention aims to solve]

[0014] Therefore, it is necessary to improve the conventional technology. [Means for solving the problem]

[0015] The present invention improves upon the prior art. To this end, the present invention relates to a method for decoding an encoded data stream representing a multiview video, wherein the encoded data stream includes encoded data representing at least one atlas, the at least one atlas corresponds to an image including at least one patch, the at least one patch corresponds to a set of pixels extracted from at least one component of a view of the multiview video, and the view is not encoded within the encoded data stream. This decoding method is - Decoding the at least one atlas, including decoding the at least one patch from the coded data stream, - Determining whether a transformation must be applied to the at least one decoded patch, and which transformation must be applied, that the transformation belongs to a group of transformations that include at least one oversampling of the patch or modification of the pixel values ​​of the patch. - Applying the determined transformation to the decoded patch Includes.

[0016] In connection therewith, the present invention relates to a method for encoding a data stream representing a multiview video, the encoding method being - Extracting at least one patch from at least one view component of a multiview video that is not coded within the data stream, corresponding to a set of pixels in the component, - Determining whether a transformation must be applied to the at least one extracted patch, and which transformation must be applied, that the transformation belongs to a group of transformations that include at least one subsampling of the patch or modification of the pixel values ​​of the patch. - Applying the determined transformation to the at least one patch; - Encoding at least one atlas within the data stream, the at least one atlas corresponding to an image that at least includes the at least one patch; The invention also relates to a method including these steps.

[0017] Thus, the invention enables identifying which patches of the decoded atlas should be transformed during reconstruction. Such a transformation corresponds to the inverse transformation of the transformation applied during the encoding of the atlas.

[0018] The invention can also apply different transformations for each patch of the atlas, or transformations that may have different parameters for different patches.

[0019] Thus, the arrangement of the patches within the atlas is optimized for compression. In fact, the transformations used for the patches of the atlas can optimize the pixel occupancy of the atlas, on the one hand, by using transformations such as rotation, subsampling, encoding, etc. to arrange the patches within the atlas image.

[0020] On the other hand, the transformation can be optimized by modifying the cost of compressing the patches, particularly by reducing the pixel values of these patches, for example, by reducing the dynamic range of the pixels, or by using subsampling that results in encoding fewer pixels, or by using an optimal arrangement of the patches within the atlas image that enables obtaining the fewest possible pixels to be encoded. Reducing the pixel occupancy of the atlas reduces the ratio of pixels to be processed by the decoder, and thus reduces the computational complexity of decoding.

[0021] According to a particular embodiment of the present invention, whether a transformation must be applied to the at least one decoded patch is determined from at least one syntax element decoded from the coded data stream for the at least one patch. According to this particular embodiment of the present invention, the syntax element is explicitly coded in the data stream to indicate whether a transformation must be applied to the decoded patch and which transformation must be applied.

[0022] According to another particular embodiment of the present invention, the at least one decoded syntactic element includes at least one indicator indicating whether a transformation should be applied to the at least one patch, and if the indicator indicates that a transformation should be applied to the at least one patch, the at least one syntactic element optionally includes at least one parameter of the transformation. According to this particular embodiment of the present invention, the transformation to be applied to a patch is coded in the form of an indicator indicating whether a transformation should be applied to the patch, and, in the case where it should be applied, possibly one or more parameters of the transformation to be applied. For example, a binary indicator may indicate whether a transformation should be applied to a patch, and if it should be applied, the code indicates which transformation is used and possibly one or more parameters of the transformation, such as a magnification, a pixel dynamic range correction function, a rotation angle, etc.

[0023] In other embodiments, the conversion parameters can be set by default in the encoder.

[0024] According to another particular embodiment of the present invention, the at least one parameter of the transformation to be applied to the patch has a value that is predictively coded with respect to the predicted value. Thus, this particular embodiment of the present invention can save the signaling cost of the transformation parameters.

[0025] According to another specific embodiment of the present invention, the predicted values ​​are encoded within the view header, or within the component header of the atlas, or within the header of the atlas.

[0026] According to another specific embodiment of the present invention, the predicted value is - Patches previously processed according to the Atlas patch processing order, - A previously processed patch extracted from the same component as the component to which at least one patch of the multiview video view belongs. - A patch selected from the set of candidate patches using an index coded within the aforementioned data stream, - Patches selected from a set of candidate patches using the criteria. This corresponds to the value of the transformation parameter applied to the patch belonging to the group that includes it.

[0027] According to another particular embodiment of the present invention, the determination of whether a transformation should be applied to the at least one decoded patch is made when a syntactic element decoded from the header of the data stream indicates activation of the application of the transformation to the patch encoded in the data stream, and the syntactic element is encoded in the header of the view, or in the header of the view components, or in the header of the atlas. According to this particular embodiment of the present invention, a high-level syntactic element is encoded in the data stream to signal the use of a transformation to be applied to a patch in a multiview video. Thus, the additional cost that would arise from encoding the transformation parameters at the patch level is avoided when these transformations are not used. In addition, this particular embodiment of the present invention can limit the computational complexity of decoding when these transformations are not used.

[0028] According to another particular embodiment of the present invention, if the characteristics of at least one decoded patch meet a criterion, it is determined that a transformation must be applied to the decoded patch. According to this particular embodiment of the present invention, the instruction indicating the use of a transformation to be applied to a patch is not explicitly coded in the data stream. Such an instruction is inferred from the characteristics of the decoded patch. In this particular embodiment of the present invention, patch transformations can be used without requiring the additional cost of coding to signal the use of a transformation.

[0029] According to another specific embodiment of the present invention, the characteristic corresponds to a ratio R = H / W, where H corresponds to the height of the at least one decoded patch and W corresponds to the width of the at least one decoded patch, and the transformation to be applied to the at least one patch corresponds to vertical oversampling at a predetermined magnification when the ratio falls within a determined interval. Thus, according to this particular implementation mode of the present invention, it is possible to mix "long" patches, for which subsampling is not of interest, and "long" patches for which subsampling is performed without the need to signal that subsampling is performed on them, within the same atlas.

[0030] According to another specific implementation mode of the present invention, the characteristic corresponds to an energy E calculated from the pixel values ​​of the at least one decoded patch, and when the energy E is lower than a threshold, the transformation to be applied to the at least one patch corresponds to the multiplication of the pixel value by a determined magnification factor.

[0031] According to another specific embodiment of the present invention, when several transformations must be applied to the same patch, the order in which the transformations must be applied is predetermined. In this particular embodiment of the present invention, no signaling is required to indicate the order in which the transformations are applied. This order is determined in the encoder and decoder and remains unchanged for all patches to which these transformations are applied.

[0032] The present invention relates to a device for decoding an encoded data stream representing a multiview video, wherein the encoded data stream includes encoded data representing at least one atlas, the at least one atlas corresponds to an image including at least one patch, the at least one patch corresponds to a set of pixels extracted from at least one component of a view of the multiview video, the view is not encoded within the encoded data stream, and the decoding device - Decode the at least one atlas, including decoding the at least one patch from the coded data stream. - Determining whether a transformation must be applied to the at least one decoded patch, and which transformation must be applied, that the transformation belongs to a group that includes at least one oversampling of the patch or modification of the pixel values ​​of the patch. - Apply the determined transformation to the decoded patch. This also relates to devices that have a processor and memory configured to perform this task.

[0033] According to a particular embodiment of the present invention, such a device is contained within a terminal.

[0034] The present invention is a device for encoding a data stream representing a multiview video, - Extracting at least one patch from at least one view component of a multiview video that is not coded within the data stream, corresponding to a set of pixels of the component. - For the at least one extracted patch, determine whether a transformation must be applied to the at least one patch, and which transformation must be applied, that the transformation belongs to a group of transformations that include at least one subsampling of the patch or modification of the pixel values ​​of the patch. - Apply the determined transformation to at least one of the patches. - Encoding at least one atlas within the data stream, wherein the at least one atlas corresponds to an image containing at least one patch. This also relates to devices that have a processor and memory configured to perform this task.

[0035] According to a particular embodiment of the present invention, such a device is contained within a terminal.

[0036] Each encoding or decoding method according to the present invention can be implemented in various ways, particularly in a wired or software-based manner. According to one particular embodiment of the present invention, the encoding or decoding method is implemented by a computer program. The present invention also relates to a computer program which, when executed by a processor, includes instructions for implementing the encoding or decoding method according to any one of the particular embodiments described above. Such a program may use any programming language. Such a program may be downloaded from a communication network and / or recorded on a computer-readable medium.

[0037] This program can use any programming language and can take the form of source code, object code, or intermediate code between source code and object code, such as partially compiled code or any other desired form.

[0038] The present invention also relates to a computer-readable storage medium or data medium containing the instructions of the computer program described above. The recording medium described above can be any entity or device capable of storing the program. For example, the medium may include storage means such as ROM, for example, CD-ROM or miniature electronic circuit ROM, a USB flash drive, or magnetic recording means, for example, a hard drive. On the other hand, the recording medium may correspond to a transmittable medium such as electrical signals or optical signals that can be transmitted wirelessly or by other means, via electrical or optical cables. The program according to the present invention can be downloaded, in particular, over an internet-type network.

[0039] Alternatively, the recording medium may correspond to an integrated circuit in which a program is embedded, and this circuit is adapted to perform the method or to be used to perform the method.

[0040] Other features and advantages of the present invention will become clearer upon reading the following description of a particular embodiment provided as a simple, illustrative, and non-limiting example, and the accompanying drawings. [Brief explanation of the drawing]

[0041] [Figure 1] This diagram schematically illustrates an example of a multi-view scene capture system. [Figure 2] This diagram schematically illustrates an example of a multiview encoder based on patch coding. [Figure 3] This figure shows an example of patch extraction and atlas creation. [Figure 4] This figure shows the steps of a coding method according to a specific embodiment of the present invention. [Figure 5] This figure shows the steps of a decoding method according to a specific embodiment of the present invention. [Figure 6] This figure shows an example of a data stream according to a specific embodiment of the present invention. [Figure 7] This figure shows an example of an architecture for a coded device according to a specific embodiment of the present invention. [Figure 8] This figure shows an example of a decoding device architecture according to a specific embodiment of the present invention. [Modes for carrying out the invention]

[0042] Figure 4 shows the steps of a multiview video coding method in at least one coded data stream according to a particular embodiment of the present invention.

[0043] According to the present invention, a multiview video is coded according to the coding scheme shown with respect to Figure 2, in which one or more basic views are coded in the data stream, and sub-images or patches containing texture and depth data are also coded in the data stream. These patches originate from additional views that are not necessarily fully coded in the data stream. Such patches and one or more basic views enable the decoder to synthesize other views of the scene, which may hereafter be called virtual views, or composite views, or even intermediate views. These composite views are not coded in the data stream. A step of such a coding scheme relating to a particular embodiment of the present invention is described below.

[0044] For example, here the scene is as shown in Figure 1, with cameras C1, C2, ..., C N It is thought that the view is captured by a set of cameras. Each camera generates a view that contains at least one so-called texture component that changes over time. In other words, the texture component of a view is a sequence of 2D images corresponding to the images captured by the cameras positioned at the viewpoint of that view. Each view also contains a depth component called a depth map, which is determined for each image in the view.

[0045] Depth maps can be generated using known methods, either by estimating depth using textures or by capturing volumetric data from a scene using light detection and ranging (Lidar) techniques.

[0046] Hereafter, the term "view" will be used to refer to a sequence of texture images and depth maps representing a scene captured from a particular viewpoint. Due to linguistic misuse, the term "view" may also refer to the texture images and depth maps of a view at a given time.

[0047] Once the view of the multiview video is captured, the encoder then proceeds with the steps described below, using an encoding scheme defined, for example, in Basel Salahieh, Bart Kroon, Joel Jung, Marek Domanski, Test Model 4 for Immersive Video, ISO / IEC JTC 1 / SC 29 / WG 11 N19002, Brussels, BE - January 2020.

[0048] In step E40, one or more basic views are selected from the captured views of the multiview video.

[0049] The base view is selected from the captured set of views in the multiview video using a known method. For example, spatial subsampling can be used to select one view from two. In another example, the content of the views can be used to determine which view should be retained as the base view. In yet another example, camera parameters (position, orientation, focus) can be used to determine which view must be selected as the base view. At the end of step E40, a certain number of views that will become the base view are selected.

[0050] The remaining views that are not selected as the primary view are called "additional views."

[0051] In step E41, a pruning method is applied to the additional views to identify one or more patches to be sent to the decoder for each additional view. In this step, the patches to be sent are determined by extracting zones from the additional view images that are necessary for intermediate view synthesis. For example, such zones may correspond to invisible occlusion zones within the base view, or visible zones within the base view that have been affected by changes in lighting or have lower quality. The extracted zones can be of any size and shape. Clustering of neighboring pixels is performed to create one or more rectangular patches from the extracted zones of the same view that are easier to encode and place.

[0052] In step E42, for each patch, the encoder determines one or more transformations to be applied to the patch when it is placed in the atlas.

[0053] It should be recalled that a patch can have texture components and / or depth components.

[0054] Patches are placed within the atlas to minimize the coding cost of the atlas and / or to reduce the number of pixels that need to be processed by the decoder. To achieve this, patches are placed within the atlas. • Subsampling of 1 / Nv in vertical dimensions • Subsampling of 1 / Nh in the horizontal dimension • Subsampling of 1 / Ne at each dimension • Correction of pixel values ​​included in the patch • Rotation of patch angle i*90°, where i=0, 1, 2, or 3. It can undergo conversions that include [this].

[0055] Next, the encoder checks each patch and determines one or more transformations to apply to each patch.

[0056] In one variant, the list of transformations to be tested for a patch may include the "identity" transformation, or in other words, the zero transformation (no transformation).

[0057] The selection of a transformation from among the possible transformations can be made by evaluating a rate-distortion criterion, which is calculated using the rate required to encode the transformed patch for the reconstructed signal, and the distortion calculated between the original patch and the coded, then reconstructed, transformed patch. The selection can also be based on an assessment of the quality of the additional view synthesized using the patch being processed.

[0058] For each transformation, one or more parameters can be tested.

[0059] For example, in the case of subsampling, various magnifications Nv, Nh, and Ne can be tested. In one preferred embodiment, the magnifications Nv, Nh, and Ne are equal to 2. In other embodiments, other values ​​are possible, such as 4, 8, or 16.

[0060] A transformation that corresponds to a change in pixel values ​​is also called a "mapping." Such a mapping transformation may consist of, for example, dividing all pixel values ​​in a patch by a given value Dv, where Dv is equal to 2. However, other values ​​such as 4, 8, or 16 are possible.

[0061] In another example, the mapping is a parameterized function f PAlternatively, the x-values ​​of pixels may be transformed into new y-values ​​using (x)=y. Such a function could be, for example, a linear function per part, where each part is parameterized by its starting x-coordinate x1, as well as the parameters a and b of the linear function y=ax+b. In this case, the parameters P of the transformation would be a triplet list (x1, a, b) for each linear part of the mapping.

[0062] In another example, the mapping could be a lookup table (LUT), which is a table that associates a value y with an input x.

[0063] In the case of rotational transformations, the transformation can be a 180° vertical rotation, also known as a vertical inversion. Other rotational parameter values, such as i*90°, where i=0, 1, 2, or 3, can also be tested.

[0064] In determining the transformation associated with a patch, the number of atlases available for encoding the multiview video can be taken into account to optimize the atlas encoding rate / distortion cost, or the overall quality of the intermediate view compositing, and the placement of patches within the atlas can also be simulated.

[0065] At the end of step E42, a list of the converted patches becomes available. Each patch is associated with the conversions determined for that patch and the associated parameters.

[0066] During step E43, these patches are placed within one or more atlases. The number of atlases depends on parameters defined as inputs to the encoder, such as the size (length and height) of the atlases and the maximum number M of pixels for texture and depth in all atlases per given time or image. This maximum number M corresponds to the number of pixels in the multiview video that the decoder should process at once.

[0067] In the specific embodiment described here, each base view is considered to constitute a patch containing the texture and depth components of that base view at a given time, and this patch is coded within the atlas. In this particular embodiment, there are as many atlases as there are base views, and as many atlases as are needed to move all the patches extracted from additional views.

[0068] Depending on the size of the atlas given as input, the atlas may consist of a base view and patches, or the base view may be divided and represented on several atlases if the size of the view is larger than the size of the atlas.

[0069] According to the specific embodiment described herein, the atlas patch may correspond to the entire image of the base view, a portion of the base view, or a zone extracted from an additional view.

[0070] The texture pixels of the patch are placed within the texture components of the atlas, and the depth pixels of the patch are placed within the depth components of the atlas.

[0071] An atlas may contain only one texture component or one depth component, or it may contain both a texture component and a depth component. In other examples, an atlas may also contain other types of components that contain information useful for intermediate view compositing. For example, other types of components may include information such as a reflectance index to show how transparent a corresponding zone is, or confidence information about the depth value at that location.

[0072] During step E43, the encoder scans all patches in the patch list. For each patch, the encoder determines in which atlas this patch will be coded. This list includes both transformed and untransformed patches. Untransformed patches are those that have undergone a zero transformation or identity transformation, containing zones extracted from additional views, or containing images from the base view. Here, we consider a patch to have already been transformed when it should have been transformed.

[0073] An atlas is a set of patches spatially rearranged within an image. This image is intended to be coded. The purpose of this arrangement is to make the most use of the space within the atlas image to be coded. In fact, one of the goals of video coding is to minimize the number of pixels that need to be decoded before a view can be synthesized. To this end, patches are arranged within the atlas so as to maximize the number of patches within the atlas. Such a method is described in Basel Salahieh, Bart Kroon, Joel Jung, and Marek Domanski, Test Model 4 for Immersive Video, ISO / IEC JTC 1 / SC 29 / WG 11 N19002, Brussels, BE - January 2020.

[0074] Following step E43, a list of patches for each atlas is generated. Note that this arrangement also determines the number of atlases that need to be coded for a given time.

[0075] During step E44, the atlas is coded into the data stream. In this step, each atlas, which includes texture and / or depth components in the form of a 2D image, is coded using a conventional video encoder such as HEVC, VVC, MV-HEVC, or 3D-HEVC. As explained above, the base view is considered as a patch here. Therefore, coding the base view is involved in coding the atlas.

[0076] During step E45, information related to each atlas is encoded within the data stream. This information is generally encoded by an entropy encoder.

[0077] For each atlas, the patch list includes the following items for each patch in the list: • The location of a patch within the atlas, in the form of 2D coordinates, for example, the position of the upper-left corner of the rectangle representing the patch. • The location of the patch in its original view, in the form of 2D coordinates, i.e., the position of the patch within the image of the view from which the patch was extracted, for example, the position of the upper-left corner of the rectangle representing the patch within the image. • Patch dimensions (length and height), • Identifier of the original view of the patch, • Information about the transformations applied to the patch.

[0078] In step E45, for at least some patches of the atlas, information about the transformations to be applied to the patches during decoding is encoded within the data stream. The transformations to be applied to the patches during decoding correspond to the inverse transformations determined above, which were applied to the patches when the atlas patches were placed.

[0079] In one particular embodiment of the present invention, information indicating the transformation to be applied is transmitted for each patch.

[0080] In the specific embodiments described herein, it is assumed that what is indicated is the transformation to be applied to decoding, not the transformation applied to encoding (corresponding to the inverse transformation of decoding). For example, when subsampling is applied during encoding, oversampling is applied during decoding. In other specific embodiments of the present invention, it will be clearly understood that the transmitted information regarding the transformation to be applied may correspond to information indicating the transformation applied to encoding, in which case the decoder infers the transformation to be applied from this information.

[0081] For example, information indicating the transformation to be applied can be an index indicating the transformation to be applied within a list of possible transformations. Such a list may further include the identity transformation. Therefore, in the case where the zero transformation is applied to the patch, an index indicating the identity transformation can be coded.

[0082] In another embodiment, a binary indicator may be coded to indicate whether or not a patch is converted, and if the binary indicator indicates that the patch is converted, an index is coded indicating which conversion from a list of possible conversions should be applied.

[0083] In one embodiment where only one conversion to be applied is possible, only a binary indicator can be coded to indicate whether or not the patch is converted.

[0084] The list of possible transformations can be known to the decoder and therefore does not need to be transmitted within the data stream. In other embodiments, the list of possible transformations can be coded within the data stream, for example, in the view header or the header of a multiview video.

[0085] The parameters associated with the transformation to be applied can also be defined by default and made known to the decoder. In another specific embodiment of the present invention, the parameters associated with the transformation applied to a patch are encoded within the data stream for each patch.

[0086] When the transformation corresponds to oversampling in one or both dimensions (equivalent to the same subsampling during coding), the parameters associated with the transformation can correspond to interpolation values ​​to be applied to all dimensions or interpolation values ​​to be applied to each dimension.

[0087] When a transformation corresponds to modifying the pixel values ​​of a patch to be coded by mapping using parameters, these transformation parameters correspond to the properties of the mapping to be applied, i.e., parameters of a partially linear (linear by parts) linear function, a lookup table (LUT), etc. In particular, possible LUTs can be known to the decoder.

[0088] When the transformation involves rotation, the parameter corresponds to the rotation angle selected from among the possible rotations.

[0089] The parameters associated with the transformation can be coded as they are, or coded by a prediction of the predicted value.

[0090] In one embodiment of a modified form, a predicted value can be defined to predict the value of a parameter, and this predicted value can be coded within the data stream, either in the header of the view containing the current patch, or in the component header, or in the image header of the view, or even in the atlas header.

[0091] Therefore, for a given atlas, the parameter value P is predicted by a value Ppred coded at the atlas level. In this case, the difference between Ppred and P is coded for each patch of the atlas.

[0092] In another embodiment, to predict the parameter value, the predicted value Ppred may correspond to the parameter value used in a previously processed patch. For example, a previously processed patch may be a previous patch in the patch processing sequence, or a previous patch belonging to the same view as the current patch.

[0093] Parameter predictions can also be obtained through a mechanism similar to the "merge" mode of the HEVC encoder. For each patch, a list of candidate patches is defined, and an index pointing to one of these candidate patches is coded for that patch.

[0094] In another embodiment, the index need not be transmitted because the patch can be identified from the list of candidate patches using a criterion. Thus, for example, a patch that maximizes the degree of similarity with the current patch can be selected, or even a patch whose dimensions are closest to those of the current patch can be selected.

[0095] In other alternative embodiments, if the use of a transformation is enabled, information indicating whether a patch has to undergo the transformation can be decomposed into a part indicating the use of the transformation (e.g., a binary indicator) and a part indicating the parameters of the transformation. This signaling mechanism can be used independently for each possible transformation on the patch.

[0096] In a particular embodiment of the invention, to activate the use of a determined transformation on an atlas patch, a view patch, or a component patch, a binary indicator can be coded at the level of the header of that atlas, or the header of that view, or the header of that component. In that case, the application of the determined transformation to the patch depends on the value of this binary indicator.

[0097] For example, two binary indicators I A and I B associated respectively with the activation of transformation A and the activation of transformation B are coded within the header of the atlas. The value of binary indicator I A indicates that the use of transformation A is possible, while the value of binary indicator I B indicates that the use of transformation B is not possible. In this example, for each patch, the binary indicator indicates whether transformation A is applied to that patch and perhaps the associated parameters. In this example, for each patch, there is no need to code a binary indicator to indicate whether transformation B is applied to that patch.

[0098] This particular embodiment, which activates the use of the conversion at the patch level or at a higher level, can save on signal transmission costs, especially when no patch uses this conversion.

[0099] If this binary activation indicator is coded at the view or component level, its value applies to all patches belonging to that view or component, regardless of which atlas the patch is coded in. Therefore, an atlas may contain patches to which a particular transformation can be applied according to the indicator coded for that patch, and patches to which the same transformation cannot be applied. In the case of the latter patches, the indicator for this transformation is not coded in the patch information.

[0100] In another specific embodiment of the present invention, information indicating a transformation is not encoded at the patch level. The transformation is inferred in the decoder from the characteristics of the patch. The transformation is then applied to the patch as soon as certain criteria are met. This particular mode is described in more detail below with respect to the decoding process.

[0101] Figure 5 shows steps of a method for decoding an encoded data stream representing a multiview video according to a particular embodiment of the present invention. For example, the encoded data stream is generated by the encoding method described with respect to Figure 4.

[0102] During step E50, the atlas information is decoded. This information is generally decoded by a suitable entropy decoder.

[0103] The information includes a list of patches and, for each patch, the following elements: - Locations within the atlas that take the form of patches and coordinates. - The location of the patch in the form of coordinates within the patch's original view. - Patch dimensions, - Identifier of the original view of the patch, - Information indicating whether a conversion must be applied to the patch.

[0104] Similar to the coding method, this information can serve as an index indicating a conversion from a list of possible conversions, or as an indicator for each possible conversion showing whether a conversion must be applied to the patch.

[0105] For transformations that correspond to the same oversampling in both dimensions, the information can be either a binary indicator showing the use of the transformation, or an interpolation value that should be applied to all dimensions.

[0106] For transformations that correspond to separate oversampling in two dimensions, the information can correspond to a binary indicator showing the use of the transformation, or to the interpolation values ​​to be applied for each dimension.

[0107] For transformations that involve modifying pixels in a patch to be decoded by mapping using parameters, the information may include informational items indicating the use of the mapping and possibly information representing the properties of the mapping to be applied (parameters of a partially linear linear function, a lookup table, etc.).

[0108] In the case of a transformation involving rotation, the parameter indicates which rotation was selected from among the possible rotations.

[0109] The transmitted information, which identifies the transformation to be applied to the patch, is decoded in a format suitable for the applied encoding. Therefore, the information can be decoded either directly or predictively, in the same format as the encoder.

[0110] According to a particular embodiment of the present invention, when the use of the conversion is activated, the information for identifying the conversion to be applied to the patch can include a portion indicating the use of the conversion (binary indicator) and a portion indicating the parameters of the conversion.

[0111] Similar to the case of the encoding method, according to a particular embodiment of the present invention, the decoding of an item of information for identifying the conversion to be applied to a given patch can depend on the activation binary indicator encoded in the header of the atlas, in the header of the view, or in the header of the component to which the patch belongs.

[0112] According to another particular embodiment of the present invention, the information for identifying the conversion to be applied to the patch is not encoded together with the patch information, but is obtained from the characteristics of the decoded patch.

[0113] For example, in one embodiment, the energy of the decoded pixels in the patch is measured by calculating the root mean square error of the patch. If this energy is below a given threshold, for example a root mean square error of less than 100, the pixel values of the patch are converted by multiplying all the values of the patch by a specified magnification Dv. For example, Dv = 2. Other thresholds and other patch value correction magnifications are possible.

[0114] According to another variant, if the ratio of the decoded dimensions of the patch, where H is the height of the patch and W is the length of the patch, is within a given range, for example 0.75 < H / W < 1.5, the patch is interpolated in the vertical dimension at a given magnification, for example magnification 2. The patch dimensions considered here are the patch dimensions decoded from the atlas information in which the patch is encoded. These are the dimensions of the patch before conversion for the decoder (and thus after conversion for the encoder). When the H / W ratio is determined to be within the determined range, the patch is oversampled and its dimensions are recalculated as a result.

[0115] With this variant, it becomes possible to mix, within the same atlas, "long" patches for which subsampling is not of interest with respect to them and "long" patches for which subsampling is performed on them without signaling to make them comply with the criteria allowing them to be interpolated in the decoder. Other thresholds can be used, for example more restrictive values such as 0.9 < H / W < 1.1 etc.

[0116] During step E51, the components of the atlas are decoded. Each atlas containing 2D texture components and / or 2D depth components is decoded using a conventional video decoder such as AVC or HEVC, VVC, MV-HEVC, 3D-HEVC etc.

[0117] During step E52, the transformation identified in step E50 is applied to the texture component and / or depth component of each patch within the atlas of each patch, depending on whether the transformation is applied to the texture component, or to the depth component, or to both components, so that the decoded patch is reconstructed.

[0118] In the case of additional views, this step consists of modifying each patch individually by applying the transformation identified for this patch. This can be done in several ways, for example by modifying the pixels of that patch within the atlas containing the patch, by copying the modified patch into a buffer memory zone, or by copying the transformed patch into its associated view.

[0119] Depending on the information decoded previously, one of the following transformations can be applied to each patch to be reconstructed. · Subsampling by a factor of Nv in the vertical dimension, · Subsampling by a factor of Nh in the horizontal dimension, · Subsampling by a factor of Ne in each dimension, · Modification of the pixel values contained within the patch • Patch rotation.

[0120] The modification of pixel values ​​is similar to coding and decoding. Note that the transmitted mapping parameters can be either encoder mapping parameters (in which case the decoder must apply the inverse function of that mapping) or decoder mapping parameters (in which case the encoder must apply the inverse function of that mapping).

[0121] According to a particular embodiment of the present invention, it is possible to apply several transformations to a patch to the encoder. These transformations are signaled within the information encoded about the patch in the stream, or otherwise inferred from the characteristics of the decoded patch. For example, the encoder may subsample the patch by half at each dimension, followed by mapping the pixel values ​​of the patch, and then rotation.

[0122] According to this particular embodiment of the present invention, the order of transformations to be applied is predetermined and known to the encoder and decoder. For example, in the encoder, the order is: rotation, then subsampling, then mapping.

[0123] When a decoder reconstructs a patch, if several transformations must be applied to the patch, they are applied in reverse order (mapping, oversampling, then rotation). Therefore, both the decoder and the encoder know in what order the transformations should be applied to produce the same result.

[0124] At the end of step E52, the reconstructed set of patches becomes available.

[0125] During step E53, at least one intermediate view is synthesized using at least one base view and at least one previously reconstructed patch. The selected virtual view synthesis algorithm is applied to the decoded and reconstructed data of the multiview video sent to the decoder. As previously described, this algorithm uses the pixels of the base view component and the patch view component to generate a view from the viewpoint between the cameras.

[0126] For example, a synthesis algorithm generates an intermediate view using at least two textures and two depth maps from a base view and / or additional views. Synthesizers are known to exist, and they belong to the DIBR category (Depth Image Based Rendering), for example. For example, frequently used algorithms by standard mechanisms are as follows: - Developed by Nagoya University and enhanced by MPEG, VSRS, representing View Synthesis Reference Software, applies a forward projection of the depth map using homography between the reference view and the intermediate view, followed by a filling step to remove artifacts from forward warping. - RVS, which represents the Reference View Synthesizer, initiated by the University of Brussels and refined by Philips, begins by projecting a reference view using calculated parallax. The reference is partitioned into triangles and distorted. The distorted views of each reference are then blended together, and basic restoration filling is applied to fill in the disocclusion. - VVS, developed by Orange and representing the Versatile View Synthesizer, sorts the references, applies a transformation to certain depth map information, and then conditionally merges these depths. Next, it applies reverse warping of textures, followed by merging of various textures and depths. Finally, it applies spatiotemporal restoration and then spatial filtering of the intermediate images.

[0127] Figure 6 shows an example of a data stream according to a particular embodiment of the present invention, in particular showing atlas information encoded within the stream and used to identify one or more transformations to be applied to the atlas patch. For example, this data stream is generated by an encoding method according to one of the particular embodiments described with respect to Figure 4 and is suitable for decoding by a decoding method according to one of the particular embodiments described with respect to Figure 5.

[0128] According to this particular embodiment of the present invention, such streams include, among other things, the following: - An Act code is encoded within the atlas header to indicate whether a given transformation is activated or not. Trf indicator, - Predicted value Ppred, which serves as a predicted value for the transformation parameter value. - Number of coded patches in the atlas Np, - For each patch in the atlas, patch information, in particular, a TRF indicator showing whether a conversion is used for the patch, - When the TRF indicator indicates the use of a transformation to a patch, the transformation parameter Par takes the form of the residual obtained with respect to the predicted value Ppred, for example, if Ppred is coded.

[0129] As described above with respect to the encoding and decoding methods, further specific embodiments of the present invention are possible with respect to the transformation-related information encoded for the patch.

[0130] Figure 7 shows a simplified structure of a coding device (COD) adapted to carry out a coding method according to any one of the specific embodiments of the present invention.

[0131] According to a particular embodiment of the present invention, the steps of the coding method are carried out by computer program instructions. For this purpose, the coding device COD has a standard computer architecture and comprises, in particular, a memory MEM and a processing unit UT which includes, for example, a processor PROC and is driven by a computer program PG stored in the memory MEM. The computer program PG includes instructions for carrying out the steps of the coding method described above when the program is executed by the processor PROC.

[0132] During initialization, the code instructions of the computer program PG are loaded, for example, into RAM memory (not shown) and then executed by the processor PROC. Specifically, the processor PROC of the processing unit UT performs the steps of the coding method described above according to the instructions of the computer program PG.

[0133] Figure 8 shows a simplified structure of a decoding device DEC adapted to carry out a decoding method according to any one of the specific embodiments of the present invention.

[0134] According to a particular embodiment of the present invention, the decoding device DEC has a standard computer architecture and comprises, in particular, a memory MEM0 and a processing unit UT0 which includes, for example, a processor PROC0 and is driven by a computer program PG0 stored in the memory MEM0. The computer program PG0 includes instructions for performing the steps of the decoding method described above when the program is executed by the processor PROC0.

[0135] During initialization, the code instructions of the computer program PG0 are loaded, for example, into RAM memory (not shown) and then executed by the processor PROC0. Specifically, the processor PROC0 of the processing unit UT0 performs the steps of the decoding method described above according to the instructions of the computer program PG0. [Explanation of Symbols]

[0136] A0 Atlas A1 Atlas COD Coding Device C1 Camera C2 Camera C3 Camera C4 camera C5 camera DEC Decryption Device D b Basic View D s Remaining views D0 depth map, depth D1 depth map, depth MEM memory MEM0 memory PG (Computer Program) PG0 Computer Program PROC Processor PROC0 Processor T b Basic View T s Remaining views T0 texture image, texture T1 texture image, texture UT processing unit UT0 Processing Unit V0 View V1 View V2 View

Claims

1. A method for decoding an encoded data stream representing a multiview video, wherein the encoded data stream includes encoded data representing at least one atlas, the at least one atlas corresponds to an image including at least one patch, the at least one patch corresponds to a set of pixels extracted from at least one component of a view of the multiview video, the view is not encoded in the encoded data, and the decoding method is - A step of decoding the at least one atlas, including decoding the at least one patch from the coded data stream, - A step of applying a modification to the pixel value of the at least one decoded patch by reducing the dynamic range of the pixel value. Methods that include...

2. The method according to claim 1, wherein the modification is determined from at least one syntactic element decoded from the coded data stream for the at least one decoded patch.

3. The method according to claim 2, wherein the at least one decoded syntactic element includes at least one indicator indicating whether a modification must be applied to the at least one patch, and if the indicator indicates that a modification must be applied to the at least one patch, the at least one syntactic element optionally includes at least one parameter of the modification.

4. The method according to claim 3, wherein the at least one parameter of the modification to be applied to the patch has a value that is predictively coded with respect to the predicted value.

5. The method according to claim 4, wherein the predicted value is coded in the header of the view, or in the header of the component of the atlas, or in the header of the atlas.

6. The aforementioned predicted value is, - Patches previously processed according to the processing order of the patches in the aforementioned atlas, - A previously processed patch extracted from the same component as the component to which the at least one patch belongs in the view of the multiview video, - A patch selected from the set of candidate patches using an index coded within the aforementioned data stream, - Patches selected from a set of candidate patches using selection criteria The method according to claim 4, which corresponds to the value of the modification parameter applied to a patch belonging to the group including

7. The method according to claim 1, wherein the modification is applied to at least one decoded patch if a syntactic element decoded from the header of the data stream indicates activation of the application of the conversion to the patch coded in the data stream, the syntactic element being coded in the header of the view, or in the header of the components of the view, or in the header of the atlas.

8. The method according to claim 1, wherein if the characteristics of the at least one decoded patch meet the criteria, a modification is applied to the decoded patch.

9. A method for encoding a data stream representing a multiview video, wherein the encoding method is - A step of extracting at least one patch from at least one component of the view of the multiview video that is not coded in the data stream, corresponding to a set of pixels of the component, - A step of applying a modification to the pixel values ​​of the at least one extracted patch by reducing the dynamic range of the pixel values, - A step of encoding at least one atlas within the data stream, wherein the at least one atlas corresponds to an image containing at least one extracted patch. Methods that include...

10. A device for decoding an encoded data stream representing a multiview video, wherein the encoded data stream includes encoded data representing at least one atlas, the at least one atlas corresponds to an image including at least one patch, the at least one patch corresponds to a set of pixels extracted from at least one component of a view of the multiview video, the view is not encoded within the encoded data stream, and the decoding device is - Decode the at least one atlas, including decoding the at least one patch from the coded data stream. - Applying a modification to the pixel value of the at least one decoded patch by reducing the dynamic range of the pixel value. A device having a processor and memory configured to perform the following actions.

11. A device for encoding a data stream representing a multiview video, - Extracting at least one patch from at least one component of the view of the multiview video that is not coded within the data stream, corresponding to a set of pixels of the component. - Applying a modification to the pixel values ​​of the at least one extracted patch by reducing the dynamic range of the pixel values, - Encoding at least one atlas within the data stream such that the at least one atlas corresponds to an image containing at least one extracted patch. A device having a processor and memory configured to perform the following actions.

12. A computer program, which, when executed by a processor, includes instructions for carrying out the decoding method described in any one of claims 1 to 8 and / or instructions for carrying out the coding method described in claim 9.

Images