Multi-view video decoding method and apparatus, and image processing method and apparatus

Abstract

The invention relates to a multi-view video decoding method and apparatus, and an image processing method and apparatus. The present invention relates to a method and an apparatus for decoding a data stream representing a multi-view video. A syntax element is obtained from at least a portion of the streaming data (E20) and used to reconstruct at least one image of a view of the video (E21). Subsequently, at least one metadata (E23) in a predetermined form is obtained from the obtained at least one syntax element and provided to an image processing module (E24). The invention also relates to an image processing method and device configured to read the at least one metadata in a predetermined form and to generate at least one image of a virtual view from a reconstructed view of a multi-view video using the at least one metadata.

Description

[0001] This application is a divisional application of the invention patent application with application number 201980029982.8, application date April 16, 2019, and invention name “Multi-view video decoding method and device and image processing method and device”. Technical Field

[0002] The present invention generally relates to the field of 3D image processing, and more particularly to decoding of multi-view image sequences and synthesis of intermediate view images. Background Art

[0003] In the field of virtual reality, free navigation allows the viewer to see the scene from any viewpoint, whether that viewpoint corresponds to a viewpoint captured by a camera or a viewpoint not captured by a camera. This view not captured by a camera is also called a virtual view or an intermediate view, because it lies between the views captured by the camera and must be synthesized for restoration.

[0004] Free navigation in a scene requires that each movement of a user watching a multi-view video be managed correctly and that the discomfort that a viewer may feel when the images are not optimally presented is avoided.

[0005] Typically, a restoration device, such as a HMD (Head Mounted Device) virtual reality helmet, will correctly account for the user's movements.

[0006] However, providing the correct pixels for display regardless of the user's movement (rotation or translation) remains a problem. In fact, the calculation of the image to be displayed requires the use of multiple captured views to enable the display of images of additional virtual (i.e. synthetic) views. Such virtual views correspond to viewpoints that are not captured by the camera. Therefore, the virtual views need to be calculated from the decoded captured views and the associated depth.

[0007] Therefore, a codec that provides free navigation functionality must be used efficiently to encode multiple views and the associated depth, and allow optimal rendering of virtual views, i.e., display using synthesis algorithms is required.

[0008] Multi-view video encoders designed for decoding multi-view sequences are known, such as the standards MV-HEVC or 3D-HEVC (H Series: Audiovisual and Multimedia Systems - Audiovisual Services Infrastructure - Motion Video Coding, High Efficiency Video Coding, ITU-T H.265 Recommendation, International Telecommunication Union, December 2016).

[0009] The MV-HEVC encoder applies very basic inter-view prediction, while the 3D-HEVC encoder includes several additional tools, allowing to exploit not only temporal redundancy but also inter-view redundancy. In addition, 3D-HEVC has specific tools that allow efficient encoding of depth maps. Compared to conventional video codecs that process 2D video sequences, such as the HEVC standard, both codecs, especially 3D-HEVC, effectively reduce the bit rate when encoding multiple views with associated depth.

[0010] In a virtual reality context, after decoding the view captured by a camera and encoded into a data stream, the virtual view can be synthesized, for example, following the movements of the user.

[0011] For example, a VSRS tool for synthesizing such virtual views is known (Wegner, Stankiewicz, Tanimoto, Domanski, Enhanced view synthesis reference software (VSRS)) for free-viewpoint television, ISO / IEC JTC1 / SC29 / WG11m31520, October 2013, Geneva, Switzerland).

[0012] Figure 1 The diagram illustrates a conventional free navigation system, in which a decoder DEC (e.g., 3D-HEVC) decodes a data stream STR to generate decoded views (VD1, VD2). Such views are then used by a view synthesizer SYNTH (e.g., VSRS) to produce a synthesized view VS(1+2). A restoration device DISP then displays the decoded views and the synthesized views according to the user's movements.

[0013] Figure 2 A conventional decoder DEC is illustrated in FIG. Typically, such a decoder performs an analysis (E20) of a data stream STR to obtain the relevant data to be decoded and then applies a decoding process (E21) to reconstruct decoded views (VD1, VD2) which can then be used by a synthesis module SYNTH to generate a virtual view.

[0014] From this point of view, the process for decoding views from the data stream and the process for synthesizing virtual views are unrelated. In particular, the synthesis process is a difficult task that does not involve the decoder. The decoder only produces decoded views reconstructed from the data stream that are available to the synthesis module.

[0015] The technical problem faced by VR applications is that the encoder and decoder do not know in advance the final view required by the user, especially in the case of free navigation. Multi-view video encoders and decoders do not even know the synthesis process that will eventually be used to synthesize the virtual view. In fact, unlike multi-view video decoders, the synthesis methods used to synthesize virtual views are currently not standardized, making the synthesis methods used by VR applications still proprietary tools.

[0016] Therefore, the quality of the synthesized virtual view depends on the synthesis tools and synthesis algorithms used by such applications. In general, this quality depends on the complexity of the synthesis tools used and the resources of the device implementing these synthesis tools.

[0017] Virtual reality applications, and especially those implementing free navigation, must be real-time. Virtual view synthesis modules typically provide virtual views of mediocre quality, especially when the number of captured and decoded views is insufficient, and even when the decoded and reconstructed captured views are of high visual quality. Summary of the invention

[0018] The present invention improves the prior art.

[0019] The present invention relates to a method for decoding a data stream representing a multi-view video implemented by a decoding device, comprising: obtaining syntax elements from at least a portion of the stream data, and reconstructing at least one image of a view of the video from the obtained syntax elements. Advantageously, the decoding method also comprises: obtaining at least one metadata of a predetermined form from at least one syntax element, and providing the at least one metadata to an image processing module.

[0020] This decoding method therefore allows providing metadata representing the data of the video stream and usable by an image processing module, for example an image processing module external to the decoder. The processing implemented within the image processing module is therefore less complex. For example, in the case of a virtual view synthesis module, some data used by the synthesis algorithm and available from the decoder do not have to be recalculated. In addition, the present invention also enables the image processing module to access data that it cannot calculate itself and to use the data to improve its operation. For example, in the case of a virtual view synthesis module, the decoder can provide the synthesis module with an occlusion map that is difficult to determine by the synthesis module only from the reconstructed image of the view of the video.

[0021] The processing implemented within the image processing module may thus be improved, since the complexity of the operations for obtaining data available at the decoder is reduced, so that more complex and thus more powerful image processing algorithms may be more easily implemented within the image processing module.

[0022] In the case of the virtual view synthesis module, the quality of the virtual view is thus improved. This also allows improving the user's free navigation in the multi-view video by providing smoother transitions between views. Furthermore, the improvement in virtual view synthesis also allows reducing the number of cameras required to capture the scene.

[0023] Providing metadata in a predetermined format allows facilitating communication between the decoder and the image processing module. For example, metadata is provided in the form of an indexed and standardized table. The image processing module thus knows which metadata is stored at each index in the table.

[0024] It is known to use metadata for video data communication. For example, the SEI (Supplemental Enhancement Information) message introduced with the H.264 / AVC standard is data related to optional processing operations implemented at the decoder. SEI messages are sent to the decoder via the video data bitstream. However, this SEI message data is created at the encoder and is only used by the decoder in an optional manner to improve the quality of the decoded and reconstructed view.

[0025] According to a particular embodiment of the invention, obtaining at least one metadata further comprises calculating the at least one metadata from at least a portion of the syntax element.

[0026] This particular embodiment of the invention makes it possible to calculate new metadata corresponding, for example, to information that is not used by the decoder to reconstruct the view, such as confidence values ​​calculated for depth information, or to information used by the decoder in another form, such as motion information with a coarser granularity than that used when reconstructing the image.

[0027] According to a particular embodiment of the invention, said at least one metadata is not used for reconstructing said at least one image.

[0028] According to a particular embodiment of the invention, said at least one metadata corresponds to information included in the group comprising:

[0029] - Camera parameters,

[0030] - decoded and scaled motion vectors,

[0031] - reconstruct the segmentation of the image,

[0032] - a reference image used to reconstruct a block of an image of a view,

[0033] - the coding mode of the image of the reconstructed view,

[0034] - the quantization parameter value of the image of the reconstructed view,

[0035] - the prediction residual values ​​of the image of the reconstructed view,

[0036] - a map representing the motion in the image of the reconstructed view,

[0037] - a map indicating the presence of occlusions in the image of the reconstructed view,

[0038] - A map representing the confidence values ​​associated with the depth map.

[0039] According to another particular embodiment of the invention, said predetermined form corresponds to an index table in which at least one metadata is stored in association with the index.

[0040] According to another particular embodiment of the invention, said at least one metadata is obtained according to a granularity level specified at the decoding device.

[0041] According to this particular embodiment of the invention, metadata generated from syntax elements may be obtained at different levels of granularity. For example, for motion information, motion vectors may be provided at the granularity used at the decoder (i.e., as used by the decoder), or at a coarser granularity (e.g., by providing one motion vector per block of size 64×64).

[0042] According to another specific embodiment of the invention, the decoding method further comprises the decoding device receiving a request from the image processing module indicating at least one metadata required by the image processing module. According to this specific embodiment of the invention, the image processing module indicates the information it needs to the decoder. The decoder can thus only generate the required metadata available to the image processing module, which allows limiting the complexity at the decoder and the use of storage resources.

[0043] According to another particular embodiment of the invention, said request comprises at least one index indicating desired metadata in a predetermined list of available metadata.

[0044] The present invention also relates to a decoding device configured to implement the decoding method according to any one of the above-mentioned specific embodiments. Of course, the decoding device may include different characteristics related to the decoding method according to the present invention. Therefore, the characteristics and advantages of such a decoding device are the same as those of the decoding method, and the characteristics and advantages will not be further detailed.

[0045] According to a particular embodiment of the invention, such a decoding device is comprised in a terminal or a server.

[0046] The present invention also relates to an image processing method, comprising generating at least one image of a virtual view from at least one image of a view decoded by a decoding device. According to the present invention, the image processing method further comprises reading at least one metadata in a predetermined form, the at least one metadata being obtained by the decoding device from at least one syntax element, the at least one syntax element being obtained from a data stream representing a multi-view video, and generating the at least one image by using the read at least one metadata.

[0047] Thus, the image processing method generates images of virtual views of a multi-view video using metadata available to a decoder.Such metadata may correspond to data that the image processing device cannot access, or to data that it can recalculate, but at the expense of high operational complexity.

[0048] The virtual view here refers to the view of a new viewpoint of the scene, for which the camera of the scene acquisition system does not capture any image sequence.

[0049] According to a particular embodiment of the invention, the image processing method further comprises sending a request to the decoding device indicating at least one metadata required to generate the image.

[0050] The present invention also relates to an image processing device configured to implement the image processing method according to any one of the above-mentioned specific embodiments. Of course, the image processing device may include different characteristics related to the image processing method according to the present invention. Therefore, the characteristics and advantages of the image processing device are the same as those of the image processing method, and the characteristics and advantages will not be further detailed.

[0051] According to a particular embodiment of the invention, such an image processing device is comprised in a terminal or a server.

[0052] The present invention also relates to an image processing system for displaying a multi-view video from a data stream representing the multi-view video, comprising a decoding device according to any one of the above embodiments, and an image processing device according to any one of the above embodiments.

[0053] According to the decoding method of the present invention, the corresponding image processing method can be implemented in various ways, in particular in the form of a cable or software. According to a specific embodiment of the present invention, the decoding method, correspondingly the image processing method is implemented by a computer program. The present invention also relates to a computer program, which includes instructions for implementing the decoding method or image processing method according to any one of the aforementioned specific embodiments when the program is executed by a processor. Such a program can use any programming language. The program can be downloaded from a communication network and / or recorded on a computer readable medium.

[0054] Such a program may use any programming language, and may be in the form of source code, object code, or an intermediate code between source code and object code, such as a partially compiled form, or in any other desired form.

[0055] The present invention also relates to a computer-readable recording medium or information medium comprising instructions of a computer program as described above. The above-mentioned recording medium can be any entity or device capable of storing the program. For example, the medium can include a storage device, such as a ROM, such as a CD-ROM or a microelectronic circuit ROM, a USB flash drive, or a magnetic recording medium, such as a hard disk drive or the like. On the other hand, the recording medium can correspond to a transmittable medium such as an electrical signal or an optical signal that can be carried via an electric cable or an optical cable, by radio or by other means. The program according to the present invention can be downloaded in particular on an Internet-type network.

[0056] Alternatively, the recording medium may correspond to an integrated circuit in which the program is embedded, the circuit being suitable for executing the method in question or for use in the execution of the method in question. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Other characteristics and advantages of the invention will appear more clearly on reading the following description of specific embodiments given as simple non-limiting illustrative examples and the accompanying drawings, in which:

[0058] - Figure 1 A schematic diagram illustrates a system for free navigation within a multi-view video according to the prior art,

[0059] - Figure 2 schematically illustrates a decoder for a data stream representing a multi-view video according to the prior art,

[0060] - Figure 3 A schematic diagram illustrates a system for free navigation within a multi-view video according to a particular embodiment of the present invention,

[0061] - Figure 4 The diagram illustrates the steps of a method for decoding a data stream representing a multi-view video according to a particular embodiment of the present invention,

[0062] - Figure 5 Schematic diagram illustrating a decoder of a data stream representing multi-view video according to a particular embodiment of the invention,

[0063] - Figure 6 The steps of the image processing method according to a specific embodiment of the present invention are illustrated.

[0064] - Figure 7The diagram illustrates the steps of a decoding method and an image processing method according to another specific embodiment of the present invention,

[0065] - Figure 8 A schematic diagram illustrates a device suitable for implementing a decoding method according to a particular embodiment of the invention,

[0066] - Fig. 9 schematically illustrates an apparatus suitable for implementing an image processing method according to a particular embodiment of the invention,

[0067] - Fig.10 The diagram illustrates an arrangement of views of a multi-view capture system. DETAILED DESCRIPTION

[0068] The invention is based on modifying the decoding process of a data stream representing a multi-view video so as to facilitate an image processing process based on the views reconstructed by the decoding process. For example, the image processing process corresponds to a process for synthesizing a virtual view. To this end, the decoder not only provides an image of the view reconstructed from the data stream, but also metadata associated with such an image, which metadata can then be used for the synthesis of the virtual view. Advantageously, this metadata is formatted, i.e., is placed in a predetermined form to facilitate interoperability between a decoder and a synthesizer. Therefore, any synthesizer configured to read metadata in a predetermined form can be used to synthesize a virtual view.

[0069] Figure 3 The schematic diagram illustrates a system for free navigation within a multi-view video according to certain embodiments of the present invention. Figure 3 System and About Figure 1 The system described operates similarly, with the difference that in addition to the images reconstructed VD1 and VD2, the decoder DEC also provides metadata MD1, MD2 at the output. Such metadata MD1, MD2 are provided at the input to a synthesizer which then generates, for example, a virtual view VS(1+2) from the reconstructed views VD1, VD2. The decoder DEC and the synthesizer SYNTH then form an image processing system according to the invention. They can be comprised in the same device or in two separate devices which can communicate with each other.

[0070] For example, in a non-limiting and non-exhaustive manner, such metadata may correspond to:

[0071] - the camera parameters of the view reconstructed by the decoder,

[0072] - the decoded and scaled motion vectors of the image reconstructed by the decoder,

[0073] - reconstruct the segmentation of the image,

[0074] - an indication of a reference image used to reconstruct a block of the image,

[0075] - the coding mode of the reconstructed image,

[0076] - quantization parameter value of the reconstructed image,

[0077] - Prediction residual values ​​of the reconstructed image.

[0078] This information may be provided as used by the decoder. Alternatively, this information may be processed by the decoder, for example to be provided at a finer or coarser granularity than that used by the decoder.

[0079] Metadata can also be calculated and shared by decoders, for example:

[0080] - a map representing the general motion in the image of the reconstructed view or in the set of images; such a map may be, for example, a binary map obtained by thresholding the motion vectors of said image or of said set of images,

[0081] - a map indicating the presence of occlusions in the image of the reconstructed view; for example, such a map may be a binary map obtained by taking into account the level of information contained in the prediction residual for each pixel in the case of inter-view prediction, or the information on the possible location of the occlusions may be derived from the disparity vectors of the image or from an edge map,

[0082] - A map representing confidence values ​​associated with a depth map; for example, a decoder may compute such a map by comparing the coding modes of the texture and the corresponding depths.

[0083] Some of the metadata provided by the output may be data related to a single view: the metadata is therefore intrinsic to that view. Other metadata may be obtained from two or more views. In this case, the metadata represents differences or dependencies between the views (camera parameters, occlusion maps, differences in decoding modes, etc.).

[0084] Figure 4 The diagram illustrates the steps of a method of decoding a data stream representing a multi-view video according to a particular embodiment of the invention.

[0085] At the input of the decoder DEC, a data stream STR is provided, for example in the form of a bitstream. The data stream STR for example comprises data of a multi-view video encoded by a video encoder suitable for encoding the multi-view video with redundancy between views according to the prior art, or by a single-view video encoder applied separately to each view of the multi-view video.

[0086] In step E20, the decoder DEC decodes at least a portion of the stream data, thereby providing decoded syntax elements. Such decoding E20 corresponds, for example, to the path of the data stream, and to entropy decoding of the bitstream, in order to extract syntax elements necessary for reconstructing the current image of the view to be reconstructed (for example the view viewed by the user). Such syntax elements correspond, for example, to the coding mode of the blocks of the current image, to the motion vectors in the case of inter-image or inter-view prediction, to the quantization coefficients of the prediction residual, etc.

[0087] Conventionally, during step E21, the current image of the view to be reconstructed (VD1, VD2) is reconstructed from the decoded syntax elements and possibly from images of the view to be reconstructed (VD1, VD2) or other previously reconstructed views. This reconstruction of the current image is carried out according to the coding mode and the prediction technique used at the encoder to encode the current image.

[0088] At the input of the image processing module SYNTH the image of the reconstructed view is provided.

[0089] In step E23, at least one metadata is obtained from at least one decoded syntax element. Such metadata is formatted in a predetermined form. This predetermined form corresponds, for example, to a specific syntax according to which the data is arranged to be sent or stored in a memory. When the multi-view video decoder is a decoder compliant with a specific standard, the syntax of the metadata may, for example, be described in the specific standard or a standard associated with the specific decoding standard.

[0090] According to a particular embodiment of the invention, the predetermined form corresponds to an index table in which at least one metadata is stored in association with an index. According to this particular embodiment, each metadata type is associated with an index. An example of such a table is illustrated in Table 1 below.

[0091] index Metadata Type 0 Camera parameters 1 Motion Vector 2 Image Segmentation 3 The reference image of the block of the image 4 Coding mode for blocks of the image 5 Quantization parameter of the image block 6 The prediction residual value of the image block 7 Motion diagram of image 8 Global motion map of a set of images 9 Occlusion Map 10 Depth Map 11 Confidence map of depth map

[0092] Table 1: Example of metadata table

[0093] Each metadata item is stored in association with its index in an appropriate format according to the metadata type.

[0094] For example, the camera parameters of a view are stored in the form of data triplets, which respectively include, for example, position information corresponding to the coordinates of a point in a 3D coordinate system (the point in the 3D coordinate system corresponds to the position of the camera in the scene), orientation information, for example defined by the values ​​of 3 angles in the 3D coordinate system, and depth of field.

[0095] According to another example, the motion vectors are stored in the form of a table including the value of the corresponding motion vector for each block of the corresponding reconstructed image.

[0096] The metadata table illustrated below is only a non-limiting example. Metadata can be stored in other predetermined forms. For example, when only one metadata type is possible, it is not necessary to associate an index with that metadata type.

[0097] According to a particular embodiment of the invention, at step E22 , before obtaining step E23 , at least one metadata is calculated from at least a portion of the decoded syntax elements.

[0098] This particular embodiment of the invention thus makes it possible to obtain metadata of a current image that is not used for reconstructing a view to be reconstructed, but that can be used for synthesizing a virtual view, such as an occlusion map, from the reconstructed current image.

[0099] This particular embodiment of the invention also makes it possible to obtain metadata with a different granularity than the granularity used to reconstruct the current image. For example, for a block of size 64×64 pixels in the entire image, the motion vector can be calculated in a coarser way from the reconstructed motion vectors of all sub-blocks of the current image contained in the 64×64 block. For example, for each 64×64 block, the motion vector is calculated by taking the minimum or maximum value, the average or median value, or any other function of the motion vectors of the sub-blocks.

[0100] In step E24, the metadata MD1, MD2 obtained in step E23 are provided to an image processing module SYNTH external to the decoder DEC, such as a virtual view synthesis module. A module external to the decoder means a module whose functionality is neither necessary for decoding the data stream nor for displaying the views reconstructed by the decoder.

[0101] For example, the metadata is stored in a memory accessible to the image processing module. According to another example, when the decoder and the image processing module are integrated in the same device, the metadata is sent to the image processing module via a connection link such as a data transmission bus, or when the decoder and the image processing module are integrated in separate devices, the metadata is sent to the image processing module via a cable or wireless connection.

[0102] Figure 5 The schematic diagram illustrates a decoder of a data stream representing multi-view video according to a particular embodiment of the invention.

[0103] Conventionally, the decoding of a view to be reconstructed from a data stream STR is implemented as follows. The decoding of the view to be reconstructed is performed image by image and, for each image, block by block. For each block to be reconstructed, the elements corresponding to the block are decoded from the data stream STR by an entropy decoding module D, thereby providing a set of decoded syntax elements SE (texture coding mode, motion vectors, disparity vectors, depth coding mode, reference image indexes ...) and quantization coefficients coeff.

[0104] The quantized coefficients coeff are sent to the inverse quantization module (Q -1 ), and then sent to the inverse transform module (T -1 ) to provide the prediction residual value res of the block rec The decoded syntax elements (SE) are sent to the prediction module (P) in order to also use the previously reconstructed image I ref (a portion of the current image, or a previously reconstructed reference image of this view, or a reference image of another previously reconstructed view) to calculate the predictor block pred. Then the prediction pred and the prediction residual res of the block are combined. rec Add to reconstruct the current block (B rec ). Then the reconstructed block (B rec ) is stored in the memory MEM for later use in reconstructing the current image or another image or another view.

[0105] According to the invention, at the output of the entropy decoding module, the decoded syntax elements SE, and optionally the quantization coefficients of the blocks, are transmitted to a module FORM configured to select at least a part of the decoded syntax elements SE, and optionally the quantization coefficients, and store them in a predetermined form in order to provide metadata MD associated with a reconstructed image or a group of images.

[0106] The selection of the decoded syntax elements SE to be formatted may be fixed, for example explicitly stated in a standard describing the operation of the decoder. Alternatively, different types of selection may be fixedly defined, for example via a decoder profile, and the parameterization of the decoder allows configuring it so that the formatting module FORM selects the corresponding syntax elements. According to yet another variant, the decoder is able to communicate with the image processing module to which it provides metadata. In this case, the image processing module explicitly indicates to the decoder the type of metadata it wants to receive, and the module FORM of the decoder selects only the required decoded syntax elements.

[0107] When metadata may be provided at a different level of granularity than that used by the decoder, such a level of granularity may be fixedly defined in a standard describing the operation of the decoder, or via a decoder profile. When the image processing module communicates with the decoder to obtain metadata, the image processing module may explicitly indicate to the decoder the level of granularity at which it wants to receive some metadata.

[0108] According to a particular embodiment of the invention, the decoded syntax elements SE and optionally the quantization coefficients at the output of the entropy decoding module are sent to a module CALC configured to calculate metadata from the syntax elements SE and / or the quantization coefficients. As previously mentioned, the metadata to be calculated may be defined explicitly in a standard describing the operation of the decoder, according to different profiles or not, or determined from a communication with an image processing module for which said metadata is intended.

[0109] According to a particular embodiment of the invention, the module FORM selects, in particular, the camera parameters of the view to be reconstructed.

[0110] To synthesize a new viewpoint, the synthesis module must create a model that describes how each pixel of the original (reconstructed) view is projected into the virtual view. Most synthesizers, such as those based on DIBR (depth image based rendering) techniques, use depth information to project the pixels of the reconstructed view into 3D space. Then, the corresponding points in 3D space are projected from the new viewpoint into the camera plane.

[0111] This projection of the image point in 3D space can be modeled by the following equation: M=K.RT.M', where M is the coordinate matrix of the point in 3D space, K is the matrix of the intrinsic parameters of the virtual camera, RT is the matrix of the non-intrinsic parameters of the virtual camera (camera position and orientation in 3D space), and M' is the pixel matrix of the current image.

[0112] If the camera parameters are not sent to the synthesis module, then the synthesis module must calculate them at the cost of high complexity and accuracy, and the calculation cannot be done in real time, or the camera parameters must be obtained through external sensors. Providing these parameters by the decoder makes it possible to limit the complexity of the synthesis module.

[0113] According to another particular embodiment of the invention, the module FORM selects, in particular, syntax elements related to a reference image used for reconstructing a current image.

[0114] In case the synthesis module has the possibility to select a reference image among different available and previously reconstructed images of the view for generating the virtual view, it may be beneficial for the synthesis module to know which reference views were used when encoding the view for synthesis. Fig.10 The diagram illustrates the arrangement of views of a multi-view capture system comprising 16 cameras. The arrows between each view indicate the decoding order of the views. If the synthesis module is to be a viewpoint located between view V6 and view V10 (in Fig.10 If a virtual view VV is generated by a composite module (denoted by a cross in FIG), then conventionally, the composite module must check the availability of each view to optimally compose the virtual view.

[0115] According to the specific embodiments described herein, if the composition module has metadata for a view indicating the reference view used to reconstruct the view, the composition module may simply select the available view that is closest to the virtual viewpoint (in Fig.10 In this embodiment, the composition module may determine which images to use to generate the virtual view. For example, if a block of view V6 uses an image of view V7 as a reference image, the composition module may also decide to use view V7, which must be available because it is used by view V6. Such an embodiment thus reduces the complexity of the composition module by avoiding having to check the availability of each view during composition.

[0116] According to another particular embodiment of the invention, the module CALC selects, in particular, syntax elements associated with motion vectors to generate a motion map.

[0117] In areas with little motion, virtual view synthesis often suffers from a lack of temporal coherence due to inaccuracies in the depth map. These incoherences are extremely disturbing for visualization from a virtual viewpoint.

[0118] In this particular embodiment, the decoder's module CALC selects the decoded and reconstructed motion vectors, i.e. after inverse prediction of the motion vectors and scaling of the motion vectors. The module CALC thresholds the reconstructed motion vectors of each block to generate a motion map, usually a binary map, in which each element takes the value 0 or 1, indicating locally whether there is motion in that area. The binary map can be improved, for example, by using mathematical morphology (e.g. erosion, expansion, opening, closing).

[0119] The motion binary map may then be formatted to indicate whether motion is present in the view, depending on, among other things, the desired granularity (pixel level map, block or sub-block level map, or a map defined for a specific block size in an image, etc.).

[0120] A synthesis module receiving such a motion map can then adapt its operation, for example by applying different synthesis processing depending on whether the area is marked as having motion. For example, to address the problem of temporal incoherence, the conventional synthesis processing can be disabled in fixed (static) areas, and the values ​​of the pixels of the previous image are simply inherited.

[0121] The synthesis module can of course use other means, such as generating the motion map itself by estimating the motion as the encoder does. However, this operation has a significant impact on the complexity of the synthesis algorithm, and on the accuracy of the resulting motion, since the encoder estimates the motion from uncoded images that are no longer available at the output of the decoder.

[0122] exist Fig.10In the example illustrated in FIG and the embodiments described above, not only the closest available view can be used, but also the effective reference view can be calculated by averaging the reference views of the neighborhood of the virtual viewpoint. For example, the reference views V6, V7, V10 and V11 can be averaged by the module CALC of the decoder and the resulting average view is sent to the synthesis module.

[0123] According to another variant, the module CALC of the decoder can calculate an occlusion map in which it indicates, for each pixel or block of the image, whether the area corresponds to an occluded area. For example, the module CALC can determine whether the area corresponds to an occluded area by using information of the reference image(s) used by the decoder to reconstruct the area. For example, in Fig.10 In the case of, for example, if most blocks in the image of view V6 use temporal prediction relative to view V2, while some blocks in the image of view V6 use inter-view prediction, then these blocks may correspond to occluded areas.

[0124] A composition module receiving such an occlusion map may then decide to apply different composition processing depending on whether the region is marked as an occluded region or not.

[0125] According to another particular embodiment of the invention, the module CALC selects, in particular, a coding mode associated respectively with the texture of the reconstructed image and with the depth map of the image.

[0126] According to the prior art, synthesis algorithms mainly use depth maps. Such depth maps usually present errors that produce artifacts in the synthesized virtual view. By comparing the coding mode between the texture and the depth map, the decoder can derive a confidence measure associated with the depth map, such as a binary map indicating whether the depth is associated with the texture (value 1) or not (value 0).

[0127] For example, the confidence value can be derived from the coding mode. If the texture coding mode and the depth coding mode are different, for example one is intra mode and the other is inter mode, then this means that the texture and depth are not related. Therefore, the confidence value will be low, for example 0.

[0128] The confidence value can also be positioned according to the motion vector. If the texture and depth have different motion vectors, then this means that the texture and depth are not related. Therefore, the confidence value will be low, such as 0.

[0129] The confidence value can also be located according to the reference image used by the texture and depth. If the reference image is different, it means that the texture and depth are not related. Then, the confidence value will be low, such as 0.

[0130] A synthesis module receiving such a confidence map can then decide to apply different synthesis processing depending on whether the region is marked with a low confidence value. For example, for such a region, another reference view that provides a better confidence value for the region can be used to synthesize the corresponding region.

[0131] Figure 6 The steps of the image processing method according to a specific embodiment of the present invention are illustrated. This processing is performed, for example, by a virtual view synthesis module, from, for example, Figure 5 The described decoding method is implemented by decoding and reconstructing the views.

[0132] In step E60, the synthesis module reads at least one metadata (MD1, MD2). The metadata read by the synthesis module corresponds to syntax elements decoded from a stream representing a multi-view video and is associated with one or more views. The metadata may also correspond to information calculated during the method of decoding the stream from the decoded syntax elements. The metadata is stored or sent to the synthesis module in a predetermined form so that any synthesis module with an appropriate reading module can read it.

[0133] In step E61, the synthesis module receives at the input a video signal received by a multi-view video decoder, for example according to Figure 5 The described decoding method reconstructs at least one image of a view (VD1, VD2). The synthesis module generates at least one image of a virtual viewpoint VS (1+2) using these received views VD1, VD2 and the read metadata MD1, MD2. In particular, the synthesis module uses the metadata MD1, MD2 to determine a synthesis algorithm to be used for certain areas of the image, or to determine the view to be used for generating the image of the virtual view ....

[0134] Figure 7 The diagram illustrates the steps of a decoding method and an image processing method according to another specific embodiment of the present invention.

[0135] Typically, a decoder of a multi-view video does not know the type of synthesizer that will be used to generate the virtual viewpoints. In other words, the decoder does not know which synthesis algorithm will be used, nor what metadata types may be useful to it.

[0136] According to the specific embodiments described herein, the decoder and the synthesis module are adapted to be able to communicate in a bidirectional manner. For example, the synthesis module can indicate to the decoder a list of metadata that it will need in order to achieve better synthesis. Before or after a request from the synthesis module, the decoder can inform the synthesis module of the metadata that it can send to the synthesis module. Advantageously, the list of metadata that the decoder can share is standardized, that is, all decoders that meet the decoding standard must be able to share the metadata in the list. Therefore, for a given decoding standard, the synthesis module knows which metadata may be available. The metadata list can also be adapted according to the profile of the decoder standard. For example, for a profile predetermined for a decoder requiring low operational complexity, the list of metadata only includes the syntax elements of the decoding of the stream, while for a profile predetermined for a decoder capable of handling higher operational complexity, the list of metadata can also include metadata obtained by calculating from the syntax elements of the decoding of the stream, such as motion maps, occlusion maps, trust maps, etc.

[0137] At step E70, the synthesis module sends a request to the decoder indicating at least one metadata required to generate the image of the virtual viewpoint. For example, the request includes an index or a list of indices corresponding to the required metadata respectively.

[0138] Such a request is sent in a predetermined format, that is, in a predetermined syntax, so that the synthesis module and the decoder can understand each other. For example, such a syntax can be:

[0139] nb

[0140] for i (integer from 0 to nb-1), list[i]

[0141] Wherein the syntax element nb indicates the number of metadata required by the synthesis module and therefore the number of indices to be read by the decoder, and list[i] indicates the corresponding indices of the required metadata.

[0142] According to an example, taking the metadata given in Table 1 above as an example, the synthesis module may indicate in the request that nb=2, and indexes 0 and 9 corresponding to the camera parameters and the occlusion map, respectively.

[0143] According to a variant, the synthesis module may also indicate the level of granularity in association with the index of the desired metadata, for example by specifying a predetermined value of a "grlevel" syntax element associated with the metadata. For example, in the case of an occlusion map, the synthesis module may indicate a value of 1 for the "level" element associated with index 9 if it wants an occlusion map at the pixel level, or a value of 2 for the "level" element associated with index 9 if it wants a coarser granularity, for example an occlusion map of block size 8×8.

[0144] In step E71, the decoder obtains the corresponding metadata. To this end, and in accordance with the above Figure 4 In the example described in 5, the decoder retrieves the decoded syntax elements required to obtain the metadata and calculates metadata that is not used by the decoder for reconstruction, such as occlusion maps. The metadata is then formatted in a predetermined form so that the synthesis module can read it.

[0145] In step E72 , the decoder sends the metadata to the synthesis module, which can then use it in its synthesis algorithm.

[0146] Figure 8 The schematic diagram illustrates a device DEC suitable for implementing the decoding method according to the aforementioned specific embodiment of the invention.

[0147] Such a decoding device comprises a memory MEM, a processing unit UT provided with, for example, a processor PROC and controlled by a computer program PG stored in MEM. The computer program PG comprises instructions for implementing the steps of the decoding method as previously described when the program is executed by the processor PROC.

[0148] According to a particular embodiment of the invention, the decoding device DEC comprises a communication interface COM0 which notably enables the decoding device to receive, via a communication network, a data stream representative of a multi-view video.

[0149] According to another particular embodiment of the invention, the decoding device DEC comprises a communication interface COM1 enabling the decoding device to send metadata to an image processing device, such as a composition module, and to send images of views reconstructed from a data stream.

[0150] During initialization, the code instructions of the computer program PG are loaded into a memory, for example, before being executed by the processor PROC. In particular, the processor PROC of the processing unit UT implements the instructions of the computer program PG in accordance with the instructions of the computer program PG. Figure 4 , 5 and 7. The memory MEM is particularly suitable for storing metadata obtained during the decoding method in a predetermined form.

[0151] According to a particular embodiment of the invention, the aforementioned decoding device is included in a terminal such as a television receiver, a mobile phone (eg a smartphone), a set-top box, a virtual reality helmet or the like.

[0152] Fig. 9 The schematic diagram illustrates a device SYNTH suitable for implementing the image processing method according to the aforementioned specific embodiment of the invention.

[0153] Such a device comprises a memory MEM9, a processing unit UT9 provided with, for example, a processor PROC9 and controlled by a computer program PG9 stored in the memory MEM9. The computer program PG9 comprises instructions for implementing the steps of the image processing method as described above, when said program is executed by the processor PROC9.

[0154] According to a particular embodiment of the invention, the device SYNTH comprises a communication interface COM9 enabling the device to receive metadata originating from a decoding device, such as the above-mentioned device DEC, and images of views reconstructed by the device DEC from a data stream representative of a multi-view video.

[0155] At initialization, the code instructions of the computer program PG9 are loaded into a memory, for example, before being executed by the processor PROC9. In particular, the processor PROC9 of the processing unit UT9 implements the instructions of the computer program PG9 in accordance with the instructions of the computer program PG9. Figure 6 and Figure 7 The steps of the image processing method are described.

[0156] According to a particular embodiment of the invention, the device SYNTH comprises an output interface AFF9 enabling the device SYNTH to send an image to a display device, such as a screen. For example, such an image may correspond to an image of a virtual viewpoint, generated by the device SYNTH with the help of an image of the reconstructed view and metadata received from the device DEC.

[0157] According to a particular embodiment of the invention, the device SYNTH is a synthesis module. It is comprised in a terminal such as a television receiver, a mobile phone (e.g. a smartphone), a set-top box, a virtual reality helmet or the like.

[0158] The principle of the invention is illustrated in the case of a multi-view decoding system, in which multiple views are decoded from the same stream (bitstream) and metadata is obtained for each view. The principle applies similarly to the case of encoding multi-view video with the aid of multiple streams (bitstreams), where one view per stream is encoded. In this case, each view decoder provides metadata associated with the view it decodes.

Claims

1. A method for decoding a data stream representing a multi-view video implemented by a decoding device, comprising: - obtaining syntax elements from at least a portion of the data stream (E20), and - reconstructing at least one image of a view of said video from the obtained syntax elements (E21), The decoding method is characterized in that it also includes: - obtaining at least one metadata (E23) from at least one syntax element, the at least one metadata being associated with at least one reconstructed image and corresponding to a camera parameter representative of a position of at least one camera in the scene, - providing the at least one metadata to an image synthesis device (E24) to synthesize at least one virtual view from the reconstructed at least one image and the at least one metadata, the at least one virtual view being different from the views of the multi-view video.

2. The decoding method according to claim 1, wherein obtaining at least one metadata further comprises calculating the at least one metadata from at least a portion of the syntax element.

3. Decoding method according to claim 1 or 2, wherein the at least one metadata is not used to reconstruct the at least one image.

4. The decoding method according to claim 1 or 2, wherein the at least one metadata is obtained in a predetermined form corresponding to an index table in which the at least one metadata is stored in association with an index.

5. The decoding method according to claim 1 or 2, wherein the at least one metadata is obtained according to a granularity level specified at the decoding device.

6. The decoding method according to claim 1 or 2 further comprises receiving, by the decoding device, a request from the image synthesis device, wherein the request indicates at least one metadata required by the image synthesis device.

7. The decoding method according to claim 6, wherein the request comprises at least one index indicating the desired metadata in a predetermined list of available metadata.

8. A decoding device for decoding a data stream representing a multi-view video, the decoding device comprising a processor, the processor being configured to: - obtaining syntax elements from at least a portion of the data stream, and - reconstructing at least one image of a view of said video from the obtained syntax elements, The decoding device is characterized in that it is also configured to: - obtaining at least one metadata from at least one syntax element, the at least one metadata being associated with at least one reconstructed image and corresponding to a camera parameter representing a position of at least one camera in the scene, - providing the at least one metadata to an image synthesis device to synthesize at least one virtual view from the reconstructed at least one image and the at least one metadata, the at least one virtual view being different from the views of the multi-view video.

9. An image synthesis method for an image synthesis device, comprising generating at least one image of a virtual view from at least one image of a view of a multi-view video decoded by a decoding device, the virtual view being different from the view of the multi-view video, the image synthesis method being characterized by comprising: - reading at least one metadata (E60), the at least one metadata being obtained by the decoding device from at least one syntax element and corresponding to a camera parameter representing a position of at least one camera in a scene, the at least one syntax element being obtained from a data stream representing a multi-view video, - generating said at least one image (E61) comprising using said at least one metadata read.

10. The image synthesis method according to claim 9, further comprising sending a request to a decoding device, the request indicating at least one metadata required to generate the image.

11. An image synthesis device configured to generate at least one image of a virtual view from at least one image of a view of a multi-view video decoded by a decoding device, the virtual view being different from a view of the multi-view video, the image synthesis device comprising a processor, characterized in that: - the processor is configured to read at least one metadata, the at least one metadata being obtained by the decoding device from at least one syntax element and corresponding to a camera parameter representing a position of at least one camera in a scene, the at least one syntax element being obtained from a data stream representing a multi-view video, and - utilizing said at least one metadata read when generating said at least one image.

12. An image processing system for displaying a multi-view video from a data stream representing the multi-view video, comprising: - a decoding device according to claim 8, and - An image synthesis device according to claim 11.

13. A computer-readable storage medium having instructions stored thereon, wherein when the instructions are executed by a processor, the processor implements the decoding method according to any one of claims 1 to 7, or implements the image synthesis method according to any one of claims 9 to 10.

Images