Method and device for decoding multi-view video, and method, computer program and device for image synthesis

Abstract

To provide method and device for decoding a sequence of multi-view images and synthesizing intermediate view images.SOLUTION: A method for decoding a data stream representing a multi-view video includes (E20) acquiring a syntax element from at least a part of a data stream, (E21) using the syntax element to reconstruct at least one image of a view of a video, then (E23) acquiring at least one metadata in a predetermined format from the at least one acquired syntax element, and (E24) providing the metadata to an image synthesis module. The image synthesis module includes synthesizing at least one virtual view different from the view of the multi-view video from the at least one reconstructed image and at least one item of the metadata.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates generally to the field of 3D image processing, and more particularly to the simulation of multi-view images. This paper deals with decoding the sequence and synthesizing the intermediate view images. [Background technology]

[0002] In the field of virtual reality, free navigation allows the viewer to View the scene from any viewpoint, where that viewpoint corresponds to the viewpoint captured by the camera. whether it corresponds to a viewpoint not captured by the camera or Such views that are not captured by a camera are called virtual views. This view is also called the intermediate view, because it is the view captured by the camera. Because it is between the captured views and the views, it must be composited for restoration. It is.

[0003] Free navigation within a scene is a method to view all the movements of a user viewing a multi-view video. This ensures that the image is correctly managed and that the viewer does not perceive it as being suboptimal. It is necessary that the discomfort experienced be avoided.

[0004] Generally, the user's movements are reproduced by a virtual device such as a head mounted device (HMD). This is properly taken into account by the reality headset.

[0005] However, it is correct for display regardless of the user's movement (rotation or translation). Providing the pixels remains a problem. In practice, the calculation of the image to be displayed is ,Some ,captures ,are ,used ,to ,display ,an ,additional ,image ,of ,the ,virtual ,(i.e., ,synthesized) ,view. Such a virtual view requires the use of a captured view. The decoded captured view and the related It is required to calculate them from the associated depths.

[0006] Therefore, a codec that provides free navigation functionality must be able to use several views and It is useful for encoding the image and associated depth, while finding the optimal resolution of the virtual view. rendering must be possible, i.e. the compositing algorithm must be available for display. and requires.

[0007] Multiview video encoder designed to encode multiview sequences For example, standard MV-HEVC or 3D-HEVC (Series H: Audiovisua l and multimedia systems - Infrastructure of audio visual services - Coding of m oving video, High Efficiency Video Coding, Recommendation ITU-T H.265, Internati onal Telecommunication Union, December 2016) is known.

[0008] The MV-HEVC encoder applies very basic inter-view prediction, while the 3D-H The EVC encoder includes some additional tools to allow for more than just temporal redundancy: It also exploits inter-view redundancy. In addition, 3D-HEVC efficiently encodes depth maps. These two codecs, and especially 3D-HEV C is a 2D standard such as the HEVC standard when encoding multiple views with associated depth. Effectively reduces bitrate compared to traditional video codecs for handling video sequences do.

[0009] In a virtual reality situation, it is captured by a camera and encoded into a data stream. After decoding the received view, a virtual view is synthesized based on, for example, the user's movements. It is possible.

[0010] For example, the VSRS tool (Wegner, St ankiewicz, Tanimoto, Domanski, Enhanced view synthesis reference software (VSRS) for free-viewpoint television, ISO / IEC JTC1 / SC29 / WG11 m31520, October 2013, Gen eva, Switzerland) are known.

[0011] FIG. 1 shows how a decoder DEC (e.g., 3D-HEVC) decodes a data stream STR. The conventional free navigation system generates decoded views (VD1, VD2) by downloading the In this case, such views are represented by a view synthesiser SYNT H (e.g., VSRS) to generate the synthesized view VS(1+2). Then the decoded and synthesized views are restored according to the user's movements. Displayed by the source device DISP.

[0012] A conventional decoder DEC is shown in FIG. 2. Typically, such a decoder Perform analysis of the data stream STR (E20) to obtain the relevant data to be decoded. Then, a decoding process (E21) is applied to generate the virtual view using the synthesis module. Decoded views (VD1, VD2) that can be used later by SYNTH Reconstruct.

[0013] Therefore, the process of decoding the views from the data stream and synthesizing the virtual views is In particular, the synthesis process is a difficult one that does not involve a decoder. The decoder simply extracts the decoded video signal reconstructed from the data stream. The queue is made available to the synthesis module.

[0014] The technical problem facing virtual reality applications is that encoders and decoders, in particular In the case of free navigation, advance knowledge of the final viewpoint required by the user Multi-view video encoders and decoders also have the following features: , and has no knowledge of the compositing process that will ultimately be used to synthesize the virtual view. The synthesis method used to synthesize the virtual view is different from that used in multiview video decoders. It is currently not standardized and therefore is not used by virtual reality applications. The synthetic methods used remain unique tools.

[0015] Therefore, the quality of the synthesized virtual view is limited by the quality of the images used by such applications. In general, such quality depends on the synthesis tools and algorithms used. The complexity of the synthesis tools used and the resources of the equipment implementing these synthesis tools will determine the do.

[0016] Virtual reality applications, and more particularly virtual reality with free navigation The application must be real-time. The virtual view synthesis module: In general, decoding is difficult, especially if the number of views captured and decoded is insufficient. and the reconstructed captured view is of high visual quality, but of medium quality. Provides a virtual view of the Summary of the Invention [Problem to be solved by the invention]

[0017] The present invention improves upon the current state of the art. [Means for solving the problem]

[0018] The present invention relates to a data stream representing a multi-view video, which is implemented by a decoding device. A method for decoding a frame, comprising: extracting a syntax element from at least one portion of a data stream. The method obtains the syntax elements of the video and calculates the number of views of the video from the syntax elements. Advantageously, this relates to a decoding method, which comprises reconstructing at least one image. The decoding method derives at least some metadata in a predefined format from at least one syntax element. At least one item of metadata must be acquired and at least one item of metadata must be transferred to the image processing model. and providing the device to the user.

[0019] Therefore, such a decoding method may be implemented externally to an image processing module, e.g. The synthesis module represents the data of the video stream and uses it for the image processing module. This allows the image processing module to provide metadata that can be used to The processing performed within the virtual view synthesis module is less complex. In this case, the data used by the synthesis algorithm and available to the decoder is Moreover, the present invention provides a method for processing an image by a computer, the image processing module, Access data that it cannot compute on its own and use that data to drive its operations For example, in the case of the virtual view synthesis module, the decoder ,We can provide an occlusion map to the synthesis module, and such an occlusion The motion is determined by the synthesis module only from the reconstructed image of the video view. It is difficult to do so.

[0020] Thus, the processing performed in the image processing module can be improved. Because the computational complexity of obtaining data available at the decoder level is reduced, This reduces the number of pixels required for image processing, and therefore requires more complex and therefore more powerful image processing algorithms. This allows the system to be more easily implemented in the image processing module.

[0021] In the case of the virtual view synthesis module, the quality of the virtual view is thus improved. This improves multi-view video by providing smoother transitions between views. It also improves the user's free navigation in the scene. This also reduces the number of cameras needed to capture a scene.

[0022] By providing metadata in a predefined format, decoders and image processing modules For example, metadata is indexed and It is provided in the form of a standardized table. Image processing modules can therefore For each table, you can see what metadata is stored in this index. do.

[0023] It is known to use metadata in video data communication. The SEI (Supplemental Enhancement Information) message introduced in the C standard The message is data regarding optional processing operations performed at the decoder level. The SEI message is transmitted to the decoder via the video data bitstream. However, such SEI message data is created at the encoder level. and is used only by the decoder, and is optionally decoded and reconstructed. The quality of the views is improved.

[0024] According to a particular embodiment of the invention, obtaining at least one item of metadata extracts at least one part of the syntax element from at least one item of the metadata above The method further includes calculating the eye.

[0025] Such a particular embodiment of the present invention may, for example, be implemented using a decoder to reconstruct the views. Information that is not used by the The information used by the decoder, e.g. the granularity used when reconstructing an image It allows the computation of new metadata that corresponds to coarse-grained motion information. do.

[0026] According to another particular embodiment of the invention, at least one item of said metadata comprises at least are not used to reconstruct at least one image.

[0027] According to another particular embodiment of the invention, at least one item of said metadata comprises: The following: - camera parameters, - the decoded and scaled motion vectors, -Segmentation of the reconstructed image, - a reference image used by a block of the image of the reconstructed view, - coding mode of the image of the reconstructed view, - the quantization parameter value of the image of the reconstructed view, - the predicted residual value of the image of the reconstructed view, - a map representing the motion within the image of the reconstructed view, - a map representing the presence of occlusions in the image of the reconstructed view, - a map representing confidence values ​​associated with the depth map, corresponds to one item of information contained within a group that contains

[0028] According to another particular embodiment of the invention, the predetermined format is at least one of the metadata It corresponds to an indexed table, where items are stored in association with an index.

[0029] According to another particular embodiment of the invention, at least one item of said metadata is The information is obtained based on the level of granularity specified in the code device.

[0030] According to this particular embodiment of the invention, the metadata generated from the syntax elements is For example, motion information can be obtained at different levels of granularity. the granularity used in the coder (i.e., as used by the decoder), or (For example, provide one motion vector per block of size 64x64. It is possible to provide motion vectors with a coarser granularity (by

[0031] According to another particular embodiment of the present invention, the decoding method comprises: This request indicates at least one item of metadata required by the processing module. According to this particular embodiment of the present invention, For example, the image processing module indicates to the decoder the information that the image processing module requires. Thus, the decoder makes only the necessary metadata available to the image processing module. This limits the complexity and memory resource usage in the decoder. It will be what was done.

[0032] According to another particular embodiment of the invention, the request is made based on a predefined list of available metadata. The metadata object includes at least one index pointing to an item of required metadata in the

[0033] The present invention also relates to a decoding method according to any one of the specific embodiments defined above. The present invention relates to a decoding device adapted to carry out the method. It can include different features related to the decoding device according to the invention. The features and advantages of the decoding device are the same as those of the decoding method and will be described further in detail. do not have.

[0034] According to a particular embodiment of the invention, such a decoding device is located in a terminal or a server. Included.

[0035] The present invention relates to a method for decoding an image from at least one image of a view decoded by a decoding device, comprising: The present invention also relates to a method of image synthesis comprising generating at least one image of a virtual view. According to the description, such an image processing method comprises the step of: and reading at least one item of said metadata by a decoding device. At least one symbol obtained from a data stream representing a multi-view video is the at least one image is obtained from the at least one tax element of the metadata; It is generated with one retrieved item.

[0036] Therefore, the image synthesis method utilizes the metadata available to the decoder to Such metadata is used by the image processor to generate an image of a virtual view of the audiovisual. This can correspond to data that the user does not have access to or that can be recalculated. However, the calculations become very complicated.

[0037] The virtual view here refers to the sequence of images captured by the camera of the scene capture system. It means a new perspective view of a scene that has not been captured before.

[0038] According to a particular embodiment of the present invention, the image synthesis method includes: The method further includes sending a request indicating at least one item of metadata required for the

[0039] The present invention also relates to an image processing method according to any one of the specific embodiments defined above. The present invention relates to an image processing device configured to carry out the method. It can include different features relating to the image processing method according to the invention. The characteristics and advantages of the processing device are the same as those of the image processing method and are described in more detail below. do not have.

[0040] According to a particular embodiment of the present invention, such an image processing device is implemented in a terminal or a server. Included.

[0041] The present invention also relates to a method for extracting multi-view video from a data stream representing the multi-view video. An image processing system for displaying a decoding image according to any one of the above embodiments. and an image processing apparatus according to any one of the above embodiments. Regarding the logical system.

[0042] The decoding method, respectively the image processing method according to the invention, can be implemented in various ways, in particular by This can be implemented in linear or software form. The image processing method, the decoding method, and the image processing method are implemented by a computer program. The present invention also relates to a method for implementing the present invention in accordance with any one of the specific embodiments described above when executed by a processor. A computer program including instructions for implementing the decoding method or image processing method according to any one of the methods. Such programs may use any programming language. The program can be downloaded from a communication network and / or read from a computer. It can be recorded on a readable medium.

[0043] This program can be written in any programming language, and can be Object code, intermediate code between source code and object code, e.g. The file may be in a partially compiled form, or in any other desired form.

[0044] The present invention relates to a computer-readable storage medium that includes instructions for the computer program described above. The above-mentioned recording medium can store a program. The medium may be any entity or device. For example, the medium may be a storage means, e.g. For example, a ROM, such as a CD-ROM or a microelectronic circuit ROM, a USB flash drive On the other hand, the recording medium may include a hard disk, a USB stick, or a magnetic recording means, such as a hard drive. , which may be transmitted over electrical or optical cables, wirelessly or by other means. The program according to the present invention can correspond to a transmission medium such as an electrical signal or an optical signal. The RAM can be downloaded, in particular over an Internet-type network.

[0045] Alternatively, the recording medium may correspond to an integrated circuit in which the program is embedded, The circuitry is adapted to perform or be used in the performance of the method.

[0046] Further features and advantages of the present invention will become apparent from the following detailed description, which is given by way of example only and is not intended to be limiting, with reference to the accompanying drawings, in which: This will become clearer on reading the following description of specific embodiments, given as examples. [Brief description of the drawings]

[0047] [Figure 1] FIG. 1 illustrates diagrammatically a system for free navigation in multi-view video according to the prior art; [Diagram 2] FIG. 1 diagrammatically illustrates a decoder of a data stream representing multi-view video according to the prior art; [Diagram 3] FIG. 1 illustrates a schematic diagram of a system for free navigation in multiview video according to certain embodiments of the present invention. [Figure 4]FIG. 2 illustrates steps of a method for decoding a data stream representing a multiview video according to a particular embodiment of the invention. [Diagram 5] FIG. 2 illustrates diagrammatically a decoder of a data stream representing a multi-view video according to a particular embodiment of the invention; [Figure 6] FIG. 2 illustrates steps of an image processing method according to a particular embodiment of the invention. [Figure 7] FIG. 4 illustrates steps of a decoding method and an image processing method according to another particular embodiment of the invention. [Figure 8] FIG. 2 shows diagrammatically an apparatus adapted to implement a decoding method according to a particular embodiment of the invention; [Figure 9] 1 shows diagrammatically an apparatus adapted to implement an image processing method according to a particular embodiment of the invention; [Figure 10] FIG. 1 is a diagram showing the arrangement of views in a multi-view capture system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0048] The present invention modifies the decoding process of a data stream representing a multi-view video. This is based on the view reconstructed by the decoding process. For example, the image processing process is simplified for the process of synthesizing a virtual view. For this purpose, the decoder only retrieves the images of the reconstructed views from the data stream. Instead, it also provides metadata associated with such images, which are then , can be used for the synthesis of virtual views. Advantageously, such metadata is formatted to facilitate interoperability between decoders and synthesizers. Therefore, to synthesize the virtual view, Any synthesizer configured to read metadata in a video file may be used.

[0049] FIG. 3 illustrates a method for free navigation in a multi-view video according to a particular embodiment of the present invention. The system in FIG. 3 is similar to that described in relation to FIG. It works similarly to the system described above, but the decoder DEC outputs the reconstructed The difference is that it provides metadata MD1, MD2 in addition to images of views VD1 and VD2. Such metadata MD1, MD2 are provided at the input to the synthesis device. Next, the synthesizer derives a virtual view VS( 1+2). The decoder DEC and the synthesizer SYNTH then generate the The decoder DEC and the synthesizer SYNTH form an image processing system. It can be contained within one device, or it can be contained within two separate devices that can communicate with each other. can.

[0050] For example, but not limited to and not exhaustive, such metadata may correspond to the following: This can be done. - the camera parameters of the view reconstructed by the decoder, - decoding the image reconstructed by the decoder and the scaled motion vectors; Kutlu, -Segmentation of the reconstructed image, - indication of the reference images used by the blocks of the reconstructed image, - the coding mode of the reconstructed image, the quantization parameter value of the reconstructed image, - Prediction residual values ​​of the reconstructed image.

[0051] Such information can be provided for use by a decoder. Alternatively, such information may be provided by a decoder, e.g., to a decoder using a granularity test. The particles can be processed to provide a finer or coarser granularity than the above mentioned.

[0052] Metadata may also be computed and shared by decoders, for example: do. A map representing the global motion within an image or group of images of the reconstructed view. For example, such a map can be created by thresholding the motion vectors of an image, or a group of images. It may be a binary map obtained by processing. - A map representing the presence of occlusions in the image of the reconstructed view. For example, Such a map represents the level of information contained in the prediction residual for each pixel in the case of inter-view prediction. It can be a binary map obtained by considering the occlusion The information on the possible locations of the motions is derived from the disparity vectors or edge maps of the images. It is possible. A map representing confidence values ​​associated with the depth map. For example, such a map could be: By comparing the coding mode of the texture with the coding mode of the corresponding depth It can be calculated by the decoder.

[0053] Some of the output metadata may be data relating to a single view. In the case of , this output metadata is specific to the view. Other metadata is One or more views can be obtained. In this case, the metadata is Difference or correlation (difference in camera parameters, occlusion map, decode mode) etc.).

[0054] FIG. 4 illustrates a data stream representing a multi-view video according to a particular embodiment of the invention. 4 shows steps of a method for decoding a

[0055] The data stream STR is applied to the input of the decoder DEC, e.g. as a bit stream. The data stream STR is provided in the form of a A conventional video encoder adapted for encoding multi-view video is or single-view video encoding applied separately to each view of the multiview video. It contains multi-view video data encoded by the decoder.

[0056] In step E20, the decoder DEC determines whether the decoded syntax elements are Decode at least a portion of the data stream to be decoded. 20 is, for example, a current view of the view to be reconstructed, e.g. the view viewed by the user. Parsing the data stream to extract the syntax elements needed to reconstruct the image Such a syntax supports decoding the bitstream's entropy. The data element may, for example, be the coding mode of the blocks of the current image, the inter-image prediction or the inter-view prediction. These correspond to motion vectors in the case of prediction, quantization coefficients of prediction residuals, etc.

[0057] Conventionally, during step E21, the current image of the view to be reconstructed (VD1, VD2) is An image consists of the decoded syntax elements and possibly their views or other previous Such a reconstruction of the current image is The coding mode used at the encoder level to encode the image of This is carried out according to predictive techniques.

[0058] The images of the reconstructed views are provided at the input of the image processing module SYNTH. do.

[0059] In step E23, at least one item of metadata is Such items of metadata are obtained from coded syntax elements. Such a predefined format may be, for example, the time at which the data is to be transmitted or The multi-view video signal corresponds to a particular syntax that is configured to be stored in a memory. If the decoder is a compliant decoder, the metadata syntax is: For example, the specific standard or the standard associated with the specific decoding standard may be used. It can be assumed that the

[0060] According to a particular embodiment of the invention, the predetermined format is defined by at least one item of metadata: corresponds to an indexed table in which the According to a particular embodiment, each metadata type is associated with an index. An example of such a table is shown in Table 1 below.

[0061] [Table 1]

[0062] Each item of metadata is associated with its index and is based on the metadata type. It is stored in a suitable format.

[0063] For example, the camera parameters of a view each correspond to, for example, the position of the camera in the scene. The coordinates of the point in the corresponding 3D coordinate system and the three angles in the 3D coordinate system a triplet of data including orientation information defined by the values ​​of It is stored as such.

[0064] According to another example, the motion vectors are given for each block of the corresponding reconstructed image by , are stored in the form of a table containing the corresponding motion vector values.

[0065] The metadata table shown below is only a non-limiting example. The metadata may be any other predefined For example, if only one metadata type is possible, It is not necessary to associate an index with that metadata type.

[0066] According to a particular embodiment of the invention, in step E22, at least An item is at least one of the decoded syntax elements before the acquisition step E23. is also calculated from one part.

[0067] Thus, according to such a particular embodiment of the present invention, the current It is not used to reconstruct the image, but synthesizes a virtual view from the current reconstructed image. It is possible to obtain metadata, e.g. an occlusion map, that can be used to It becomes possible.

[0068] According to such a particular embodiment of the present invention, the grain size used to reconstruct the current image is It is also possible to obtain metadata with a different granularity than the motion For example, for a block of 64x64 pixels across the entire image, the vector is 64x The reconstructed motion vectors of all sub-blocks of the current image contained within the block of 64 are From the motion vector, it can be calculated more coarsely, for example, for each 64x64 block. The motion vector is the minimum or maximum of the motion vectors of the subblocks. The value may be calculated by selecting the mean or median, or any other function.

[0069] In step E24, the metadata MD1, MD2, MD3, MD4, MD5, MD6, MD7, MD8, MD9, MD10, MD11, MD12, MD13, MD14, MD15, MD16, MD17, MD18, MD19 ...9, MD112, MD13, 2 is an image processing module SYNTH external to the decoder DEC, e.g. a virtual view synthesis module The decoder's external modules are responsible for decoding the data stream. This behavior is necessary both for the decoder to display the reconstructed view and for the decoder to display the reconstructed view. This means that the module is not

[0070] For example, the metadata may be stored in a memory accessible to the image processing module. According to this example, the metadata is stored in a memory where the decoder and the image processing module are integrated in the same device. In this case, the image data is transmitted to the image processing module via a connection link such as a data transmission bus. or a cable if the decoder and image processing module are integrated in a separate device. The image data is transmitted to the image processing module via a digital or wireless connection.

[0071] FIG. 5 illustrates a data stream representing a multi-view video according to a particular embodiment of the invention. 1 shows a schematic of a decoder for

[0072] Conventionally, decoding of the views reconstructed from the data stream STR is done as follows: The decoding of the reconstructed views is performed image-by-image and block-by-block for each image. For each block to be reconstructed, the elements corresponding to that block are Decoded from the data stream STR by an entropy decoding module D, Decoded syntax elements SE (texture encoding mode, motion vector torsion vector, disparity vector, depth encoding mode, reference image index, ...) and quantity A set of coefficients coeff is provided.

[0073] The quantization coefficient coeff is -1 ) and then the inverse transformation model Joule (T -1 ) to obtain the prediction residual value res of the block. rec is provided. The coded syntax elements (SE) are sent to a prediction module (P) to Reconstructed image I ref (A portion of the current image or a previously reconstructed view) A prediction block pre is also calculated using a reference image, or a reference image of another previously reconstructed view. Then, the current block is calculated by dividing the prediction pred by the prediction residual re s rec (B rec ). Then, the reconstructed block B rec ) can be used later to reconstruct the current image or another image or another view. The data is stored in the memory MEM so that

[0074] According to the invention, at the output of the entropy decoding module, the decoding of the blocks The encoded syntax element SE and the optional quantization coefficients are and selecting at least a portion of the quantization coefficients SE and the optional quantization coefficients, and dividing them into a predetermined a module FORM configured to store the reconstructed image in the form of , or metadata MD relating to a group of images is provided.

[0075] The selection of the decoded syntax elements SE to be formatted may be determined by, for example, may be fixed as specified in the standard that describes the operation of The choice of different types can be defined fixedly, e.g. via a decoder profile. The decoder parameterization is done by the syntax supported by the format module FORM. This can be configured to select the x element. The decoder exchanges with the image processing module to which it provides metadata. In this case, the image processing module can transmit to the decoder the The decoder module FORM explicitly indicates the type of metadata it wishes to receive. , select only the requested decoded syntax elements.

[0076] Providing metadata at a level of granularity different from that used by the decoder If possible, such a level of granularity should be specified in the standard describing the decoder's operation. It can be defined in the image processing model or fixedly via the decoder profile. When the module communicates with the decoder to get metadata, the image processing module , the decoder specifies the level of granularity at which this image processing module desires to receive parts of the metadata. can be shown explicitly in

[0077] According to a particular embodiment of the invention, at the output of the entropy decoding module The decoded syntax element SE and the optional quantization coefficients are stored in the syntax element S a module CALC configured to calculate metadata from E and / or the quantized coefficients As mentioned before, the metadata that is computed describes the behavior of the decoder. may be explicitly defined in the applicable standard or according to different profiles or otherwise. or it can be determined from the exchange with the image processing module in question.

[0078] According to a particular embodiment of the invention, the module FORM is in particular Select the camera parameters.

[0079] To synthesize a new viewpoint, the synthesis module first synthesizes each pixel of the original (reconstructed) view. We must create a model that describes how the xels are projected onto the virtual view. Most compositors, e.g., those based on DIBR (Depth Image Based Rendering) techniques, The device uses the depth information to project the pixels of the reconstructed view into 3D space. Then, the corresponding points in 3D space are projected onto the camera plane from the new viewpoint.

[0080] Such a projection of image points in 3D space is calculated using the following formula: M=K.RT.M' where M is the coordinate matrix of the points in 3D space and K is is the matrix of intrinsic parameters of the virtual camera, and RT is the A row of the camera's extrinsic parameters (camera position and orientation in 3D space) column, and M' is the pixel matrix of the current image.

[0081] If the camera parameters are not sent to the synthesis module, the synthesis module will perform high-complexity and their camera parameters must be calculated at the expense of accuracy, and the calculation is It cannot be done in real time or must be obtained by external sensors. Therefore, by providing these parameters through the decoder, the synthesis module This makes it possible to limit the complexity of the rules.

[0082] According to another particular embodiment of the invention, the module FORM is in particular adapted to replay the current image. Select the syntax element for the reference image that will be used to construct it.

[0083] To generate the virtual view, a synthesis module uses various previously reconstructed available If there is a possibility to select a reference image from among the images of the views, the synthesis module Which reference view was used when coding the view used for For example, Figure 10 shows a multi-camera system with 16 cameras. The diagram shows the arrangement of views in the view capture system. The arrows between each frame indicate The decode order of the views is shown. The synthesis module is between view V6 and view V10. A virtual view VV for a viewpoint placed at Conventionally, when a scene needs to be generated, the synthesis module constructs the best virtual view. To achieve this, the availability of each view must be checked.

[0084] According to certain embodiments described herein, for a view, If the synthesis module has metadata indicating the reference view used to reconstruct the image, The virtual viewer uses the virtual image to determine which image should be used to generate the virtual view. Only the available view closest to the point (view V6 in the case of Figure 10) can be selected. For example, if a block in view V6 uses an image in view V7 as a reference image, The composition module is used by view V6 and therefore needs to be available. It may also be decided to use V7. Thus, such an embodiment may be By avoiding the need to check the availability of each view in a synthesis module, Reduce noise.

[0085] According to another particular embodiment of the invention, the module CALC is in particular The motion vector syntax element is selected to generate a motion vector group.

[0086] In areas with little motion, virtual view synthesis generally suffers from inaccuracies in the depth map. These incoherences are due to the virtual viewpoint. This is very hindering for visualization of the

[0087] In this particular embodiment, the decoder module CALC is The motion vectors, i.e., the inverse prediction of the motion vectors and the motion vectors The module CALC selects the motion vector after the scaling of the motion map. Typically, the reconstructed motion vectors of each block are used to generate a binary map. In a binary map, each element takes the value 0 or 1, and the area is Binary maps indicate whether a region has motion or not. For example, erosion, expansion, opening, closing. This can be improved by using (closing)).

[0088] The motion binary map is then scaled to the desired granularity (pixel-level map, block Level map or sub-block level map, or for a specific block size in the image The motion is formatted according to the map defined for the view, and the motion is not It can be indicated whether or not

[0089] A synthesis module that receives such a motion map then may, for example, Apply different compositing processes depending on whether or not is marked as having motion For example, the time incoherence can be reduced by To solve the problem, the traditional synthesis process is disabled in fixed (motionless) regions. and simply inherits the pixel values ​​from the previous image.

[0090] Of course, the synthesis module may be implemented using other means, e.g., as an encoder. By estimating the motion, a motion map can be generated by itself. However, such an operation increases the complexity of the synthesis algorithm and the resulting model. This significantly impacts the accuracy of the conversion because the encoder The reason is to estimate motion from uncoded images that are no longer available. .

[0091] In the example shown in FIG. 10 and in the embodiment described above, the closest available view is Effectively, the method not only uses the virtual viewpoint but also averages the reference views in the vicinity of the virtual viewpoint. For example, we can calculate the reference views V6, V7, V10 and V 11 can be averaged by the decoder module CALC, resulting in The resulting average view can be sent to a synthesis module.

[0092] In another variant, the decoder module CALC calculates the occlusion map. where the occlusion map can be calculated for each pixel or block of the image as: Indicates whether a region corresponds to an occlusion region. For example, the module CALC The information of the reference image(s) used by the decoder to reconstruct the region By using the For example, in the case of FIG. 10, most of the blocks in the image of view V6 are predicted using temporal prediction. At the same time, some blocks in the image of view V6 are inter-view predicted, e.g. When using inter-view prediction for 2, these blocks correspond to occlusion regions. It is highly likely that

[0093] A synthesis module that receives such an occlusion map then determines whether the region is occluded. Deciding to apply different compositing processes depending on whether the region is marked as a region or not. This can be done.

[0094] According to another particular embodiment of the invention, the module CALC is in particular Select a coding mode associated with the texture of the image and the depth map of the image. do.

[0095] According to the prior art, synthesis algorithms mainly use depth maps. The map typically indicates errors that produce artifacts in the synthesized virtual view. By comparing the encoding modes between the texture and the depth map, the decoder , a confidence measure associated with the depth map, e.g., whether depth and texture are correlated (value A binary map can be derived that indicates whether correlations occur (value 1) or are uncorrelated (value 0).

[0096] For example, the confidence value can be derived from the encoding mode. The code mode and depth encoding mode are different. For example, one is intra mode (intr If one is a mode and the other is inter mode, this means that the texture This means that the depth is not correlated with the image, so the confidence value is low, e.g. 0. .

[0097] The confidence values ​​can also be arranged according to the motion vectors. Texture and Depth If the textures and depths have different motion vectors, this indicates that the textures and depths are uncorrelated. Therefore, the confidence value is low, e.g. 0.

[0098] The confidence values ​​can also be arranged according to the reference images used by the texture and depth. If the reference images are different, this means that the texture and depth are uncorrelated. Therefore, the confidence value is low, e.g., 0.

[0099] A synthesis module that receives such a confidence map then determines whether the region has a low confidence value. You can decide to apply different compositing operations depending on whether the For example, for such a region, another reference that provides a better confidence value for the region may be used. The views can be used to synthesize corresponding regions.

[0100] FIG. 6 illustrates steps of an image processing method according to a particular embodiment of the invention. Such processing can be decoded and reconstructed, for example, by the decoding method described in relation to FIG. This is performed, for example, by a virtual view synthesis module from the generated views.

[0101] In step E60, at least one item of metadata (MD1, MD2) is Read by the synthesis module. Metadata read by the synthesis module corresponds to a syntax element decoded from a stream representing multiview video. Each view is associated with one or more views, which are the sum of the views from the decoded syntax elements. It can also accommodate information calculated during the method of decoding the stream. The data is stored or transmitted to the synthesis module in a predetermined format, so that it can be read in a suitable manner. Any composite module that has a read module can read it.

[0102] In step E61, the synthesis module determines at its input, for example, The views reconstructed by the multiview video decoder according to the disclosed decoding method The synthesis module receives at least one image of each of the following four frames: Using the received views VD1 and VD2 and the read metadata MD1 and MD2, Generate at least one image from the imaginary viewpoint VS(1+2). In particular, metadata MD1 , MD2 is used by the synthesis module to determine the A synthesis algorithm to be used to generate the virtual view images is determined. The view to be used is determined.

[0103] FIG. 7 illustrates steps of a decoding method and an image processing method according to another particular embodiment of the present invention. This shows that.

[0104] In general, a multi-view video decoder uses the synthesis device used to generate the virtual viewpoints. In other words, the decoder has no knowledge of the type of synthesis algorithm used. I don't know if it's possible to do this or what metadata types are useful to a decoder.

[0105] In accordance with the particular embodiment described herein, the decoder and synthesis module , adapted to be able to exchange in both directions. For example, the synthesis module ,decode the list of metadata that the synthesis module needs to achieve,better synthesis. Before or after a request from the synthesis module, the decoder can Informs the decoder module of the metadata that the decoder can send to the synthesis module. Advantageously, the list of metadata that the decoders can share can be are standardized, i.e. all decoders conforming to the decoding standard will Therefore, for a given decoding standard, it is necessary to share the same data. Thus, the synthesis module knows what metadata is available. The list of data can also be adapted according to the profile of the decoder specification. For example, For profiles aimed at decoders requiring low computational complexity, the list of metadata is: It contains only the decoded syntax elements of the stream, while at the same time it has a higher computational complexity. For profiles for decoders that can handle it, the list of metadata is Decoded data of the stream, such as the segmentation map, occlusion map, and confidence map. It may also include metadata derived by computation from syntax elements.

[0106] At step E70, the synthesis module instructs the decoder to generate an image from a virtual viewpoint. A request is sent indicating at least one item of metadata required to contains an index or list of indexes, each of which corresponds to the required metadata. nothing.

[0107] Such requests are made according to a predefined format, i.e., The message is sent according to a specific syntax so that it can be understood by any user. For example, Such a syntax could be: nb For an integer i in the range 0 to nb-1, list[i] where the syntax element nb is the number of metadata required by the synthesis module. , which indicates the number of indices to be read by the decoder, and list[i ] indicates the index of each of the required metadata.

[0108] By way of example, taking the example metadata given by Table 1 above, the composite module The rule is to specify nb=2 in the request, and the camera parameters and occlusion map are The corresponding indexes 0 and 9 can be indicated.

[0109] According to one variant, the synthesis module calculates the index of the item of required metadata. For example, the "grlevel" syntax element associated with a metadata item. A level of granularity can also be indicated by specifying a predefined value for the occlusion. For occlusion maps, the synthesis module computes the occlusion map at the pixel level. If you want to use a level element with a value of 1 associated with index 9, or a more coarse For example, you may want an occlusion map for a block of size 8x8 at a low level. In this case, the "level" element associated with index 9 can have a value of 2.

[0110] In step E71, the decoder retrieves the corresponding metadata. For this purpose, According to the example described above in relation to FIG. 4 or FIG. 5, the decoder obtains the metadata. It finds the decoded syntax elements needed to reconstruct the occlusion map, etc. Calculate the metadata that is not used by the decoder for construction. Then, The metadata is stored in a file according to a given format so that the synthesis module can read it. It will be formatted.

[0111] In step E72, the decoder transmits the metadata to the synthesis module, after which The synthesis module can use the metadata in its synthesis algorithms. .

[0112] FIG. 8 shows a block diagram of a decoding method according to the above-described specific embodiment of the present invention. 1 shows diagrammatically the adapted device DEC.

[0113] Such a decoding device comprises a memory MEM and, for example, a processor PROC. A processing unit UT controlled by a computer program PG stored in a MEM The computer program PG is configured to include the following: instructions that, when executed by the ROC, implement the steps of the decoding method described above; include.

[0114] According to a particular embodiment of the invention, the decoding device DEC comprises, inter alia, receiving a data stream representing a multi-view video over a communications network; It has a communications interface COM0 that enables

[0115] According to another particular embodiment of the invention, the decoding device DEC is adapted to receive a composite model The metadata is then sent to the image processing device, such as a digital camera, and reconstructed from the data stream. It has a communications interface COM1 that allows the transmission of images of the generated views. .

[0116] At initialization, the code instructions of the computer program PG are executed by the processor PRO It is loaded into memory before being executed by C. In particular, the processor of the processing unit UT PROC executes the program according to the instructions of the computer program PG in relation to Figs. 4, 5 and 7. The steps of the described decoding method are carried out. The memory MEM contains, inter alia, a memory of a predetermined format and adapted to store the metadata obtained during the decoding method.

[0117] According to a particular embodiment of the invention, the above-described decoding device is Mobile phones (e.g., smartphones), set-top boxes, virtual reality headsets, etc. Contained within the terminal.

[0118] FIG. 9 illustrates a block diagram of a computer-implemented image processing method according to the above-described specific embodiment of the present invention. 1 shows diagrammatically the adapted device SYNTH.

[0119] Such a device comprises a memory MEM9 and, for example, a processor PROC9, A processing unit UT controlled by a computer program PG9 stored in EM9 The computer program PG9 is a program that is executed by a processor. When executed by PROC9, it performs the steps of the image processing method as described above. This includes an order to enforce the same.

[0120] According to a particular embodiment of the invention, the device SYNTH is a device as described above. It receives metadata sent from a decoding device such as DEC, and also receives the metadata from the device DEC. receiving images of reconstructed views from a data stream representing a multi-view video; The NI PXIe-8110 is equipped with a communications interface COM9 that enables

[0121] At initialization, the code instructions of the computer program PG9 are, for example, It is loaded into memory before being executed by OC9. In particular, the process of the processing unit UT9 The processor PROC9 executes the steps shown in FIG. 6 and FIG. 7 according to the instructions of the computer program PG9. The image processing method steps described above are carried out.

[0122] According to a particular embodiment of the invention, the device SYNTH is Output interface AFF9 that allows sending images to a device, e.g. a screen For example, such an image may be obtained by combining the images of the reconstructed views and the images received from the device DEC. The image from the virtual viewpoint is generated by the device SYNTH using the transmitted metadata. We can respond to this.

[0123] According to a particular embodiment of the invention, the device SYNTH is a synthesis module. Modules include television sets, mobile phones (e.g., smartphones), set-top boxes, The technology is included within devices such as smartphones, tablets, and virtual reality headsets.

[0124] The principles of the present invention will now be described in the context of a multi-view video decoding system, In this case, multiple views are decoded from the same stream (bitstream) and Data is acquired view by view. This principle is used because multi-view video consists of multiple streams. Each stream is encoded using a single view (bitstream). The same is true when the view decoder is encoded in the same way. In this case, each view decoder Provides metadata associated with the view to be loaded.

Claims

1. Decoding of a data stream representing a multi-view video, performed by a decoding device. A method for coding, comprising: Obtaining syntax elements from at least one portion of the data stream ( E20) and Reconstructing at least one image of a view of the video from the retrieved syntax elements. Constructing (E21); and the decoding method comprises: At least one piece of metadata in a predetermined format from at least one syntax element. Obtaining an item (E23); Providing at least one item of said metadata to an image synthesis module (E24). )and, 4. The method of decoding, further comprising:

2. Obtaining at least one item of metadata includes obtaining at least one of the syntax elements. and calculating at least one item of said metadata from both portions.

2. The decoding method according to claim 1.

3. At least one item of metadata is used to reconstruct the at least one image.

3. The method of claim 1, wherein the decoding method is not used for

4. At least one item of metadata may be: Camera parameters, the decoded and scaled motion vectors, Segmentation of the reconstructed image; a reference image used by a block of an image of said reconstructed view; a coding mode of the image of the reconstructed view; a quantization parameter value of the image of the reconstructed view; a prediction residual value of the image of the reconstructed view; a map representing motion within the images of said reconstructed views; a map indicative of the presence of occlusions in the image of the reconstructed view; a map representing confidence values ​​associated with the depth maps; According to any one of claims 1 to 3, which corresponds to an item of information contained within a group including How to decode.

5. The predetermined format may include at least one item of metadata associated with the index. The data according to any one of claims 1 to 4, which corresponds to an indexed table, Code method.

6. At least one item of the metadata is encoded at a specified granularity in the decoding device. The decoding method according to any one of claims 1 to 5, wherein the level is obtained based on the level.

7. The decoding device receives metadata required by the image synthesis module. receiving from the image synthesis module a request to indicate at least one item of A decoding method according to any one of claims 1 to 6.

8. The request selects the required metadata from a predefined list of available metadata.

8. The method of claim 7, further comprising at least one index indicating an item of data.

9. A device for decoding a data stream representing a multi-view video, comprising: The device comprises: Obtaining syntax elements from at least one portion of the data stream; Reconstructing at least one image of a view of the video from the retrieved syntax elements. Compose, (UT, MEM, COM1), The decoding device comprises: At least one piece of metadata in a predetermined format from at least one syntax element. Get the item, providing at least one item of said metadata to an image synthesis module; The decoding device further comprises:

10. From at least one image of the view decoded by the decoding device, a virtual view 13. A method of image synthesis comprising: generating at least one image of 、 Retrieving (E60) at least one item of metadata in a predetermined format, comprising: At least one item of the metadata is encoded by the decoding device into a multi-view video. At least one syntax element obtained from a data stream representing video. To be done, using at least one retrieved item of said metadata. generating another image (E61); 13. An image synthesis method comprising:

11. The decoding device includes at least one of the metadata required to generate the image. The method of claim 10 further comprising sending a request indicating one item.

12. From at least one image of the view decoded by the decoding device, a virtual view an image synthesis device configured to generate at least one image of The device is The image synthesis device is configured to read at least one item of metadata in a predetermined format. (UT9, MEM9, COM9), and at least one item of said metadata is The decoding device obtains a small amount of the multi-view video signal from a data stream representing the multi-view video. is derived from at least one syntax element; and At least one of the metadata is read when the at least one image is generated. The extracted items are used, An image synthesis device comprising:

13. Image displaying a multi-view video from a data stream representing the multi-view video - Patent Application 20070123633 1. A processing system comprising: A decoding device according to claim 9 ; An image synthesis device according to claim 12; An image processing system comprising:

14. A method of decoding according to any one of claims 1 to 8, when executed by a processor. or a computer including instructions for carrying out the image synthesis method according to claim 10 or 11. Computer program.

Images