Method and apparatus for encoding, transmitting and decoding volumetric video

Abstract

Methods, devices and streams for encoding, decoding and transmitting multi-view frames are disclosed. A non-pruned MVD frame is obtained and a graph representing pruning priority relationships between views is determined. The MVD is pruned by using these priority relationships. The pruned MVD and data representing the graph are encoded in a data stream. At decoding, the contribution of each view of a pixel of a viewport frame to be generated is determined from the decoded pruning graph.

Description

1. TECHNICAL FIELD

[0001] The present principles generally relate to the domain of three-dimensional (3D) scene and volumetric video content. The present document is also understood in the context of encoding, formatting and decoding data representing textures and geometry of 3D scenes, to render volumetric content on end-user devices such as mobile devices or head-mounted displays (HMDs). Among other subjects, the present principles relate to pruning pixels of multi-view images to guarantee optimal bitstream and rendering quality. 2. BACKGROUND

[0002] This section is intended to introduce the reader to various aspects of art, which can be related to various aspects of the present principles that are described and / or claimed below. This discussion is intended to provide the reader with background information to facilitate a better understanding of the various aspects of the present principles. Accordingly, it should be understood that this section is to be read in this light, and not as admissions of prior art.

[0003] Recently, there has been a growth in available large field of view content (up to 360°). A user watching content on an immersive display device (such as a head-mounted display, smart glasses, PC screen, tablet, smartphone, etc.) can not be able to see the whole of such content. This means that at a given moment, the user can only watch a part of the content. However, the user can usually navigate within the content through various means such as head movement, mouse movement, touch screen, voice, and the like. It is generally desirable to encode and decode this content.

[0004] Immersive video (also called 360° planar video) allows a user to watch everything around him by rotating his head around a static viewpoint. The rotation allows only a 3 degrees of freedom (3DoF) experience. Even if 3DoF video is sufficient to meet the requirements of a first omnidirectional video experience (e.g. using a head-mounted display (HMD device)), 3DoF video can quickly become frustrating for a viewer expecting more freedom (e.g. by experiencing parallax). Moreover, 3DoF can also cause dizziness because a user never only rotates his head but also translates his head in three directions, which are not reproduced in a 3DoF video experience.

[0005] Among others, the large field of view content can be a three-dimensional computer graphics image scene (3D CGI scene), a point cloud or an immersive video. Many terms can be used to design such immersive video: for example, Virtual Reality (VR), 360, panoramic, 4π steradians, immersive, omnidirectional or large field of view.

[0006] Volume videos, also called 6 Degrees of Freedom (6DoF) videos, are an alternative to 3DoF videos. When watching a 6DoF video, in addition to rotations, the user can translate his head, and even his body, in the content of the watch, and experience parallax and even volume. This video significantly increases the immersion and the perception of the depth of the scene and prevents dizziness by providing a consistent visual feedback during head translation. The content is created by dedicated sensors that allow the simultaneous recording of the color and depth of the scene of interest. Even if technical difficulties still exist, the use of a color camera equipment combined with photogrammetry techniques is one way to perform such a recording.

[0007] While 3DoF videos consist of a sequence of images resulting from the de-mapping of texture images (e.g. spherical images encoded according to a latitude / longitude projection mapping or an equirectangular projection mapping), 6DoF video frames embed information from multiple viewpoints. They can be seen as a temporal sequence of point clouds resulting from a three-dimensional capture. Two kinds of volume videos can be considered depending on the viewing conditions. The first one, i.e. full 6DoF, allows a complete free navigation within the video content, while the second one, also called 3DoF+, restricts the user viewing space to a limited volume called the viewing bounding box, allowing a limited head translation and parallax experience. This second case is a valuable compromise between free navigation and passive viewing conditions for seated audiences.

[0008] 3DoF+ content can be set as a set of Multi-View+Depth (MVD) frames. Such content can have been captured by dedicated cameras or can be generated from existing Computer Graphics (CG) content with the help of dedicated (possibly realistic) rendering. Volume information is delivered as a combination of color and depth patches stored in corresponding color and depth atlas, which are video encoded with regular codecs (e.g. HEVC). Each combination of color and depth patches represents a sub-portion of the MVD input view and the set of all patches is designed at the encoding stage to cover the whole scene while there is as little redundancy as possible. At the decoding stage, the atlas is first video decoded and the patches are rendered in a view synthesis process to recover the viewport associated with the desired viewing position. The problem of such a solution is related to the way the patches are created to be sufficiently non-redundant and complementary. 3. SUMMARY

[0009] The following presents a simplified summary of the principles of the application in order to provide a basic understanding of some aspects of the application. This summary is not an extensive overview of the principles of the application. It is not intended to identify key or critical elements of the principles of the application. The following summary merely presents some aspects of the principles of the application in a simplified form as a prelude to the more detailed description provided below.

[0010] The principles of the application relate to a method for encoding a pruned multi-view frame in a data stream.

[0011] The method comprises:

[0012] - obtaining an acyclic graph linking views of the non-pruned multi-view frame, the links of the graph representing view pruning priorities;

[0013] - pruning pixels of views of the multi-view image in an order determined so that the first view is pruned after the views linked to the first view by pruning priority links; and

[0014] - encoding the graph and the pruned views in a data stream.

[0015] The principles of the application also relate to a device comprising a processor configured to implement the method.

[0016] The principles of the application also relate to a method of decoding a pruned multi-view frame from a data stream. The method comprises:

[0017] - obtaining the pruned multi-view frame from the data stream;

[0018] - obtaining an acyclic graph linking views of the multi-view image from the data stream, the links of the graph representing view pruning priorities;

[0019] - generating a viewport frame from a viewing pose by determining a contribution of each view of the pruned multi-view frame according to the pruning priorities of the graph.

[0020] The principles of the application also relate to a device comprising a processor configured to implement the method.

[0021] The principles of the application also relate to a data stream comprising:

[0022] - data representative of a pruned multi-view frame; and

[0023] - data representative of an acyclic graph linking views of the multi-view image, the links of the graph representing view pruning priorities. 4. BRIEF DESCRIPTION OF DRAWINGS

[0024] The present disclosure will be better understood and other specific features and advantages will emerge upon reading the following description, the description being given with reference to the attached drawings in which:

[0025] - Figure 1 A three-dimensional (3D) model of an object and points of a point cloud corresponding to this 3D model are shown according to a non-limiting embodiment of the principles of the application;

[0026] - Figure 2Non-limiting examples of encoding, transmitting, and decoding data representing a sequence of 3D scenes according to a non-limiting embodiment of the principles of the present invention are shown;

[0027] - Figure 3 The non-limiting embodiments shown in this invention can be configured to implement regarding... Figure 11 and Figure 12 Example architecture of the device described in the method;

[0028] - Figure 4 Examples of embodiments of the syntax of a stream when transmitting data via a packet-based transport protocol are shown, according to a non-limiting embodiment of the principles of the present invention;

[0029] - Figure 5 A patch atlas method with four projection centers is shown as an example of a non-limiting embodiment based on the principles of the present invention;

[0030] - Figure 6 An example of a texture atlas including points of a 3D scene, according to a non-limiting embodiment of the principles of the present invention, is shown;

[0031] - Figure 7 The illustration shows a non-limiting embodiment based on the principles of the invention, including... Figure 6 An example of a 3D scene atlas containing depth information of points;

[0032] - Figure 8 The process used by the view compositor when generating an image of a given viewport from an untrimmed MVD frame is illustrated by a non-limiting embodiment of the principle of the invention;

[0033] - Figure 9 The non-limiting embodiments of the present invention are shown. Figure 8 The same view synthesized from trimmed MVD frames;

[0034] - Figure 10 A 4x4 multi-view frame and an example trimmed diagram for such MVD frames are shown, representing a non-limiting embodiment of the invention based on the principles of the present invention.

[0035] - Figure 11 A method for encoding multi-view frames in a data stream is shown according to a non-limiting embodiment of the principles of the present invention;

[0036] - Figure 12 A method for decoding trimmed multiview frames from a data stream is shown according to a non-limiting embodiment of the principles of the present invention. 5. Detailed Implementation

[0037] The principles of the present application will be more fully understood in view of the detailed description and the drawings attached hereto. The principles of the present application can be embodied in many alternative forms and should not be construed as limited to the examples set forth herein. Accordingly, while the principles of the present application are susceptible to various modifications and alternative forms, specific examples thereof are shown by way of example in the drawings and will be described herein in detail. It should be understood, however, that there is no intent to limit the principles of the present application to the particular forms disclosed, but on the contrary, the intention is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the principles of the present application as defined by the claims.

[0038] The terminology used herein is for the purpose of describing particular examples only and is not intended to be limiting of the principles of the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Additionally, when an element is referred to as being "responsive" or "connected" to another element, it can be directly responsive or connected to the other element, or indirectly responsive or connected to the other element through one or more other elements. In contrast, when an element is referred to as being "directly responsive" or "directly connected" to another element, there are no intervening elements. As used herein the term "and / or" includes any and all combinations of one or more of the associated items and can be abbreviated as " / ".

[0039] It should be understood that, although the terms first, second, etc. can be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element, without departing from the teachings of the present principles.

[0040] Although some of the diagrams include arrows on communication paths to show a primary direction of communication, it is to be understood that communication can occur in the opposite direction to the depicted arrows.

[0041] Some examples are described with respect to block and operational flow diagrams that include blocks interconnecting with other blocks representing circuit elements, modules or code portions for performing specified functions. It should also be noted that in other specific implementations, the function(s) noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved.

[0042] Reference in this document to "one example" or "an example" means that a particular feature, structure, or characteristic described in connection with the example is included in at least one implementation of the present principles. The appearances of the phrase "one example" or "an example" in various places in the specification are not necessarily all referring to the same example, nor are they necessarily mutually exclusive, or alternative examples to one another.

[0043] Reference signs in the claims are presented by way of illustration only and should not be construed as limiting the scope of the claims. Although not explicitly described, the present examples and variants can be employed in any combination or sub-combination.

[0044] Figure 1 A three-dimensional (3D) model 10 of an object and points of a point cloud 11 corresponding to the 3D model 10 are shown. The 3D model 10 and the point cloud 11 can for example correspond to a possible 3D representation of an object of a 3D scene comprising other objects. The model 10 can be a 3D mesh representation and the points of the point cloud 11 can be the vertices of the mesh. The points of the point cloud 11 can also be points distributed on the surface of the faces of the mesh. The model 10 can also be represented as a splatted version of the point cloud 11, the surface of the model 10 being created by splatting the points of the point cloud 11. The model 10 can be represented by many different representations such as voxels or splines. Figure 1 The fact that a point cloud can be defined with a surface representation of a 3D object and that a surface representation of a 3D object can be generated from a cloud of points is shown. As used herein, projecting points of a 3D object (by extension points of a 3D scene) onto an image is equivalent to projecting any representation of this 3D object, for example a point cloud, a mesh, a spline model or a voxel model.

[0045] A point cloud can be represented in memory as for example a vector-based structure where each point has its own coordinates in the frame of reference of a viewpoint (for example three-dimensional coordinates XYZ, or an angle in space and a distance from / to the viewpoint (also called depth) and one or more attributes, also called components. One example of a component is a color component which can be represented in various color spaces, for example RGB (red, green and blue) or YUV (Y is the luminance component and UV are two chrominance components). The point cloud is a representation of a 3D scene comprising an object. The 3D scene can be seen from a given viewpoint or range of viewpoints. The point cloud can be obtained in many ways, for example:

[0046] • from a capture of a real object taken by a camera rig, optionally complemented with depth active sensing devices;

[0047] • from a capture of a virtual / synthetic object taken by a virtual camera rig in a modeling tool;

[0048] • from a mix of both real and virtual objects.

[0049] 3D scenes, especially when produced for 3DoF+ rendering, can be represented by Multi-View+Depth (MVD) frames. Then, volumetric videos are sequences of MVD frames. In this approach, volumetric information is delivered as a combination of color and depth patches stored in corresponding color and depth atlases, which are then video coded with regular codecs, typically HEVC. Each combination of color and depth patches represents a sub-portion of the MVD input view and the set of all patches is designed at the encoding stage to cover the whole scene while there is as little redundancy as possible. At the decoding stage, the atlases are first video decoded and the patches are rendered in a view synthesis process to recover the viewport associated with the desired viewing position.

[0050] Figure 2 Non-limiting examples of encoding, transmitting and decoding data representing a sequence of 3D scenes are illustrated. The encoding format can be compatible with 3DoF, 3DoF+ and 6DoF decoding, for example, at the same time.

[0051] A sequence of 3D scenes 20 is obtained. As a sequence of pictures is a 2D video, a sequence of 3D scenes is a 3D (also called volumetric) video. The sequence of 3D scenes can be provided to a volumetric video rendering device for 3DoF, 3Dof+ or 6DoF rendering and display.

[0052] The sequence of 3D scenes 20 can be provided to an encoder 21. The encoder 21 takes as input one 3D scene or a sequence of 3D scenes and provides a bitstream representing the input. The bitstream can be stored in a memory 22 and / or on an electronic data medium and can be transmitted over a network 22. The bitstream representing the sequence of 3D scenes can be read from the memory 22 and / or received from the network 22 by a decoder 23. The decoder 23 takes as input the bitstream and provides a sequence of 3D scenes in a point cloud format, for example.

[0053] The encoder 21 can comprise several circuits implementing several steps. In a first step, the encoder 21 projects each 3D scene onto at least one 2D picture. A 3D projection is any method of mapping three-dimensional points into a two-dimensional plane. Since most current methods for displaying graphical data are based on planar (pixel information from several bitplanes) two-dimensional media, the use of this type of projection is widespread, especially in computer graphics, engineering, and cartography. A projection circuit 211 provides a sequence of 3D scenes 20 with at least one two-dimensional frame 2111. The frame 2111 comprises color information and depth information representing the 3D scene projected onto the frame 2111. In a variant, the color information and the depth information are encoded in two separate frames 2111 and 2112.

[0054] The metadata 212 is used and updated by the projection circuit 211. The metadata 212 comprises information about the projection operation (e.g. projection parameters) and information about the way color and depth information is organized within the frames 2111 and 2112, as explained in connection with Figures 5 to 7

[0055] The video encoding circuit 213 encodes the sequence of frames 2111 and 2112 into a video. The pictures 2111 and 2112 of the 3D scene (or the sequence of pictures of the 3D scene) are encoded by the video encoder 213 in a stream. The video data and the metadata 212 are then encapsulated by the data encapsulation circuit 214 in a data stream.

[0056] The encoder 213 is for example compatible with encoders such as:

[0057] - JPEG, specification ISO / CEI 10918-1 UIT-T Recommendation T.81, https: / / www.itu.int / rec / T-REC-T.81 / en;

[0058] - AVC, also known as MPEG-4 AVC or h264. Specified in both UIT-T H.264 and ISO / CEI MPEG-4 Part 10 (ISO / CEI 14496-10), http: / / www.itu.int / rec / T-REC-H.264 / en, HEVC (whose specification is found on the ITU website, T recommendation, H series, h265, http: / / www.itu.int / rec / T-REC-H.265-201612-I / en);

[0059] - 3D-HEVC (extension of HEVC, whose specification is found on the ITU website, T recommendation, H series, h265, http: / / www.itu.int / rec / T-REC-H.265-201612-I / en annex G and I);

[0060] - VP9 developed by Google;

[0061] - AV1 (AOMedia Video 1) developed by the Alliance for Open Media; or

[0062] - future standards such as Versatile Video Coding or MPEG-I or future versions of MPEG-V.

[0063] ​The data stream is stored in a memory accessible by the decoder 23, for example through the network 22. The decoder 23 comprises different circuits implementing different decoding steps. The decoder 23 takes as input the data stream generated by the encoder 21 and provides a sequence of 3D scenes 24 to be rendered and displayed by a volumetric video display device such as a head mounted device (HMD). The decoder 23 obtains the stream from a source 22. For example, the source 22 belongs to a group comprising:

[0064] - a local memory, for example a video memory or a RAM (or Random Access Memory), a flash memory, a ROM (or Read Only Memory), a hard disk;

[0065] - a storage interface, for example an interface with a mass storage device, a RAM, a flash memory, a ROM, an optical or magnetic support;

[0066] - a communication interface, for example a wired interface (for example a bus interface, a wide area network interface, a local area network interface) or a wireless interface (such as an IEEE 802.11 interface or a Bluetooth® interface);

[0067] and

[0068] - a user interface enabling a user to input data, such as a graphical user interface.

[0069] The decoder 23 comprises a circuit 234 for extracting the data encoded in the data stream. The circuit 234 takes as input the data stream and provides metadata 232 corresponding to the metadata 212 encoded in the stream and a two-dimensional video. The video is decoded by a video decoder 233 providing a sequence of frames. The decoded frames comprise color and depth information. In a variant, the video decoder 233 provides two sequences of frames, one containing color information and the other containing depth information. The circuit 231 uses the metadata 232 to project the color and depth information from the decoded frames to provide a sequence of 3D scenes 24. The sequence of 3D scenes 24 corresponds to the sequence of 3D scenes 20, possibly with a loss of precision related to the encoding as a 2D video and to the video compression.

[0070] Figure 3 An example architecture of a device 30 that can be configured to implement the methods described with respect to Figure 11 and Figure 12 is illustrated. Figure 2 The encoder 21 and / or the decoder 23 can implement this architecture. Alternatively, each circuit in the encoder 21 and / or the decoder 23 can be a device according to the architecture of Figure 3 linked together, for example via their buses 31 and / or via their I / O interfaces 36.

[0071] The device 30 comprises the following elements connected together by data and address buses 31 : ​

[0072] - a microprocessor 32 (or CPU), which is for example a DSP (or Digital Signal Processor);

[0073] - a ROM (or Read Only Memory) 33;

[0074] - a RAM (or Random Access Memory) 34;

[0075] - a storage interface 35;

[0076] - an I / O interface 36 for receiving data to be transmitted from an application; and

[0077] - a power supply, for example a battery.

[0078] According to one example, the power supply is external to the device. In each of the mentioned memories, the word "register" used in the description can correspond to an area of small capacity (a few bits) or to a very large area (for example, the entire program or a large amount of received or decoded data). The ROM 33 comprises at least the program and the parameters. The ROM 33 can store, according to the principles of the application, algorithms and instructions for performing the techniques. When switched on, the CPU 32 uploads the program in the RAM and executes the corresponding instructions.

[0079] The RAM 34 comprises the program in the registers executed by the CPU 32 and uploaded after switching on of the device 30, the input data in the registers, the intermediate data in the different states of the method in the registers, and other variables for executing the method in the registers.

[0080] The specific implementations described herein can be implemented in, for example, a method or process, an apparatus, a computer program product, a data stream, or a signal. Even if discussed in the context of a single form of implementation (for example, discussed only as a method or as an apparatus), the implementation of the discussed features can also be implemented in other forms (for example, program). An apparatus can be implemented in, for example, appropriate hardware, software, and firmware. The method can be implemented in, for example, an apparatus that is generally referred to as a processing device, such as, for example, a processor, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes communication devices, such as, for example, a computer, a cell phone, a portable / personal digital assistant ("PDA"), and other devices that facilitate communication of information between end users.

[0081] According to an example, the device 30 is configured to implement the method described in relation to Figure 11 and Figure 12 The method described is also in the set comprising:

[0082] - a mobile device;

[0083] - a communication device;

[0084] - a game device;

[0085] - a tablet (or tablet computer);

[0086] - a laptop;

[0087] - a still picture camera;

[0088] - a video camera;

[0089] - an encoding chip;

[0090] - a server (e.g. a broadcast server, a video on demand server or a web server).

[0091] Figure 4 An example of an implementation of the syntax of a stream when transmitting data through a packet-based transmission protocol is shown. Figure 4 An exemplary structure 4 of a volumetric video stream is shown. The structure is contained in a container that organizes the stream in independent elements of syntax. The structure can comprise a header part 41 that is a set of data common to each syntax element of the stream. For example, the header part comprises some metadata about the syntax elements, describing the nature and the role of each of them. The header part can also comprise a part of the metadata 212, for example the coordinates of the center viewpoint used to project the points of the 3D scene onto the frames 2111 and 2112. The structure comprises a payload that comprises syntax elements 42 and at least one syntax element 43. The syntax elements 42 comprise data representing color and depth frames. The images can have been compressed according to a video compression method. Figure 2

[0092] The syntax element 43 is part of the payload of the data stream and can comprise metadata about how the frames of the syntax elements 42 are encoded, for example parameters used to project and pack the points of the 3D scene onto the frames. Such metadata can be associated with each frame or group of frames of the video, also called group of pictures (GoP) in video compression standards.

[0093] Figure 5 A patch atlas approach is shown as an example with 4 projection centers. The 3D scene 50 comprises a person. For example, the projection center 51 is a perspective camera and the camera 53 is an orthographic camera. The cameras can also be omnidirectional cameras with for example a spherical mapping (e.g. equirectangular mapping) or a cubical mapping. According to the projection operations described in the projection data of the metadata, the 3D points of the 3D scene are projected onto 2D planes associated with virtual cameras located at the projection centers. In the example, the projection of the points captured by the camera 51 is mapped onto the patch 52 according to a perspective mapping and the projection of the points captured by the camera 53 is mapped onto the patch 54 according to an orthographic mapping. Figure 5

[0094] ​​The clusters of projected pixels yield a plurality of 2D patches, which are packed in a rectangular atlas 55. The organization of the patches within the atlas defines an atlas layout. In an embodiment, there are two atlases with the same layout: one for texture (i.e. color) information, and one for depth information. Two patches, captured by the same camera or by two different cameras, can include information representative of the same part of the 3D scene, as for example patches 54 and 56.

[0095] The packing operation yields patch data for each generated patch. The patch data includes a reference to the projection data (e.g. an index in a table of projection data or a pointer to the projection data (i.e. an address in memory or in a data stream)) and information describing the position and size of the patch within the atlas (e.g. top-left coordinates, size and width in pixels). The patch data items are added to the metadata to be encapsulated in the data stream in association with the compressed data of one or both atlases.

[0096] Figure 6 An example of an atlas 60 including texture information (e.g. RGB data or YUV data) of points of a 3D scene is shown, according to a non-limiting embodiment of the present principles. As explained in connection with Figure 5 The atlas is a packed image of patches, the patches being views obtained by projecting a part of points of the 3D scene.

[0097] In the example of Figure 6 , the atlas 60 includes a first part 61 including texture information of points of the 3D scene visible from the viewpoint and one or more second parts 62. The texture information of the first part 61 can be obtained for example according to an equirectangular projection mapping, which is an example of a spherical projection mapping. In the example of Figure 6 , the second parts 62 are arranged at the left and right borders of the first part 61, but the second parts can be arranged differently. The second parts 62 include texture information of parts of the 3D scene complementary to the part visible from the viewpoint. The second parts can be obtained by removing from the 3D scene the points visible from the first viewpoint (whose texture is stored in the first part) and projecting the remaining points according to the same viewpoint. The latter process can be iteratively repeated to obtain each time a hidden part of the 3D scene. According to a variant, the second parts can be obtained by removing from the 3D scene the points visible from a viewpoint (e.g. a central viewpoint) (whose texture is stored in the first part) and projecting the remaining points according to a viewpoint different from the first viewpoint, for example from one or more second viewpoints centered in the central viewpoint (e.g. a viewing space of a 3DoF rendering).

[0098] The first part 61 can be seen as a first large texture patch (corresponding to a first part of the 3D scene) and the second part 62 comprises smaller texture patches (corresponding to a second part of the 3D scene complementary to the first part). Such atlas has the advantage of being compatible with both 3DoF rendering (when only the first part 61 is rendered) and 3DoF+ / 6DoF rendering.

[0099] Figure 7 An example of an atlas 70 comprising depth information of points of a 3D scene according to a non-limiting embodiment of the present principles is shown. The atlas 70 can be seen as a depth image corresponding to the texture image 60 Figure 6 Figure 6 The atlas 70 comprises a first part 71 comprising depth information of points of the 3D scene visible from the center viewpoint and one or more second parts 72. The atlas 70 can be obtained in the same way as the atlas 60 but contains depth information associated with points of the 3D scene instead of texture information.

[0100] For 3DoF rendering of a 3D scene, only one viewpoint is considered, typically the center viewpoint. The user can rotate his head around the first viewpoint with three degrees of freedom to watch various parts of the 3D scene, but the user cannot move this unique viewpoint. The points of the scene to be encoded are the points visible from this unique viewpoint and only the texture information needs to be encoded / decoded for 3DoF rendering. For 3DoF rendering, the points of the scene that are not visible from this unique viewpoint do not need to be encoded as the user cannot access them.

[0101] For 6DoF rendering, the user can move the viewpoint around in the scene. In this case, every point of the scene (depth and texture) in the bitstream needs to be encoded as the user who can move his / her viewpoint can access every point. At the encoding stage, there is no means to know a priori from which viewpoint the user will observe the 3D scene.

[0102] For 6DoF rendering, the user can move the viewpoint around in the scene. In this case, every point of the scene (depth and texture) in the bitstream needs to be encoded as the user who can move his / her viewpoint can access every point. At the encoding stage, there is no means to know a priori from which viewpoint the user will observe the 3D scene.

[0103] ​For 3DoF+ rendering, the user can move the viewpoint within a limited space around the center viewpoint. This enables the experience of parallax. The data representing the part of the scene visible from any point of the viewing space will be encoded into the stream, including the data representing the 3D scene visible from the center viewpoint (i.e. the first part 61 and 71). For example, the size and shape of the viewing space can be decided and determined at the encoding step and encoded in the bitstream. The decoder can obtain this information from the bitstream and the renderer will limit the viewing space to the one determined by the obtained information. According to another example, the renderer determines the viewing space according to hardware constraints, for example related to the capabilities of the sensors detecting the user movements. In this case, if a point visible from a point within the viewing space of the renderer has not been encoded in the bitstream at the encoding stage, this point will not be rendered. According to yet another example, the data representing each point of the 3D scene (e.g. the texture and / or the geometry) is encoded in the stream without considering the rendering viewing space. To optimize the size of the stream, only a subset of the points of the scene can be encoded, for example a subset of the points that can be seen according to the rendering viewing space.

[0104] The patches are created to be sufficiently non-redundant and complementary. The process for generating patches from a multi-view+depth (MVD) representation of a 3D scene includes “pruning” the input source views to remove any redundant information. To this end, each input view (color+depth) is iteratively pruned one after the other. A set of unpruned views (called base views) is first selected among the source views and transmitted completely. Then, a set of remaining views (called additional views) is iteratively processed to remove information redundant (in terms of color and depth similarity) with the base views and the additional views that have already been pruned. The color or depth value of pruned pixels is replaced by a predetermined value, for example 0 or 255.

[0105] Figure 8 The process used by the view synthesizer 231 of Figure 2 is illustrated when generating an image for a given viewport from a non-pruned MVD frame. To deliver volumetric video, the key step includes removing redundant information between base and additional views. However, even if it significantly reduces the amount of information to be transmitted, simply removing redundant information without any other signaling can significantly alter the view synthesis process at the decoding stage and strongly degrade the end-user experience. When trying to synthesize a pixel to synthesize 81 for a viewport 80, the synthesizer (e.g. the circuit 231 of Figure 2 deprojects the rays (e.g. rays 82 and 83) passing through this given pixel and checks the contribution of each source camera 84 to 87 along this ray. As Figure 8As shown, when some objects in the scene create obstruction from one camera to another, or when visibility cannot be guaranteed due to camera settings, a consensus on the characteristics of the pixels to be composited can not be found among all source cameras (84-87). Figure 8 In the example, as they all "see" the object along the light rays to be composited, the first group of three cameras 84 to 86 "vote" to composite pixel 81 using the color of the foreground object 88. The second group of one camera 87 cannot see the object because it is outside its viewport. Therefore, camera 87 "votes" for the background object 89 to be used for composite pixel 81. The strategy to eliminate this situation is to blend and / or merge each camera's contribution with a weight based on their distance from the viewport to be composited. Figure 8 In the example, the first group of cameras 84 to 86 contributes the most as more of these cameras are used and as they are closer to the viewport to be composited. Ultimately, it can be predicted that the characteristics of the foreground object 88 will be utilized to composite pixel 81.

[0106] Figure 9 It shows the relationship with Figure 8 The same view synthesized from trimmed MVD frames. In trimmed MVD frames, pixels from cameras that share the same information are removed and are no longer transmitted or considered. Figure 9 In the example, the three cameras from the previous group are now reduced to a single camera 96 ​​carrying information about the foreground object 88. The corresponding pixel information 92 from the views of cameras 84 and 85 has been trimmed. The second group of cameras associated with the background object 89 remains unchanged and only includes the view from camera 87. In this case, the background's contribution to the synthesized pixel 91 is no longer negligible relative to the foreground's contribution, as the "opposites" are now reduced to a 1:1 ratio. Even though the weight of object 88 is still slightly higher than that of background 89, the blending of the two contributions will still contain a significant amount from the background that does not correspond to what the user expects and leads to visual artifacts. Therefore, when attempting to synthesize a new view from the graph, it may be crucial at the decoding stage that some camera contribution information from after the trimming stage has been lost.

[0107] According to the principles of the present invention, a method for overcoming these drawbacks is disclosed. At the encoding stage, a trimmed map is obtained. This trimmed map constrains the trimming performed by each camera relative to the other cameras in a given subgroup. Data representing the trimmed map is encoded in a data stream and provided to the decoder in a compact manner. At the decoding stage, this metadata can be used to recover the trimmed map and to reconstruct the contribution information of each trimmed camera.

[0108] Figure 10A 4x4 multi-view frame and an example pruning graph for such MVD frame is shown. According to the principles of the application, for each camera (i.e. views 111 to 144), a set of other cameras is determined. Each camera is associated through pruning priority relations with zero, one or several other cameras in a loop-free manner (i.e. the pruning graph obtained from the pruning priority relations does not comprise any loop). In order to have an efficient pruning, the priority relations are chosen such that two connected views have a high amount of potential redundancy. This possibility can be determined for example based on the distance between the optical centers of the two cameras of interest, their overlap ratio or the angle / distance between their optical axes. In order to obtain a loop-free graph, a 2-step strategy can be envisaged: first, connect all cameras in a dense manner according to the selected criteria of priority; and second, prune the obtained graph in a greedy manner to keep a minimum amount of connections, guaranteeing the loop-free feature. The base view (example of view 133) does not point to any other camera as the base view is not pruned. Some views (examples of 111, 114, 141 and 144) have no predecessor in the graph. Figure 10 Figure 10

[0109] During the pruning procedure, a pruning order is determined such that a camera will always be pruned after all its parent cameras in the sense of pruning priority. In the example of Fig. 1, the pruning order can be (133, 123, 132, 134, 143, 113, 122, 124, 131, 142, 144, 112, 114, 121, 141). The pruning procedure of all cameras is performed in this order. If and only if a pixel can be pruned with respect to each camera of the set it is involved in (i.e. all the reference cameras carry the same information), the pixel to prune of the camera is pruned with respect to the camera related to this camera. If a part of the parent camera set has already been pruned during the procedure, in order to avoid any drift effect, the pruning is attempted recursively with respect to its unique or multiple parent cameras until a non-pruned region has been found. If no consensus is found, the considered pixel to prune is not pruned and its value is unchanged. Otherwise, the pixel (and its value) is discarded. Each two-by-two comparison that occurs on the path of the pruning tree, there is a small alignment error in depth. While the error is below the threshold used for comparison between two close cameras (i.e. topological neighbor views), this is not the case for two remote cameras that are compared indirectly through the path of the pruning tree. The drift effect is the cumulation of small alignment errors in depth between the cameras along the path of the pruning tree. Figure 10

[0110] In order to be used at the decoding stage, according to a non-limiting embodiment of the principles of the application, the pruning graph is encoded in the data stream.

[0111] ​​​In a first embodiment, the data representing each priority relationship of the pruning graph is encoded as a list comprising for each camera a camera related list according to the syntax format depicted in Table 2, each camera being identified by its position in the camera parameter list following the syntax format proposed in Table 1. If the number of cameras is small (e.g. lower than 64), a mask / bit array can be used instead to describe the pruning priority, each ith bit being set to 1 if the ith camera has a priority relationship, e.g. following the syntax format as depicted in Table 3.

[0112]

[0113]

[0114] Table 1

[0115]

[0116] Table 2

[0117]

[0118] Table 3

[0119] In another embodiment, the pruning relationships are integrated within the camera parameter list as a new parameter for each camera (as an array or as a mask), e.g. according to the syntax format proposed in Table 4 and Table 5.

[0120]

[0121]

[0122] Table 4

[0123]

[0124] Table 5

[0125] At the decoding stage, the pruning graph is recovered from the metadata and used to correctly handle the weighting policy of the renderer. In an embodiment, for each pixel to be synthesized, the contribution of all cameras is considered iteratively. For each camera that provides an effective contribution, the fact that each camera has been pruned with respect to this camera is considered iteratively by browsing the pruning graph in the pruning order (from a parent camera towards its children cameras). If the browsed camera has been pruned with respect to the camera of interest of the considered pixel, its weight is combined (e.g. added) to the weight of the current camera and then its children cameras are similarly processed. If the browsed camera has not been pruned with respect to this camera of the considered pixel, because it keeps different effective information, the browsing stops along the associated branch of the graph and the weight of the camera of interest remains unchanged.

[0126] According to the principles of the application, the pruned camera contributions are correctly recovered at the decoder stage after pruning, preventing visual artifacts, as described in connection with Figure 9

[0127] Figure 11 A method 110 for encoding a multi-view frame in a data stream according to non-limiting embodiments of the principles of the application is shown. At step 111, an MVD frame is obtained from a source. At this step, the MVD frame needs to encode a large amount of data. At step 112, a graph linking the views of the MVD is determined according to pruning priority relationships. This graph is built to be acyclic, that is to say, no view can be pruned before the view that precedes it. Some views have no predecessor and are not meant to be pruned (also called base views) have no successor in the graph. At step 113, the pruned views according to the priority relationships of the graph as described in connection with Figure 10 At this stage, the redundant information (color and depth) of the initial MVD obtained at step 111 has been removed and the result needs to encode less data. The remaining useful information can be organized into unique frames called atlases as described in connection with Figures 5 to 7 At step 114, the pruned MVD or the corresponding atlases are encoded in the stream in association with dedicated metadata. According to the principles of the application, the pruning priority relationships of the pruning graph are also encoded in the stream, following one of the syntax formats proposed for example. In a further step, the data stream can be stored in a memory or a non-transitory storage medium or transmitted to a remote or local device through a network or a data bus.

[0128] Figure 12 ​A method 120 for decoding a pruned multi-view frame from a data stream is shown according to a non-limiting embodiment of the principles of the application. At step 121, a data stream is obtained and data representing a pruned MVD, for example in a tile format, is obtained from the data stream. The pruned MVD is decoded from the data, for example by using a video codec. At step 122, a pruning graph of the linked views of the MVD is obtained from the data stream. Steps 121 and 122 can be performed in any order or in parallel. The pruning graph is an acyclic structure of pruning priority relationships between the views of the MVD, as described in detail in the present application. At step 123, a viewport frame is generated for a viewing pose, i.e. a position and orientation in the 3D space of the renderer. For a pixel of the viewport frame, the weight of the contribution of each view, also called "camera" in the present application, is determined according to the pruning priority relationships between the views of the obtained pruning graph. For each camera providing an effective contribution, each camera that has been pruned with respect to this camera is considered iteratively by walking the pruning graph in pruning order (from a parent camera towards its child cameras). If a walked camera has been pruned with respect to the camera of interest of the considered pixel, its weight is combined, for example added, to the weight of the current camera, and then its child cameras are similarly processed. If a walked camera has not been pruned with respect to the camera of interest of the considered pixel, because it keeps a different effective information, the walk stops along the associated branch of the graph, and the weight of the camera of interest remains unchanged.

[0129] In an embodiment, at the decoding stage, the pruning graph can be used to un-prune pruned input views. According to the principles of the application, each source view of a received pruned MVD is reconstructed by restoring the missing redundant parts suppressed by the pruning process. To this end, an inverse procedure is applied. Starting from the root node to the leaves, the effective (non-pruned) pixel p of the view associated with the node N is considered. Then:

[0130] 1) The pixel p is de-projected onto the (still "un-pruned") view associated with a sub-view of its view, and the associated de-projected pixel state is retrieved if it contributes to its viewport.

[0131] 2) If the de-projected pixel is identified as pruned (and still has no valid value), its color and depth values are set to one of the ones of the pixel p (color and / or depth), and the procedure is iteratively repeated to the sub-views of the following view.

[0132] 3) If the de-projected pixel is identified as non-pruned (and has a valid value), its color and depth values remain unchanged, and no more check of the graph is made for the sub-views of this following view.

[0133] 4) If the pixel p does not fall within the viewport of one of its sub-views, the procedure is recursively repeated to the grand-children views.

[0134] This could feed to a multi-view display that needs to show all views of the MVD content at all times (not just a composite virtual view in the HMD), while transmitting trimmed content at a reduced bit rate.

[0135] The specific implementations described herein may be implemented, for example, in methods or processes, apparatus, computer program products, data streams, or signals. Even if discussed only in the context of a single form of implementation (e.g., discussed only as a method or apparatus), the specific implementations of the discussed features may also be implemented in other forms (e.g., programs). Apparatus may be implemented, for example, in suitable hardware, software, and firmware. Methods may be implemented in apparatus (such as, for example, a processor) that generally refers to a processing device, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices, such as, for example, smartphones, tablets, computers, mobile phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate communication of information between end users.

[0136] Specific implementations of the various processes and features described herein can be found in a wide variety of devices or applications, particularly those associated with data encoding, data decoding, view generation, texture processing, and other processing of images and related texture and / or depth information. Examples of such devices include encoders, decoders, post-processors that process the output from decoders, pre-processors that provide input to encoders, video encoders, video decoders, video codecs, web servers, set-top boxes, laptops, personal computers, cellular phones, PDAs, and other communication devices. It should be understood that the devices can be mobile, even mounted in mobile vehicles.

[0137] Additionally, the method can be implemented by instructions executed by a processor, and such instructions (and / or data values ​​generated by the implementation) can be stored on a processor-readable medium, such as, for example, an integrated circuit, a software carrier, or other storage device, such as, for example, a hard disk, a compact disk (“CD”), an optical disk (such as, for example, a DVD, commonly referred to as a digital versatile optical disk or digital video optical disk), random access memory (“RAM”), or read-only memory (“ROM”). Instructions can form an application program tangibly embodied on the processor-readable medium. Instructions can be, for example, hardware, firmware, software, or a combination thereof. Instructions can be found, for example, in an operating system, a standalone application, or a combination of both. Thus, a processor can be characterized, for example, as a device configured to execute a process and a device comprising a processor-readable medium (such as a storage device) having instructions for executing the process. Furthermore, in addition to or instead of instructions, the processor-readable medium can store data values ​​generated by the implementation.

[0138] It will be apparent to those skilled in the art that the specific implementations can produce various signals formatted to carry information that can be, for example, stored or transmitted. The information can include, for example, instructions for performing a method, or data created by one of the specific implementations. For example, a signal can be formatted to carry as data the rules for writing or reading the syntax of a described implementation, or to carry as data the actual syntax-values written by a described implementation. Such a signal can be formatted, for example, as an electromagnetic wave (for example, using a portion of the spectrum that is visible, or that is not visible to humans) or a baseband signal. The formatting can include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information that the signal carries can be, for example, analog or digital information. The signal can be transmitted over a variety of different wired or wireless links, as is known. The signal can be stored on a processor-readable medium.

[0139] A number of implementations have been described. Nevertheless, it will be understood that various modifications can be made. For example, elements of different implementations can be combined, supplemented, modified, or removed to produce other implementations. Additionally, one of ordinary skill will understand that other structures and processes can be substituted for those disclosed and the resulting implementations will perform at least substantially the same function(s) in at least substantially the same way(s) to achieve at least substantially the same result(s). Accordingly, these and other implementations are contemplated by this application.

Claims

1. A method for encoding views of a multiview frame in a data stream, the method comprising: - obtaining a graph without loops linking views of the multiview frame, a link of the graph representing a pruning priority relationship, at least one base view of the multiview frame having no pruning priority link; - pruning pixels of the views of the multiview frame in an order determined so that a given view is pruned after views linked to the given view by a pruning priority link to remove redundant information, wherein a pixel of the given view is pruned when it corresponds to information encoded in pixels of a view transitively linked to the given view by a pruning priority link, and - encoding the graph without loops, the at least one base view and the pruned views in the data stream.

2. The method of claim 1, wherein pruning a pixel of a view comprises replacing the value of the pixel by a determined value.

3. The method of claim 1 or 2, wherein the graph without loops is signaled in the data stream as a list comprising, for each view of the multiview frame, a list of views linked to the view.

4. A device for encoding views of a multiview frame in a data stream, the device comprising a processor configured for: - obtaining a graph without loops linking views of the multiview frame, a link of the graph representing a pruning priority relationship, at least one base view of the multiview frame having no pruning priority link; - pruning pixels of the views of the multiview frame in an order determined so that a given view is pruned after views linked to the given view by a pruning priority link to remove redundant information, wherein a pixel of the given view is pruned when it corresponds to information encoded in pixels of a view transitively linked to the given view by a pruning priority link, and - encoding the graph, the at least one base view and the pruned views in the data stream.

5. The device of claim 4, wherein pruning a pixel of a view comprises replacing the value of the pixel by a determined value.

6. The device of claim 4 or 5, wherein the graph is signaled in the data stream as a list comprising, for each view of the multiview frame, a list of views linked to the view.

7. A method for decoding views of a multiview frame from a data stream, the method comprising: - obtaining the views of the multiview frame from the data stream, at least one base view being unpruned, other views being pruned to remove redundant information; - obtaining a graph without loops linking views of the multiview frame from the data stream, a link of the graph representing a pruning priority relationship, the at least one base view of the multiview frame having no pruning priority link; - generating a viewport frame from a viewing pose by determining a contribution of each view of the multiview frame according to the pruning priority relationship of the graph without loops.

8. The method of claim 7, wherein pruned pixels of a pruned view have a determined value.

9. The method of claim 7 or 8, wherein the acyclic graph is signaled in the data stream as a list comprising for each view of the multi-view frame a list of views linked to the view.

10. A device for decoding views of a multi-view frame from a data stream, the device comprising a processor configured for: - obtaining from the data stream the views of the multi-view frame, at least one base view being unpruned, other views being pruned to remove redundant information; - obtaining from the data stream an acyclic graph linking views of the multi-view frame, a link of a graph representing a view pruning priority relationship, the at least one base view of the multi-view frame having no pruning priority link; - generating a viewport frame from a viewing pose by determining a contribution of each view of the multi-view frame according to the pruning priority relationship of the acyclic graph.

11. The device of claim 10, wherein pruned pixels of a pruned view have a determined value.

12. The device of claim 10 or 11, wherein the acyclic graph is signaled in the data stream as a list comprising for each view of the multi-view frame a list of views linked to the view.

13. A processor readable medium having stored thereon a code stream formed by instructions executable by a processor for performing the method of any one of claims 1-3 for encoding views of a multi-view frame in a data stream, resulting in the code stream.

Images