Method and apparatus for encoding, transmitting and decoding volumetric video

Abstract

Methods, devices and streams for encoding a depth map set representing the geometry of volumetric video are disclosed. The views to be encoded are analyzed to detect regions in the views having simple depth or color, i.e. regions where the local variance of the depth or color is below a given threshold. The resolution of such regions is reduced and the map set comprises first regions at full resolution and second regions at reduced resolution. Metadata indicate whether a tile is a reduced region and, if so, the reduction factor is associated with the map set in the data stream. A decoder uses these metadata to compose the views from the different tiles.

Description

1. TECHNICAL FIELD

[0002] The present principles generally relate to the domain of three-dimensional (3D) scene and volumetric video content. The present document is also understood in the context of encoding, formatting and decoding data representing textures and geometry of 3D scenes, to render volumetric content on end-user devices such as mobile devices or head-mounted displays (HMDs). 2. BACKGROUND

[0004] This section is intended to introduce the reader to various aspects of art that can be related to various aspects of the present principles that are described and / or claimed below. This discussion is believed to be helpful in providing the reader with background information to facilitate a better understanding of the various aspects of the present principles. Accordingly, it should be understood that these statements are to be read in this light, and not as admissions of prior art.

[0005] Recently, there has been a growth in available large field of view content (up to 360°). A user watching content on an immersive display device (such as a head-mounted display, smart glasses, PC screen, tablet, smartphone, etc.) can not be able to see the whole of such content. This means that at a given moment, the user can only watch a part of the content. However, the user can usually navigate within the content through various means such as head movement, mouse movement, touch screen, voice, and the like. It is generally desirable to encode and decode such content.

[0006] Immersive video (also called 360° planar video) allows a user to watch everything around him by rotating his head around a static viewpoint. The rotation only allows a 3 degrees of freedom (3DoF) experience. Even if 3DoF video is enough to satisfy the requirements of a first omnidirectional video experience (e.g. using a head-mounted display (HMD device)), 3DoF video can quickly become frustrating for a viewer who expects more freedom (e.g. by experiencing parallax). Moreover, 3DoF can also cause dizziness because a user never only rotates his head but also translates his head in three directions, which are not reproduced in a 3DoF video experience.

[0007] Among others, the large field of view content can be a three-dimensional computer graphics image scene (3D CGI scene), a point cloud or an immersive video. Many terms can be used to design such immersive video: for example, virtual reality (VR), 360, panoramic, 4π steradians, immersive, omnidirectional or large field of view.

[0008] Volume videos (also called 6 Degrees of Freedom (6DoF) videos) are an alternative to 3DoF videos. When watching a 6DoF video, in addition to rotations, the user can translate his head, and even his body, in the content of the watch, and experience parallax and even volume. This video significantly increases the immersion and the perception of the depth of the scene and prevents dizziness by providing a consistent visual feedback during head translation. The content is created with dedicated sensors that allow the simultaneous recording of the color and the depth of the scene of interest. Even if technical difficulties still exist, the use of a color camera equipment combined with photogrammetry techniques is one way to perform this recording.

[0009] While 3DoF videos consist of a sequence of images resulting from the de-mapping of texture images (for example, spherical images encoded according to a latitude / longitude projection mapping or an equirectangular projection mapping), 6DoF video frames embed information from multiple viewpoints. They can be seen as a temporal sequence of point clouds resulting from a three-dimensional capture. Two kinds of volume videos can be considered depending on the viewing conditions. The first one, i.e. full 6DoF, allows a full freedom of navigation within the video content, while the second one, also called 3DoF+, restricts the user viewing space to a limited volume called viewing bounding box, allowing a limited head translation and parallax experience. This second case is a valuable compromise between free navigation and passive viewing conditions for seated audiences.

[0010] 3DoF+ content can be set as a set of multi-view+depth (MVD) frames. A 3DoF+ encoder takes as input a multi-view+depth video, removes inter-view redundancies (this step is called a "pruning process), and packs the selected color patches and depth patches (i.e. the information remaining after pruning) into a plurality of atlases (i.e. images that gather pictures extracted from the pruned MVD). The bitstream is composed of a plurality of streams (for example HEVC streams) carrying the atlases of texture (i.e. color) patches and depth patches, accompanied by metadata describing the camera parameters of the input views and the atlases layout. In a known format, the patch atlases are composed of pairs of texture and depth atlas components, with the same picture size and the same layout (same packing) for texture and depth. Such a representation with the same packing and the same spatial resolution for texture patches and depth patches is suboptimal. The geometry of 3D scenes appears as large areas with flat or low varying depth and / or texture. There is a lack of a solution that exploits this feature of volume videos. 3. SUMMARY

[0012] The following presents a simplified summary of the principles of the application in order to provide a basic understanding of some aspects of the application. This summary is not an extensive overview of the principles of the application. It is not intended to identify key or critical elements of the principles of the application. The following summary merely presents some aspects of the principles of the application in a simplified form as a prelude to the more detailed description provided below.

[0013] The principles of the present invention relate to a method for encoding a view in an atlas, the method comprising:

[0014] - dividing said view into a first region and a second region, the second region being a candidate for down-sampling;

[0015] - reducing the resolution of the second region by a factor;

[0016] - packing the first region and the reduced second region in said atlas in association with metadata indicating whether a tile is a first region or a reduced second region, and if a reduced second region, a factor used to reduce the resolution;

[0017] The principles of the present invention also relate to a device comprising a processor configured to implement the method.

[0018] The principles of the present invention also relate to a method for decoding a view in an atlas comprising tiles, the method comprising:

[0019] - obtaining metadata indicating whether a tile is a first tile or a second tile, and if a second tile, a reduction factor;

[0020] - upscaling the resolution of the second tile according to the associated reduction factor; and

[0021] composing said view from said first tile and the upscaled second tile

[0022] The principles of the present invention also relate to a device comprising a processor configured to implement the method. 4. BRIEF DESCRIPTION OF DRAWINGS

[0024] The present disclosure will be better understood and other specific features and advantages will emerge upon reading the following description, the description being given with reference to the drawings in which:

[0025] - Figure 1 A three-dimensional (3D) model of an object and points of a point cloud corresponding to the 3D model are shown, according to a non-limiting embodiment of the principles of the present invention;

[0026] - Figure 2 Non-limiting examples of encoding, transmitting and decoding data representing a sequence of 3D scenes are shown, according to a non-limiting embodiment of the principles of the present invention;

[0027] - Figure 3 An exemplary architecture of a device that can be configured to implement the methods described in relation to Figure 13 and Figure 14 the principles of the present invention is shown, according to a non-limiting embodiment of the principles of the present invention;

[0028] - Figure 4 Examples of embodiments of the syntax of a stream when transmitting data via a packet-based transport protocol are shown, according to a non-limiting embodiment of the principles of the present invention;

[0029] - Figure 5 A block atlas method with four projection centers is shown as an example of a non-limiting embodiment based on the principles of the present invention;

[0030] - Figure 6 An example of a texture atlas including points of a 3D scene, according to a non-limiting embodiment of the principles of the present invention, is shown;

[0031] - Figure 7 The illustration shows a non-limiting embodiment based on the principles of the invention, including... Figure 6 An example of a 3D scene atlas containing depth information of points;

[0032] - Figure 8 A non-limiting embodiment of the present invention, based on the characteristics of depth information, is shown, which divides a basic view into regions.

[0033] - Figure 9 A non-limiting embodiment of the invention is shown for use with respect to, including, [the following]. Figure 8 An example color component of the atlas frame encoding the basic view 80 of the 3D scene;

[0034] - Figure 10 A non-limiting embodiment of the invention is shown for use with respect to, including, [the following]. Figure 8 An exemplary depth component of the atlas frame encoding the basic view 80 of the 3D scene;

[0035] - Figure 11 Conventional downsampling in the horizontal and vertical directions is shown in a non-limiting embodiment according to the principles of the present invention;

[0036] - Figure 12 An example of bilinear upsampling of blocks transmitted at different resolutions according to a non-limiting embodiment of the principles of the present invention is shown;

[0037] - Figure 13 A method for encoding a depth view in a depth-blocked atlas is shown according to a non-limiting embodiment of the principles of the present invention;

[0038] - Figure 14 A method for decoding a depth view in a depth-blocked atlas is shown according to a non-limiting embodiment of the principles of the present invention. 5. Detailed Implementation

[0040] The principles of the present application will be more fully understood in view of the detailed description and the drawings attached hereto. The principles of the present application can be embodied in many alternative forms and should not be construed as limited to the examples set forth herein. Accordingly, while the principles of the present application are susceptible to various modifications and alternative forms, specific examples thereof are shown by way of example in the drawings and will be described herein in detail. It should be understood, however, that there is no intent to limit the principles of the present application to the particular forms disclosed, but on the contrary, the intention is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the principles of the present application as defined by the claims.

[0041] The terminology used herein is for the purpose of describing particular examples only and is not intended to be limiting of the principles of the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Additionally, when an element is referred to as being "responsive" or "connected" to another element, it can be directly responsive or connected to the other element, or indirectly responsive or connected to the other element through one or more other elements. In contrast, when an element is referred to as being "directly responsive" or "directly connected" to another element, there are no intervening elements. As used herein the term "and / or" includes any and all combinations of one or more of the associated items and can be abbreviated as " / ".

[0042] It should be understood that, although the terms first, second, etc. can be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element, without departing from the teachings of the present principles.

[0043] Although some of the diagrams include arrows on communication paths to show a primary direction of communication, it is to be understood that communication can occur in the opposite direction to the depicted arrows.

[0044] Some examples are described with respect to block and operational flow diagrams that include blocks interconnecting with other blocks representing circuit elements, modules or code portions for performing specified functions. It should also be noted that in other specific implementations, the function(s) noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved.

[0045] Reference in this document to "one example" or "an example" means that a particular feature, structure, or characteristic described in connection with the example is included in at least one implementation of the present principles. The appearances of the phrase "one example" or "an example" in various places in the specification are not necessarily all referring to the same example, nor are they necessarily mutually exclusive, or alternative examples to one another.

[0046] Reference signs in the claims are presented by way of illustration only and should not be construed as limiting the scope of the claims. Although not explicitly described, the present examples and variants can be employed in any combination or sub-combination.

[0047] Figure 1 A three-dimensional (3D) model 10 of an object and points of a point cloud 11 corresponding to the 3D model 10 are shown. The 3D model 10 and the point cloud 11 can for example correspond to a possible 3D representation of an object of a 3D scene comprising other objects. The model 10 can be a 3D mesh representation and the points of the point cloud 11 can be the vertices of the mesh. The points of the point cloud 11 can also be points distributed on the surface of the faces of the mesh. The model 10 can also be represented as a splatted version of the point cloud 11, the surface of the model 10 being created by splatting the points of the point cloud 11. The model 10 can be represented by many different representations such as voxels or splines. Figure 1 The fact that a point cloud can be defined with a surface representation of a 3D object and that a surface representation of a 3D object can be generated from a cloud of points is shown. As used herein, projecting points of a 3D object (by extension points of a 3D scene) onto an image is equivalent to projecting any representation of this 3D object, for example a point cloud, a mesh, a spline model or a voxel model.

[0048] A point cloud can be represented in memory as for example a vector-based structure where each point has its own coordinates (for example three-dimensional coordinates XYZ, or an angle in solid and a distance from / to a viewpoint (also called depth) and one or more attributes, also called components, in a frame of reference of a viewpoint. One example of a component is a color component which can be represented in various color spaces such as RGB (red, green and blue) or YUV (Y is the luminance component and UV are two chrominance components). The point cloud is a representation of a 3D scene comprising an object. The 3D scene can be seen from a given viewpoint or range of viewpoints. The point cloud can be obtained in many ways such as:

[0049] • from a capture of a real object taken by a camera rig, optionally complemented with depth active sensing devices;

[0050] • from a capture of a virtual / synthetic object taken by a virtual camera rig in a modeling tool;

[0051] • from a mix of both real and virtual objects.

[0052] 3D scenes, especially when produced for 3DoF+ rendering, can be represented by multi-view+depth (MVD) frames. Then, volumetric videos are sequences of MVD frames. In this approach, volumetric information is delivered as a combination of color and depth patches stored in corresponding color and depth atlases, which are then video encoded with regular codecs, typically HEVC. Each combination of color and depth patches represents a sub-portion of the MVD input view and the set of all patches is designed at the encoding stage to cover the whole scene while there is as little redundancy as possible. At the decoding stage, the atlases are first video decoded and the patches are rendered in a view synthesis process to recover the viewport associated with the desired viewing position.

[0053] Figure 2 Non-limiting examples of encoding, transmitting and decoding data representing a sequence of 3D scenes are illustrated. The encoding format can be compatible with 3DoF, 3DoF+ and 6DoF decoding, for example, at the same time.

[0054] A sequence of 3D scenes 20 is obtained. As a sequence of pictures is a 2D video, a sequence of 3D scenes is a 3D (also called volumetric) video. The sequence of 3D scenes can be provided to a volumetric video rendering device for 3DoF, 3Dof+ or 6DoF rendering and display.

[0055] The sequence of 3D scenes 20 can be provided to an encoder 21. The encoder 21 takes as input one 3D scene or a sequence of 3D scenes and provides a bitstream representing the input. The bitstream can be stored in a memory 22 and / or on an electronic data medium and can be transmitted over a network 22. The bitstream representing the sequence of 3D scenes can be read from the memory 22 and / or received from the network 22 by a decoder 23. The decoder 23 takes as input the bitstream and provides a sequence of 3D scenes in a point cloud format, for example.

[0056] The encoder 21 can comprise several circuits implementing several steps. In a first step, the encoder 21 projects each 3D scene onto at least one 2D picture. A 3D projection is any method of mapping three-dimensional points into a two-dimensional plane. Since most current methods for displaying graphical data are based on planar (pixel information from several bitplanes) two-dimensional media, the use of this type of projection is widespread, especially in computer graphics, engineering, and cartography. A projection circuit 211 provides a sequence of 3D scenes 20 with at least one two-dimensional frame 2111. The frame 2111 comprises color information and depth information representing the 3D scene projected onto the frame 2111. In a variant, the color information and the depth information are encoded in two separate frames 2111 and 2112.

[0057] The metadata 212 is used and updated by the projection circuit 211. The metadata 212 comprises information on the projection operation (e.g. projection parameters) and information on the way color and depth information is organized within the frames 2111 and 2112, as explained in connection with Figures 5 to 7

[0058] The video encoding circuit 213 encodes the sequence of frames 2111 and 2112 into a video. The pictures 2111 and 2112 of the 3D scene (or the sequence of pictures of the 3D scene) are encoded by the video encoder 213 in a stream. The video data and the metadata 212 are then encapsulated by the data encapsulation circuit 214 in a data stream.

[0059] The encoder 213 is for example compatible with encoders such as:

[0060] - JPEG, specification ISO / CEI 10918-1 UIT-T Recommendation T.81 ;

[0061] - AVC, also known as MPEG-4 AVC or h264. Specified in both UIT-T H.264 and ISO / CEI MPEG-4 Part 10 (ISO / CEI 14496-10), HEVC (whose specification is found on the ITU website, T recommendation, H series, h265);

[0062] - 3D-HEVC (extension of HEVC, whose specification is found on the ITU website, T recommendation, H series, h265, annex G and I);

[0063] - VP9 developed by Google;

[0064] - AV1 (AOMedia Video 1) developed by the Alliance for Open Media; or

[0065] - future standards such as Versatile Video Coding or MPEG-I or future versions of MPEG-V.

[0066] The data stream is stored in a memory accessible by the decoder 23, for example through the network 22. The decoder 23 comprises different circuits implementing different decoding steps. The decoder 23 takes as input the data stream generated by the encoder 21 and provides a sequence of 3D scenes 24 to be rendered and displayed by a volumetric video display device such as a head-mounted device (HMD). The decoder 23 obtains the stream from the source 22. For example, the source 22 belongs to a group comprising:

[0067] ​- local memory, such as a video memory or a RAM (or Random Access Memory), a flash memory, a ROM (or Read Only Memory), a hard disk;

[0068] - a storage interface, such as an interface with a mass storage device, a RAM, a flash memory, a ROM, an optical disk or a magnetic support;

[0069] - a communication interface, such as a wired interface (for example a bus interface, a wide area network interface, a local area network interface) or a wireless interface (such as an IEEE 802.11 interface or a Bluetooth ® interface); and

[0070] - a user interface enabling a user to input data, such as a graphical user interface.

[0071] The decoder 23 comprises a circuit 234 for extracting the data encoded in the data stream. The circuit 234 takes as input the data stream and provides the metadata 232 corresponding to the metadata 212 encoded in the stream and a two-dimensional video. The video is decoded by a video decoder 233 providing a sequence of frames. The decoded frames comprise color and depth information. In a variant, the video decoder 233 provides two sequences of frames, one containing color information and the other containing depth information. The circuit 231 uses the metadata 232 to project the color and depth information from the decoded frames to provide a sequence of 3D scenes 24. The sequence of 3D scenes 24 corresponds to the sequence of 3D scenes 20, possibly with a loss of precision related to the encoding as a 2D video and to the video compression.

[0072] Figure 3 An exemplary architecture of a device 30 that can be configured to implement the method described with respect to Figure 13 and Figure 14 is shown. Figure 2 The encoder 21 and / or the decoder 23 can implement this architecture. Alternatively, each circuit of the encoder 21 and / or of the decoder 23 can be a device according to the architecture of Figure 3 , linked together, for example via their buses 31 and / or via their I / O interfaces 36.

[0073] The device 30 comprises the following elements connected together by data and address buses 31 :

[0074] - a microprocessor 32 (or CPU), which is for example a DSP (or Digital Signal Processor);

[0075] - a ROM (or Read Only Memory) 33;

[0076] - a RAM (or Random Access Memory) 34;

[0077] - a storage interface 35;

[0078] - an I / O interface 36 for receiving data to be transmitted from an application; and

[0079] - a power supply, for example a battery.

[0080] According to one example, the power supply is external to the device. In each of the mentioned memories, the word "register" used in the description can correspond to an area of small capacity (a few bits) or to a very large area (for example, the entire program or a large amount of received or decoded data). The ROM 33 comprises at least the program and the parameters. The ROM 33 can store, according to the principles of the application, algorithms and instructions for performing the techniques. When switched on, the CPU 32 uploads the program in the RAM and executes the corresponding instructions.

[0081] The RAM 34 comprises the program in the registers executed by the CPU 32 and uploaded after the switch-on of the device 30, the input data in the registers, the intermediate data in the different states of the method in the registers, and other variables for executing the method in the registers.

[0082] The specific implementations described herein can be implemented in, for example, a method or process, an apparatus, a computer program product, a data stream, or a signal. Even if discussed in the context of a single form of implementation (for example, discussed only as a method or as an apparatus), the implementation of the discussed features can also be implemented in other forms (for example, a program). An apparatus can be implemented in, for example, appropriate hardware, software, and firmware. The method can be implemented in, for example, an apparatus that is generally referred to as a processing device, such as, for example, a processor, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes communication devices, such as, for example, a computer, a cell phone, a portable / personal digital assistant ("PDA"), and other devices that facilitate the communication of information between end users.

[0083] According to an example, the device 30 is configured to implement the method described in relation to Figure 13 and Figure 14 The described method belongs to the set comprising:

[0084] - a mobile device;

[0085] - a communication device;

[0086] - a gaming device;

[0087] - a tablet (or tablet computer);

[0088] - a laptop;

[0089] - a still picture camera;

[0090] - a video camera;

[0091] - an encoding chip;

[0092] - a server (e.g. a broadcast server, a video on demand server or a web server).

[0093] Figure 4 An example of an embodiment of the syntax of a stream is shown when data is transmitted through a packet-based transmission protocol. Figure 4 An exemplary structure 4 of a volumetric video stream is shown. The structure is contained in a container that organizes the stream in independent elements of syntax. The structure can comprise a header portion 41 that is a set of data common to each syntax element of the stream. For example, the header portion comprises some metadata about the syntax elements, describing the nature and the role of each of them. The header portion can also comprise Figure 2 a part of the metadata 212, e.g. the coordinates of the center viewpoint used to project the points of the 3D scene onto the frames 2111 and 2112. The structure comprises a payload that comprises syntax elements 42 and at least one syntax element 43. The syntax elements 42 comprise data representing color and depth frames. The images can have been compressed according to a video compression method.

[0094] The syntax element 43 is part of the payload of the data stream and can comprise metadata about how the frames of the syntax elements 42 are encoded, e.g. parameters used to project and pack points of the 3D scene onto the frames. Such metadata can be associated with each frame or group of frames of the video, also called group of pictures (GoP) in video compression standards.

[0095] Figure 5 A tiling atlas approach is shown as an example with 4 projection centers. The 3D scene 50 comprises a person. For example, the projection center 51 is a perspective camera and the camera 53 is an orthographic camera. The cameras can also be omnidirectional cameras with e.g. spherical mapping (e.g. equirectangular mapping) or cubic mapping. According to the projection operations described in the projection data of the metadata, the 3D points of the 3D scene are projected onto 2D planes associated with virtual cameras located at the projection centers. In the example, the projection of the points captured by the camera 51 is mapped onto the tile 52 according to a perspective mapping and the projection of the points captured by the camera 53 is mapped onto the tile 54 according to an orthographic mapping. Figure 5

[0096] The clustering of the projected pixels produces a plurality of 2D tiles that are packed in a rectangular atlas 55. The organization of the tiles within the atlas defines an atlas layout. In an embodiment, there are two atlases with the same layout: one for texture (i.e. color) information and one for depth information. Two tiles captured by the same camera or by two different cameras can comprise information representative of the same part of the 3D scene, like e.g. tiles 54 and 56.

[0097] ​The packing operation produces patch data for each generated patch. The patch data includes a reference to the projected data (e.g. an index in a table of projected data or a pointer to the projected data (i.e. an address in memory or in a data stream)) and information describing the position and size of the patch within the atlas (e.g. top-left coordinates, size and width in pixels). The patch data items are added to the metadata to be encapsulated in the data stream in association with the compressed data of one or two atlases.

[0098] Figure 6 An example of an atlas 60 comprising texture information (e.g. RGB data or YUV data) of points of a 3D scene is shown according to a non-limiting embodiment of the present principles. As explained in connection with Figure 5 The atlas is an image packed patch, the patch being a picture obtained by projecting a portion of points of the 3D scene.

[0099] In the example of Figure 6 , the atlas 60 comprises a first portion 61 comprising texture information of points of the 3D scene visible from the viewpoint and one or more second portions 62. The texture information of the first portion 61 can be obtained for example according to an equirectangular projection mapping, which is an example of a spherical projection mapping. In the example of Figure 6 , the second portions 62 are arranged at the left and right borders of the first portion 61, but the second portions can be arranged differently. The second portions 62 comprise texture information of portions of the 3D scene complementary to the portion visible from the viewpoint. The second portions can be obtained by removing from the 3D scene the points visible from the first viewpoint (whose texture is stored in the first portion) and projecting the remaining points according to the same viewpoint. The latter process can be iteratively repeated to obtain each time a hidden portion of the 3D scene. According to a variant, the second portions can be obtained by removing from the 3D scene the points visible from a viewpoint (e.g. a central viewpoint) (whose texture is stored in the first portion) and projecting the remaining points according to a viewpoint different from the first viewpoint, for example one or more second viewpoints from a viewing space (e.g. a viewing space for 3DoF rendering) centered on the central viewpoint.

[0100] The first portion 61 can be seen as a first large texture patch (corresponding to the first portion of the 3D scene) and the second portions 62 comprise smaller texture patches (corresponding to the second portions of the 3D scene complementary to the first portion). Such an atlas has the advantage of being compatible with both 3DoF rendering (when only the first portion 61 is rendered) and 3DoF+ / 6DoF rendering.

[0101] Figure 7 An example of an atlas 70 comprising depth information of points of a 3D scene is shown according to a non-limiting embodiment of the present principles. The atlas 70 can be seen as corresponding to Figure 6 Figure 6 ​depth image of the texture image 60.

[0102] The atlas 70 comprises a first part 71 comprising depth information of points of the 3D scene visible from the central viewpoint and one or more second parts 72. The atlas 70 can be obtained in the same way as the atlas 60, but contains depth information associated with points of the 3D scene instead of texture information.

[0103] For 3DoF rendering of a 3D scene, only one viewpoint is considered, typically the central viewpoint. The user can rotate his head in three degrees of freedom around the first viewpoint to watch various parts of the 3D scene, but the user cannot move this unique viewpoint. The points of the scene to be encoded are the points visible from this unique viewpoint, and only the texture information needs to be encoded / decoded for 3DoF rendering. For 3DoF rendering, the points of the scene that are not visible from this unique viewpoint do not need to be encoded, as they cannot be accessed by the user.

[0104] For 6DoF rendering, the user can move the viewpoint around in the scene. In this case, every point of the scene (depth and texture) needs to be encoded in the bitstream, as the user who can move his viewpoint can access every point. At the encoding stage, there is no means to know a priori from which viewpoint the user will observe the 3D scene.

[0105] For 3DoF+ rendering, the user can move the viewpoint within a limited space around the central viewpoint. This enables the experience of parallax. The data representing the parts of the scene visible from any point of the viewing space will be encoded into the stream, including the data representing the 3D scene according to the central viewpoint, i.e. the first parts 61 and 71. For example, the size and shape of the viewing space can be decided and determined at the encoding step and encoded in the bitstream. The decoder can obtain this information from the bitstream and the renderer will limit the viewing space to the space determined by the obtained information. According to another example, the renderer determines the viewing space according to hardware constraints, for example hardware constraints related to the capabilities of the sensors detecting the user movements. In this case, if at the encoding stage, a point visible from a point within the viewing space of the renderer has not been encoded in the bitstream, this point will not be rendered. According to yet another example, the data representing every point of the 3D scene, e.g. the texture and / or the geometry, is encoded in the stream, without considering the rendering viewing space. To optimize the size of the stream, only a subset of the points of the scene can be encoded, e.g. a subset of the points that can be seen according to the rendering viewing space.

[0106] The tiles are created to be sufficiently non-redundant and complementary. The process for generating the tiles from a multi-view+depth (MVD) representation of a 3D scene includes "pruning" the input source views to remove any redundant information. To this end, the pruning is iterated one input view (color+depth) after the other. A set of unpruned views (called base views) is first selected among the source views and transmitted completely. Then, a set of remaining views (called additional views) is iterated to remove information redundant with the base views and the additional views already pruned (in terms of color and depth similarity). The color values or depth values of the pruned pixels are replaced by a predetermined value, e.g. 0 or 255. For example in Figure 6 and Figure 7 , the central views 61 and 71 are base views. In other embodiments, the atlas can capture several base views.

[0107] Figure 8 The base view 80 is shown split into regions according to the characteristics of the depth information. The geometry of the scene appears as large regions whose depth can be recovered from sparse spatial sampling with at least significantly lower spatial resolution than the color components of the same region. In the example of Figure 8 , the regions 81 to 86 are geometrically simple, e.g. composed of large flat surfaces. The depth of these parts does not need to be described with the same spatial resolution as the central parts 87 to 89 of the base view, which include volumes with complex geometry such as a person. A transmission format for immersive video with separate packing for texture tiles and depth tiles is desired, allowing selective down-sampling of the depth components of the atlas with respect to the color components of some tiles. Such a format enables to design coding-decoding schemes with reduced pixel rate, especially for large field of view content, while yielding the same bit-rate to distortion performance.

[0108] Figure 9 An exemplary color component of an atlas framework for encoding a 3D scene including Figure 8 the base view 80 is shown. The color component of the atlas framework includes the base view 80 as a whole. Although it corresponds to multiple texture tiles 81 to 89, these tiles are seamlessly arranged side by side in the texture atlas so as to form the base view 80 itself. Other tiles 90 generated from additional views (i.e. views that have been pruned to remove redundant information) are packed into the texture atlas (i.e. the color component of the atlas framework). Corresponding metadata is encoded in the stream in association with the texture atlas.

[0109] Figure 10 An exemplary color component of an atlas framework for encoding a 3D scene including Figure 8An exemplary depth component of the atlas frame encoding the 3D scene of the basic view 80. The encoder is responsible for taking the 3D scene, for example, represented by multiple texture + depth inputs, and converting it into another composition of texture + depth blocks with reduced bit rate and pixel rate. For the encoder, for example... Figure 2 The encoder 21 has the advantage of selecting or regenerating a very large portion of the visual scene as a complete full view, rather than breaking the scene down into small pieces packed within an atlas. The encoder's operating mode is to generate a central full view or reuse an existing view from the MVD, referred to as a "base view." In another operating mode, the encoder generates multiple full views or near-full views, i.e., multiple base views. The principles of this invention are described using a single base view. However, it can be simply summarized as multiple base views.

[0110] The basic view is analyzed by the encoder to select rectangular regions 81 to 86, where depth can be described more economically from the perspectives of transmission bit rate and pixel rate. This depth atlas is made smaller than the texture atlas by reducing the size of the depth blocks within it. According to the principles of the invention, three steps are implemented:

[0111] - Split the base view into rectangular block candidates for depth downsampling ( Figure 8 (81 to 86 in the middle) and blocks requiring full depth resolution ( Figure 8 (87 to 89 in the text). Candidates for spatial depth downsampling do not include "invalid" pixels, i.e., pixels for which depth information is not defined.

[0112] - Downsample depth blocks that do not carry complex geometric information, while maintaining the resolution of depth blocks that carry complex geometric information;

[0113] - A list of all blocks in the packing depth atlas, which helps to reduce the size of many blocks;

[0114] The segmentation process involves a restricted selection of image regions. Such methods are well-known in the prior art. In one implementation (also known as the quadtree method), the method for selecting blocks suitable for depth downsampling involves recursively segmenting the base view into four quadrants and testing whether each quadrant is a flat surface, allowing for spatial downsampling of its depth map without information loss. The pseudocode for such a method could be:

[0115] Step 1: The initial block is the basic view

[0116] Step 2: Determine the histogram of depth values

[0117] Step 3: If the peak value of the histogram is higher than the given threshold T1

[0118] STOP: this patch is a candidate for depth downsampling (flat surface)

[0119] ELSE

[0120] Step 4: determine the normal in each point of the patch

[0121] Step 5: determine the average normal

[0122] Step 6: determine the histogram of angular distances using the average normal

[0123] Step 7: if the histogram peak is higher than a given threshold T2

[0124] STOP: this patch is a candidate for depth downsampling (flat surface)

[0125] ELSE

[0126] Step 8: split the patch into 4 quadrants

[0127] Step 9: go to step 2 for each quadrant

[0128] The following notes apply:

[0129] - the given maximum number of splits can be an additional stop condition (i.e. a minimum patch size condition);

[0130] - steps 4 to 7 are omitted in a simplified version (only testing frontal parallel depth planes).

[0131] In the example of Figure 10 , the base view is split into nine regions 81 to 89 as illustrated in Figure 8 . For example, on the one hand, regions 87 and 88 are identified as having a complex geometry and, for this reason, are kept at full resolution. Region 87 is packed as patch 107 and region 88 is packed as patch 108 in the depth atlas. In the depth atlas, the patches relative to the regions of the base view are not necessarily arranged side by side. The corresponding metadata are stored in the stream in association with the depth atlas. On the other hand, for example, regions 81, 82 and 86 are identified as candidates for downsampling by the splitting process. Thus, patch 101 is generated by reducing the spatial resolution of region 81 and packed in the depth atlas. Similarly, patch 102 is generated for region 82 and patch 106 for region 86. The metadata comprising the position information of the patches in the atlas, the views in which the patches are generated and the parameters of the corresponding resolution reduction are stored in the stream in association with the depth atlas.

[0132] The present principles have high efficiency advantages when applied to base views. However, the present principles can be applied to any kind of tiles, even tiles obtained through a pruning process. In one embodiment, the reduction can be applied to Figure 9 color tiles. In this embodiment, the color atlas and the depth atlas have different layouts, as the tile candidates for reduction can be different for the color domain and the depth domain.

[0133] Figure 11 Conventional downsampling in horizontal and vertical directions is shown. In one embodiment, depth downsampling uses the same integer (e.g. a power of 2) as a factor in both horizontal and vertical directions. The chosen downsampling factor is encoded in the metadata to signal to the decoder. Figure 11 Examples of 2x and 4x horizontal and vertical downsampling are shown in FIG. 2, along with two possible different phases for downsampling the pixel grid: 2x (left) or 4x (right) horizontal and vertical decimation; the placement of the downsampled pixels is shifted by half a period (at the top), or aligned with the source pixels (at the bottom).

[0134] To enable signaling of the same packing for depth and texture, and separate packing, and to enable signaling of a reduced spatial resolution for the depth component of a tile with respect to the texture component, the following syntax is proposed:

[0135]

[0136] different_texture_depth_packing_flag [ i ] equal to 1 indicates that the texture tile and the depth tile are packed differently in the texture component and the depth component of the i-th atlas, respectively.

[0137]

[0138] If different_texture_depth_packing_flag is equal to 1, then the texture tile atlas and the depth tile atlas are no longer aligned: different atlas sizes, different packing. The following fields are added to atlas_params:

[0139] depth_atlas_width[a] and depth_atlas_height[a] indicate the width and the height of the a-th depth atlas, respectively.

[0140] patch_pos_in_depth_atlas_x[a][i] and patch_pos_in_depth_atlas_y[a][i] specify, respectively, the horizontal and vertical coordinates in luma samples of the top-left corner of the i-th patch of the a-th depth atlas. The number of bits used to represent patch_pos_in_depth_atlas_x[a][i] and patch_pos_in_depth_atlas_y[a][i] is Ceil (Log2 (depth_atlas_width[a])) bits and Ceil (Log2 (depth_atlas_height[a])) bits, respectively.

[0141] patch_rotation_in_depth_atlas[a][i] has the same semantics as patch_rotation[a][i] for the rotation of the patches in the depth atlas.

[0142] log2_decimation_factor[a][i] specifies the power of 2 by which the depth patches have been decimated (in the horizontal and vertical directions) with respect to the texture patches. If this descriptor uses two bits, then the depth can be decimated by a factor of 2, 4 or 8. A value of 0 indicates that the patch has not been decimated.

[0143] Figure 12 An example of bilinear upsampling of patches transmitted at different resolutions is shown. Depth patches transmitted at low resolution must be up-converted at full spatial resolution at the decoder side. In one embodiment, the up-conversion is performed by bilinear filtering. Figure 12 An example of bilinear upsampling coefficients for a 1 :4 spatial up-conversion is shown in Table 2.

[0144] Figure 13 A method 130 for encoding depth views in an atlas of depth patches, according to a non-limiting embodiment of the principles of the application, is shown. At step 131, depth views are obtained from a source for encoding. The depth views are divided into a first region and a second region that are rectangular; the second region is a candidate for depth decimation. The region is a candidate for which the depth information can be described in a more economical way from the point of view of the transmission bitrate and from the point of view of the pixel rate, as discussed in relation to Figure 10The described. At step 132, the resolution of the second region is reduced by a factor. The factor is determined according to the depth characteristics of the second region. At step 133, a depth atlas is generated by packing the first region at full resolution and the reduced second region and tiles from other depth views on the 3D scene. The metadata indicates whether a tile is a first region or a reduced second region, and if a reduced second region, the factor used to reduce the resolution.

[0145] Figure 14 A method 140 for decoding a depth view in an atlas of depth tiles according to non-limiting embodiments of the principles of the application is shown. At step 141, a depth atlas is obtained from a source and a depth tile is extracted from the atlas in association with metadata indicating whether the tile is a first tile or a second tile, and if a second tile, a reduction factor. At step 142, the resolution of the second category of tile is enlarged by the associated factor. At step 143, the depth view is composed of the first category of tile and the enlarged second category of tile.

[0146] The specific implementations of the processes described herein can be implemented in, for example, a method or a process, an apparatus, a computer program product, a data stream, or a signal. Even if only discussed in the context of a single form of implementation, a specific implementation of a discussed feature can also be implemented in other forms (e.g., program). An apparatus can be implemented in, for example, appropriate hardware, software, and firmware. The methods can be implemented in, for example, an apparatus such as, for example, a processing device (e.g., a processor) that is generally associated with a computing device, such as, for example, a computer, a mobile telephone, a laptop, a personal digital assistant (“PDA”), and other devices that process data. A processing device also includes communication devices, such as, for example, smart phones, tablets, computers, mobile phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate communication of information between end-users.

[0147] The specific implementations of the various processes and features described herein can be implemented in various devices, including as specific implementations of any of the devices or applications described herein, particularly those associated with data encoding, data decoding, view generation, texture processing, and other processing of images and related texture and / or depth information. Examples of such devices include an encoder, a decoder, a post-processor processing output from a decoder, a pre-processor providing input to an encoder, a video encoder, a video decoder, a video codec, a web server, a set-top box, a laptop, a personal computer, a cell phone, a PDA, and other communication devices. It will be readily apparent to those skilled in the art that a device can be mobile, even if not always in motion.

[0148] Additionally, the methods can be implemented by instructions executed by a processor, and such instructions (and / or data values produced by a particular implementation) can be stored on a processor-readable medium, such as, for example, an integrated circuit, a software carrier or other storage device, such as, for example, a hard disk, a compact diskette ("CD"), an optical disk, such as, for example, a DVD, commonly referred to as a digital versatile disk or a digital video disk, random access memory ("RAM"), or read-only memory ("ROM"). The instructions can form an application program embodied on a processor-readable medium. The instructions can be examples of code written in any suitable programming language or environment. The instructions can be hardware, firmware, software, or some combination of these. The instructions can be found in, for example, an operating system, a separate application, or some combination of the two. Thus, the processor can be characterized as, for example, a device configured to perform processes and a device that includes a processor-readable medium having instructions for performing processes stored on it, such as a storage device. Further, in addition, or as an alternative to, instructions, a processor-readable medium can store data produced by a particular implementation.

[0149] It will be apparent to those skilled in the art that a particular implementation can generate signals formatted to carry information that can be, for example, stored or transmitted. The information can include, for example, instructions for performing a method or data produced by one of the described implementations. For example, a signal can be formatted to carry the rules for writing or reading the syntax of a described implementation as data, or to carry actual syntax values written by a described implementation as data. Such a signal can be formatted, for example, as an electromagnetic wave (for example, using a portion of the radio frequency spectrum) or a baseband signal. The formatting can include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information that the signal carries can be, for example, analog or digital information. The signal can be transmitted over a variety of different wired or wireless links, as is known. The signal can be stored on a processor-readable medium.

[0150] A number of implementations have been described. Nevertheless, it will be understood that numerous modifications can be made. For example, an element(s) from one implementation can be combined with elements from a different implementation. Additionally, well-known elements can not have been described or can be described only briefly. Moreover, processes can be added, modified or removed without departing from the scope of the disclosure. It is intended that all such modifications that fall within the scope of the claims are included.

Claims

1. A method for encoding a depth view of a multi-view plus depth, MVD, frame in a tile atlas, wherein a tile is a rectangular region of the depth view, the method comprising: dividing the depth view into a first rectangular region and a second rectangular region based on a complexity of geometric information within the rectangular regions, wherein the second rectangular region is a candidate for downsampling; downsampling a resolution of the second rectangular region in a vertical direction by a vertical factor and in a horizontal direction by a horizontal factor; and packing the first rectangular region and the downsampled second rectangular region in the tile atlas in association with metadata indicating for each tile in the tile atlas whether the tile is a first rectangular region or a downsampled second rectangular region and, if the tile is a downsampled second rectangular region, indicating the vertical factor and the horizontal factor used to downsample the resolution.

2. The method according to claim 1, wherein the first rectangular region is a region comprising complex geometric information and the second rectangular region is a region having a flat surface.

3. The method according to claim 1 or 2, wherein the vertical factor and the horizontal factor are different for two second rectangular regions.

4. The method according to claim 1 or 2, wherein each pixel of the second rectangular region has a defined depth value.

5. The method according to claim 1 or 2, the method further comprising encoding information indicating that tiles have different resolutions, the tile atlas and the associated metadata in a data stream.

6. An apparatus for encoding a depth view of a multi-view plus depth, MVD, frame in a tile atlas, wherein a tile is a rectangular region of the depth view, the apparatus comprising a processor configured for performing: dividing the depth view into a first rectangular region and a second rectangular region based on a complexity of geometric information within the rectangular regions, wherein the second rectangular region is a candidate for downsampling; downsampling a resolution of the second rectangular region in a vertical direction by a vertical factor and in a horizontal direction by a horizontal factor; and packing the first rectangular region and the downsampled second rectangular region in the tile atlas in association with metadata indicating for each tile in the tile atlas whether the tile is a first rectangular region or a downsampled second rectangular region and, if the tile is a downsampled second rectangular region, indicating the vertical factor and the horizontal factor used to downsample the resolution.

7. The apparatus according to claim 6, wherein the first rectangular region is a region comprising complex geometric information and the second rectangular region is a region having a flat surface.

8. The apparatus according to claim 6 or 7, wherein the vertical factor and the horizontal factor are different for two second rectangular regions.

9. The apparatus according to claim 6 or 7, wherein each pixel of the second rectangular region has a defined depth value. ​ ​ 10. The device of claim 6 or 7, wherein the processor is further configured to encode in a data stream information indicating that the tiles have different resolutions, the tile atlas and the associated metadata.

11. A method for decoding a depth view of a multi-view plus depth, MVD, frame in a packed tile atlas of tiles, wherein a tile is a rectangular region of the depth view, the method comprising: obtaining metadata indicating for each tile in the tile atlas whether the tile is a first tile or a second tile, the second tile being a reduced rectangular region, and if the tile is a reduced second region, a vertical factor and a horizontal factor; upsampling a resolution of the second tile in a vertical direction by the vertical factor and in a horizontal direction by the horizontal factor; and composing the depth view from the first tile and the upscaled second tile.

12. The method of claim 11, wherein the vertical factor and the horizontal factor are different for two second tiles.

13. A device for decoding a depth view of a multi-view plus depth, MVD, frame in a packed tile atlas of tiles, wherein a tile is a rectangular region of the depth view, the device comprising a processor configured for performing: obtaining metadata indicating for each tile in the tile atlas whether the tile is a first tile or a second tile, the second tile being a reduced rectangular region, and if the tile is a reduced second region, a vertical factor and a horizontal factor; upsampling a resolution of the second tile in a vertical direction by the vertical factor and in a horizontal direction by the horizontal factor; and composing the depth view from the first tile and the upscaled second tile.

14. The device of claim 13, wherein the vertical factor and the horizontal factor are different for two second tiles.

Images