Haptic atlas encoding and decoding format

Abstract

Methods and devices for encoding and decoding a data stream representing a 3D volumetric scene comprising haptic features associated with objects of the 3D scene are disclosed. At encoding, haptic features are associated with objects of the scene, e.g. as a haptic map. Haptic components can be stored as colors in points of the 3D scene. These components are projected onto a patch picture which is packed in an atlas image. At decoding, haptic components can be de-projected as colors onto reconstructed points from the depth component of the pixels of the decoded atlas.

Description

1. TECHNICAL FIELD

[0001] The present principles generally relate to the domain of three-dimensional (3D) scene and volumetric video content. The present document is also understood in the context of encoding, formatting and decoding of enriched video formats to render volumetric content related to haptic effects on end-user devices such as mobile devices or Head-Mounted Displays (HMDs). 2. BACKGROUND

[0002] This section is intended to introduce the reader to various aspects of art that can be related to various aspects of the present principles that are described and / or claimed below. This discussion is believed to be helpful in providing the reader with background information to facilitate a better understanding of the various aspects of the present principles. Accordingly, it should be understood that these statements are to be read in this light, and not as admissions of prior art.

[0003] Research results in Virtual Reality (VR) have shown that stimulating more senses than vision and hearing increases the sense of immersion. Today's HMDs come with hand controllers, allowing the user to control an avatar within the virtual world and thus "touch" virtual objects. Haptic feedback is provided thanks to a vibrator embedded into the controller.

[0004] So far, VR content can be distributed in various formats, considering 2D or 3D 360° video representations. Among those formats, MPEG currently specifies formats for 360° volumetric video data (by MPEG-I reference), one considering point clouds (PCC, MPEG-I Part 5) and the other one considering video + depth (3DoF+, MPEG-I Part 12). However, those formats are limited to audiovisual content.

[0005] It would be valuable to add data describing haptic, gustatory or olfactory content. The MPEG-V [MPEG-V] architecture is a format for the distribution of audiovisual content augmented with "sensory effects". The MPEG-V standard describes these sensory effects, including haptic effects, but also gustatory effects and olfactory effects. MPEG-V supports several haptic effects such as vibration, rigid body motion or force feedback. In addition to the effects, it is possible to stimulate the whole body of the user or target specific points in a space shared by users.

[0006] Although this format is generic enough to describe many effects, it is not possible to associate precise spatial haptic properties with visual objects (i.e. define the texture of an object, its local rigidity and thermal properties). In addition, it is desirable to re-use the MPEG-I video container and add this haptic information. 3. SUMMARY

[0007] The following presents a simplified summary of the principles of the application in order to provide a basic understanding of some aspects of the application. This summary is not an extensive overview of the principles of the application. It is not intended to identify key or critical elements of the application. The following summary merely presents some aspects of the application in a simplified form as a prelude to the more detailed description provided below.

[0008] The principles of the application relate to a method comprising:

[0009] - obtaining a 3D scene, at least one point of said 3D scene comprising at least one component relative to a haptic feature;

[0010] - projecting a depth component of said 3D scene and said at least one component relative to a haptic feature onto patch pictures; and

[0011] - packing said patch pictures in an atlas image.

[0012] In a particular embodiment, the depth component and said component relative to a haptic feature are projected on different patch pictures, said different patch pictures being packed in different atlases. In another embodiment, the haptic feature is encoded as a color value. In some embodiments, the haptic feature belongs to a set of haptic features comprising rate-hardness, contact area expansion rate, dynamic stiffness, local surface orientation, local indentation, stroke spectral response, dynamic friction, static friction, stick-slip and thermal distribution.

[0013] The principles of the application also relate to a device comprising a processor configured to perform the above method.

[0014] The principles of the application also relate to a method comprising:

[0015] - obtaining a data stream comprising atlas image packed patch pictures, said patch pictures being projections of points of a 3D scene and comprising a depth component and at least one component relative to a haptic feature;

[0016] - de-projecting components of pixels of said atlas by associating said at least one component relative to a haptic feature with points de-projected from said depth component relative to said pixels.

[0017] In one embodiment, the method further comprises rendering a haptic effect from said at least one component relative to a haptic feature associated with de-projected points of a 3D scene and from a distance of a virtual camera used to render said 3D scene to said points.

[0018] In some embodiments, said haptic feature is encoded as a color value and / or the depth component and said component relative to a haptic feature are projected on different patch pictures, said different patch pictures being packed in different atlases.

[0019] The principles of the application also relate to a device comprising a processor configured to perform the method described above. The principles of the application also relate to a non-transitory medium storing data comprising atlas image packing patch pictures, the patch pictures being projections of points of a 3D scene and comprising a depth component and at least one component relative to haptic features. 4. BRIEF DESCRIPTION OF DRAWINGS

[0020] The present disclosure will be better understood and other specific features and advantages will emerge upon reading the following description, the description making reference to the drawings in which:

[0021] - Figure 1 A three-dimensional (3D) model of an object and points of a point cloud corresponding to this 3D model are shown according to a non-limiting embodiment of the principles of the application;

[0022] - Figure 2 Non-limiting examples of encoding, transmitting and decoding data representative of a sequence of 3D scenes are shown according to a non-limiting embodiment of the principles of the application;

[0023] - Figure 3 An example architecture of a device that can be configured to implement the methods described with respect to Figure 11 and Figure 12 the principles of the application is shown according to a non-limiting embodiment of the principles of the application;

[0024] - Figure 4 An example of an implementation of the syntax of a stream when transmitting data through a packet-based transmission protocol is shown according to a non-limiting embodiment of the principles of the application;

[0025] - Figure 5 A patch atlas method is shown according to a non-limiting embodiment of the principles of the application, taking the example of 4 projection centers;

[0026] - Figure 6 An example of an atlas comprising texture information of points of a 3D scene is shown according to a non-limiting embodiment of the principles of the application;

[0027] - Figure 7 An example of an atlas comprising depth information of points of a 3D scene is shown according to a non-limiting embodiment of the principles of the application; Figure 6

[0028] - Figure 8 The concept of haptic maps is shown. According to a non-limiting embodiment of the principles of the application, these maps encode haptic information in a texture-like format;

[0029] - Figure 9 ​A computer-graphic 3D scene including an oven is shown, which is an object with tactile properties;

[0030] - Figure 10 The illustration shows a non-limiting embodiment of the invention, utilizing a tactile attribute map obtained through a process for generating a color and / or depth atlas;

[0031] - Figure 11 A method for encoding tactile features associated with materials of 3D objects in a 3D scene for volumetric rendering is shown according to a non-limiting embodiment of the principles of the present invention.

[0032] - Figure 12 A method for decoding tactile features associated with materials of 3D objects in a 3D scene for volumetric rendering is shown, according to a non-limiting embodiment of the principles of the present invention. 5. Detailed Implementation

[0033] The principles of the invention will be described more fully below with reference to the accompanying drawings, in which examples of the principles of the invention are shown. However, the principles of the invention may be embodied in many alternative forms and should not be construed as limited to the examples set forth herein. Therefore, while the principles of the invention are susceptible to various modifications and alternatives, specific examples are shown by way of example in the drawings and will be described in detail herein. However, it should be understood that there is no intention to limit the principles of the invention to the specific forms disclosed, but rather, this disclosure is intended to cover all modifications, equivalents, and alternatives that fall within the spirit and scope of the principles of the invention as defined by the claims.

[0034] The terminology used herein is for the purpose of describing particular examples only and is not intended to limit the principles of the invention. As used herein, the singular forms “a,” “an,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It will be further understood that, when used in this specification, the terms “comprising” and / or “including” specify the presence of the stated feature, integer, step, operation, element, and / or component, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Furthermore, when an element is referred to as “responding” or “connected” to another element, it may directly respond to or be connected to the other element, or there may be intermediate elements present. Conversely, when an element is referred to as “directly responding” or “directly connected” to another element, there are no intermediate elements present. As used herein, the term “and / or” includes any and all combinations of one or more of the listed related items and may be abbreviated to “ / ”.

[0035] It should be understood that, although the terms first, second, etc. can be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element, without departing from the teachings of the present principles.

[0036] Although some of the diagrams include arrows on communication paths to demonstrate a primary communication direction, it is to be understood that communication can occur in the opposite direction to the depicted arrows.

[0037] Some examples are described with respect to block and operational flow diagrams in which each block represents a circuit element, module, or a portion of code that comprises one or more executable instructions for implementing the specified logical function. It should also be noted that in other implementations the function noted in the block can occur in a different order than that noted. For example, two blocks noted in succession can in fact be executed substantially concurrently or the blocks can sometimes be executed in reverse order according to the functionality involved.

[0038] Reference within this document to "one example" or "an example" means that a particular feature, structure, or characteristic described in connection with the example can be included in at least one implementation of the principles of the present disclosure. The appearance of the phrase "one example" or "an example" in various places in the specification are not necessarily all referring to the same example, nor are they necessarily mutually exclusive or alternative examples to one another.

[0039] The reference signs in the claims are presented by way of illustration only and should not be construed as limiting the scope of the claims. Although not explicitly described, the present examples and variants can be employed in any combination or sub-combination.

[0040] Figure 1 A three-dimensional (3D) model 10 of an object and points of a point cloud 11 corresponding to the 3D model 10 are shown. The 3D model 10 and the point cloud 11 can for example correspond to a possible 3D representation of an object of a 3D scene comprising other objects. The model 10 can be a 3D mesh representation and the points of the point cloud 11 can be the vertices of the mesh. The points of the point cloud 11 can also be points distributed on the surface of the faces of the mesh. The model 10 can also be represented as a splatted version of the point cloud 11, the surface of the model 10 being created by splatting the points of the point cloud 11. The model 10 can be represented by many different representations such as voxels or splines. Figure 1 The fact that a point cloud can be defined with a surface representation of a 3D object and that a surface representation of a 3D object can be generated from a cloud of points is shown. As used herein, projecting points of a 3D object (by extension points of a 3D scene) onto an image is equivalent to projecting any representation of this 3D object, for example a point cloud, a mesh, a spline model or a voxel model.

[0041] A point cloud can be represented in memory as, for example, a vector-based structure, where each point has its own coordinates (e.g. three-dimensional coordinates XYZ, or an angle in space and a distance from / to the viewpoint (also called depth)) in the frame of reference of the viewpoint and one or more attributes, also called components, in the frame of reference of the viewpoint. One example of a component is a color component, which can be represented in various color spaces, such as RGB (red, green, and blue) or YUV (Y is the luminance component and UV are two chrominance components). A point cloud is a representation of a 3D scene comprising objects. The 3D scene can be seen from a given viewpoint or range of viewpoints. A point cloud can be obtained in various ways, for example:

[0042] • from a capture of real objects shot by a camera rig, optionally assisted by depth active sensing devices;

[0043] • from a capture of virtual / synthetic objects shot by a virtual camera rig in a modeling tool;

[0044] • from a mix of both real and virtual objects.

[0045] A 3D scene, especially when made for 3DoF+ rendering, can be represented by a Multi-View+Depth (MVD) frame. Then, a volumetric video is a sequence of MVD frames. In this approach, volumetric information is transmitted as a combination of color and depth patches stored in corresponding color and depth atlases, which are then video encoded with a regular codec, typically HEVC. Each combination of color and depth patches typically represents a sub-portion of the MVD input view and the set of all patches is designed at the encoding stage to cover the whole scene while there is as little redundancy as possible. At the decoding stage, the atlases are first video decoded and the patches are rendered in a view synthesis process to recover the viewport associated with the desired viewing position.

[0046] Figure 2 Non-limiting examples of encoding, transmitting and decoding data representing a sequence of 3D scenes are illustrated. The encoding format can be, for example, compatible with 3DoF, 3DoF+ and 6DoF decoding at the same time.

[0047] A sequence of 3D scenes 20 is obtained. As a sequence of pictures is a 2D video, a sequence of 3D scenes is a 3D (also called volumetric) video. The sequence of 3D scenes can be provided to a volumetric video rendering device for 3DoF, 3Dof+ or 6DoF rendering and display.

[0048] A sequence of 3D scenes 20 can be provided to an encoder 21. The encoder 21 takes as input one 3D scene or a sequence of 3D scenes and provides a bitstream representing the input. The bitstream can be stored in a memory 22 and / or on an electronic data medium and can be transmitted over a network 22. The bitstream representing the sequence of 3D scenes can be read from the memory 22 and / or received from the network 22 by a decoder 23. The decoder 23 takes as input the bitstream and provides a sequence of 3D scenes in a point cloud format for example.

[0049] The encoder 21 can comprise several circuits implementing several steps. In a first step, the encoder 21 projects each 3D scene onto at least one 2D picture. A 3D projection is any method of mapping three-dimensional points into a two-dimensional plane. Since most current methods for displaying graphical data are based on planar (pixel information from several bitplanes) two-dimensional media, the use of this type of projection is widespread, especially in computer graphics, engineering, and cartography. A projection circuit 211 provides a sequence of at least one two-dimensional frame 2111 for the 3D scenes 20. The frame 2111 comprises color information and depth information representing the 3D scene projected onto the frame 2111. In a variant, the color information and the depth information are encoded in two separate frames 2111 and 2112.

[0050] Metadata 212 are used and updated by the projection circuit 211. The metadata 212 comprise information about the projection operation (e.g. projection parameters) and information about the way the color and depth information are organized within the frames 2111 and 2112, as described in connection with the Figure 5 to Figure 7

[0051] A video encoding circuit 213 encodes the sequence of frames 2111 and 2112 into a video. The frames 2111 and 2112 of the 3D scene (or the sequence of frames of the 3D scene) are encoded in a stream by the video encoder 213. Then, the video data and the metadata 212 are encapsulated in a data stream by a data encapsulation circuit 214.

[0052] The encoder 213 is compatible with encoders such as:

[0053] - JPEG, specification ISO / CEI 10918-1 UIT-T Recommendation T.81, https: / / www.itu.int / rec / T-REC-T.81 / en;

[0054] ​- AVC, also called MPEG-4 AVC or h264. Specified in both UIT-T H.264 and ISO / CEI MPEG-4 Part 10 (ISO / CEI 14496-10), http: / / www.itu.int / rec / T-REC-H.264 / en, HEVC (whose specification is found on the ITU website, T recommendation, H series, h265, http: / / www.itu.int / rec / T-REC-H.265-201612-I / en) ;

[0055] - 3D-HEVC (extension of HEVC, whose specification is found on the ITU website, T recommendation, H series, h265, http: / / www.itu.int / rec / T-REC-H.265-201612-I / en annex G and I) ;

[0056] - VP9 developed by Google;

[0057] - AV1 (AOMedia Video 1) developed by Alliance for Open Media; or

[0058] - future standards such as Versatile Video Coder or MPEG-I or future versions of MPEG-V.

[0059] The data stream is stored in a memory accessible by the decoder 23, for example through the network 22. The decoder 23 comprises different circuits implementing different decoding steps. The decoder 23 takes as input the data stream generated by the encoder 21 and provides a sequence of 3D scenes 24 to be rendered and displayed by a volumetric video display device such as a head-mounted device (HMD). The decoder 23 obtains the stream from a source 22. For example, the source 22 belongs to a group comprising:

[0060] - a memory, for example a video memory or a RAM (or Random Access Memory), a flash memory, a ROM (or Read Only Memory), a hard disk;

[0061] - an interface, for example with a mass storage memory, a RAM, a flash memory, a ROM, an optical or magnetic support;

[0062] - a communication interface, for example a wired interface (for example a bus interface, a wide area network interface, a local area network interface) or a wireless interface (such as an IEEE 802.11 interface or an interface); and

[0063] - a user interface, such as a graphical user interface, enabling a user to input data.

[0064] The decoder 23 comprises a circuit 234 for extracting the data encoded in the data stream. The circuit 234 takes as input the data stream and provides metadata 232 corresponding to the metadata 212 encoded in the stream and a two-dimensional video. The video is decoded by a video decoder 233 providing a sequence of frames. The decoded frames comprise color and depth information. In a variant, the video decoder 233 provides two sequences of frames, one containing color information and the other containing depth information. The circuit 231 uses the metadata 232 to project the color and depth information from the decoded frames to provide a sequence of 3D scenes 24. The sequence of 3D scenes 24 corresponds to the sequence of 3D scenes 20, possibly with a loss of precision related to the encoding as a 2D video and to the video compression.

[0065] Figure 3 An example architecture of a device 30 that can be configured to implement the method described with respect to Figure 11 and Figure 12 An example architecture of a device 30 that can be configured to implement the method described with respect to Figure 2 The encoder 21 and / or the decoder 23 can implement this architecture. Alternatively, each circuit in the encoder 21 and / or the decoder 23 can be a device according to the architecture of Figure 3 linked together, for example via its bus 31 and / or via the I / O interface 36.

[0066] The device 30 comprises the following elements connected together by a data and address bus 31 :

[0067] - a microprocessor 32 (or CPU), which is for example a DSP (or Digital Signal Processor);

[0068] - a ROM (or Read Only Memory) 33;

[0069] - a RAM (or Random Access Memory) 34;

[0070] - a storage interface 35;

[0071] - an I / O interface 36 for receiving data to be transmitted from an application; and

[0072] - a power supply, for example a battery.

[0073] According to one example, the power supply is external to the device. In each of the memories mentioned, the word "register" used in the description can correspond to an area of small capacity (a few bits) or to a very large area (for example, the entire program or a large amount of received or decoded data). The ROM 33 comprises at least the program and the parameters. The ROM 33 can store algorithms and instructions for performing the techniques according to the principles of the application. When switched on, the CPU 32 uploads the program in the RAM and executes the corresponding instructions.

[0074] The RAM 34 comprises the program in registers executed by the CPU 32 and uploaded after the switch-on of the device 30, the input data in registers, the intermediate data in the different states of the method in registers and other variables used for the execution of the method in registers.

[0075] The specific implementations described herein can be implemented in, for example, a method or process, an apparatus, a computer program product, a data stream, or a signal. Even if only discussed in the context of a single form of implementation, a specific implementation of a discussed feature can also be implemented in other forms (e.g., program). An apparatus can be implemented in, for example, appropriate hardware, software, and firmware. The methods can be implemented in, for example, an apparatus that generally relates to processing devices such as, for example, a processor, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices such as, for example, a computer, a cell phone, a portable / personal digital assistant ("PDA"), and other devices that facilitate the communication of information between end users.

[0076] According to an example, the device 30 is configured to implement the method described in relation to Figure 11 and Figure 12 The method described is such that it belongs to the set comprising:

[0077] - a mobile device;

[0078] - a communication device;

[0079] - a game device;

[0080] - a tablet (or tablet computer);

[0081] - a laptop;

[0082] - a still picture camera;

[0083] - a video camera;

[0084] - an encoding chip;

[0085] - a server (for example a broadcast server, a video on demand server or a web server).

[0086] Figure 4 An example of an implementation of the syntax of a stream is shown when the data is transmitted by a packet-based transmission protocol. Figure 4 An exemplary structure 4 of a volumetric video stream is shown. The structure is contained in a container that organizes the stream in independent elements of syntax. The structure can comprise a header portion 41, which is a set of data common to each syntax element of the stream. For example, the header portion comprises some metadata on the syntax elements, describing the nature and the role of each of them. The header portion can also comprise Figure 2part of the metadata 212, for example the coordinates of the center viewpoint used to project the points of the 3D scene onto the frames 2111 and 2112. This structure comprises a payload comprising a syntax element 42 and at least one syntax element 43. The syntax element 42 comprises data representative of color and depth frames. The images can have been compressed according to a video compression method.

[0087] The syntax element 43 is part of the payload of the data stream and can comprise metadata about how the frames of syntax element 42 have been encoded, for example parameters used to project and pack points of the 3D scene onto the frames. Such metadata can be associated with each frame or group of frames of the video, also called group of pictures (GoP) in video compression standards.

[0088] Figure 5 A patch atlas approach is illustrated with the example of 4 projection centers. The 3D scene 50 comprises a person. For example, the projection center 51 is a perspective camera and the camera 53 is an orthographic camera. The cameras can also be omnidirectional cameras with for example spherical mapping (e.g. equirectangular mapping) or cubic mapping. According to the projection operations described in the projection data of the metadata, the 3D points of the 3D scene are projected onto 2D planes associated with virtual cameras located at the projection centers. In the example, the projection of the points captured by the camera 51 is mapped onto the patch 52 according to a perspective mapping and the projection of the points captured by the camera 53 is mapped onto the patch 54 according to an orthographic mapping. Figure 5

[0089] The clustering of the projected pixels produces a plurality of 2D patches which are packed in a rectangular atlas 55. The organization of the patches within the atlas defines an atlas layout. In one embodiment, two atlases with the same layout: one for texture (i.e. color) information and one for depth information. Two patches captured by the same camera or by two different cameras can comprise information representative of the same part of the 3D scene, as for example patches 54 and 56.

[0090] The packing operation produces patch data for each generated patch. The patch data comprises a reference to the projection data (for example an index in a table of projection data or a pointer to the projection data (i.e. an address in memory or in the data stream)) and information describing the position and size of the patch within the atlas (for example top-left coordinates, size and width in pixels). The patch data items are added to the metadata to be encapsulated in the data stream in association with the compressed data of one or two atlases.

[0091] Figure 6 An example of an atlas 60 comprising texture information (for example RGB data or YUV data) of points of a 3D scene according to a non-limiting embodiment of the principles of the application is illustrated. As combined with the previous figures, the atlas 60 comprises a plurality of patches 61, 62, 63, 64, 65, 66, 67, 68, 69 and 70. The atlas 60 is packed in a rectangular layout. The atlas 60 comprises a patch 61 comprising texture information of points captured by the camera 51 and a patch 62 comprising texture information of points captured by the camera 53. The atlas 60 also comprises a patch 63 comprising depth information of points captured by the camera 51 and a patch 64 comprising depth information of points captured by the camera 53. Figure 5 ​The atlas, as explained, is a patchwork of images, which is a view created by projecting a portion of a 3D scene.

[0092] exist Figure 6 In the example, atlas 60 includes a first portion 61 and one or more second portions 62, the first portion including texture information of points in a 3D scene visible from the viewpoint. The texture information of the first portion 61 can be obtained, for example, according to an isometric projection map, an example of a spherical projection map. Figure 6 In the example, the second part 62 is positioned at the left and right boundaries of the first part 61, but the second part can be positioned differently. The second part 62 includes texture information of a portion of the 3D scene that is complementary to the portion visible from the viewpoint. The second part can be obtained by removing points visible from the first viewpoint (whose textures are stored in the first part) from the 3D scene and projecting the remaining points according to the same viewpoint. The latter process can be repeated iteratively to obtain the hidden portion of the 3D scene each time. According to a variant, the second part can be obtained by removing points visible from the viewpoint (e.g., the center viewpoint) from the 3D scene (whose textures are stored in the first part) and projecting the remaining points according to viewpoints different from the first viewpoint, such as one or more second viewpoints from a viewing space centered on the center viewpoint (e.g., a 3DoF rendered viewing space).

[0093] The first part 61 can be viewed as the first large texture patch (corresponding to the first part of the 3D scene), and the second part 62 includes smaller texture patches (corresponding to the second part of the 3D scene that complements the first part). This type of atlas has the advantage of being compatible with both 3DoF rendering (when only the first part 61 is rendered) and 3DoF+ / 6DoF rendering.

[0094] Figure 7 The illustration shows a non-limiting embodiment based on the principles of the invention, including... Figure 6 An example of atlas 70 containing depth information of points in a 3D scene. Atlas 70 can be viewed as corresponding to... Figure 6 The texture image is a depth image of 60.

[0095] Atlas 70 includes a first portion 71 and one or more second portions 72, the first portion including depth information of points in the 3D scene visible from a central viewpoint. Atlas 70 can be obtained in the same manner as Atlas 60, but contains depth information associated with points in the 3D scene instead of texture information.

[0096] For 3DoF rendering of a 3D scene, only one viewpoint is considered, usually the central viewpoint. The user can rotate his head around the first viewpoint with three degrees of freedom to watch various parts of the 3D scene, but the user cannot move this unique viewpoint. The points of the scene to be encoded are the points that are visible from this unique viewpoint, and only the texture information needs to be encoded / decoded for 3DoF rendering. For 3DoF rendering, the points of the scene that are not visible from this unique viewpoint do not need to be encoded, as the user cannot access them.

[0097] For 6DoF rendering, the user can move the viewpoint anywhere in the scene. In this case, every point of the scene (depth and texture) in the bitstream needs to be encoded, as the user who can move his / her viewpoint can access every point. At the encoding stage, there is no means to know a priori from which viewpoint the user will observe the 3D scene.

[0098] For 3DoF+ rendering, the user can move the viewpoint within a limited space around the central viewpoint. This enables the experience of parallax. The data representing the parts of the scene that are visible from any point of the viewing space will be encoded into the stream, including the data representing the 3D scene that is visible according to the central viewpoint (i.e. the first parts 61 and 71). For example, the size and shape of the viewing space can be decided and determined at the encoding step and encoded in the bitstream. The decoder can obtain this information from the bitstream and the renderer will limit the viewing space to the space determined by the obtained information. According to another example, the renderer determines the viewing space according to hardware constraints, for example hardware constraints related to the capabilities of the sensors that detect the user movements. In this case, if at the encoding stage, a point that is visible from a point within the viewing space of the renderer has not been encoded in the bitstream, this point will not be rendered. According to yet another example, the data representing every point of the 3D scene (e.g. texture and / or geometry) is encoded in the stream, regardless of the rendering viewing space. To optimize the size of the stream, only a subset of the points of the scene can be encoded, for example a subset of the points that can be seen according to the rendering viewing space.

[0099] Patches are created to be sufficiently non-redundant and complementary. The process for generating patches from a multi-view+depth (MVD) representation of a 3D scene includes "pruning" the input source views to remove any redundant information. To this end, each input view (color+depth) is iteratively pruned after the other. A set of unpruned views (called base views) is first selected among the source views and is transmitted completely. Then, a set of remaining views (called additional views) is iteratively processed to remove redundant information (in terms of color and depth similarity) with the base views and the additional views that have already been pruned. The color or depth values of the pruned pixels are replaced by a predetermined value, for example 0 or 255. For example, in Figure 6 and Figure 7Among others, the central view 61+71 is the base view. In other embodiments, the atlas can gather several base views.

[0100] Figure 8 The concept of "haptic maps" is illustrated. These maps encode haptic information in a texture-like format. Up to ten haptic features are described: compatibility (rate-hardness, contact area expansion rate, dynamic stiffness), surface geometry (local surface orientation, local indentation, stroke spectral response), friction (dynamic friction, static friction, stick-slip), and warmth (thermal distribution).

[0101] Thermal maps represent hidden thermal pipe systems. Each pixel pair of these maps encodes a value of a corresponding haptic metric. It is then easy to describe the precise spatial haptic information on any object surface.

[0102] Thanks to this image-based data format, existing editing tools can be easily used (such as Maya, Unity or Unreal Engine) to create this information. Similarly, to distribute it, existing video formats and containers can be extended to support this additional information. Haptic maps can be converted into atlases or into other point cloud representation formats with haptic data.

[0103] According to the principles of the invention, video atlases are used to add haptic effects to volumetric video formats to transmit color and depth data. These formats are extended by defining haptic atlases. Similarly to color atlases that include colors to be displayed, haptic atlases include haptic data to be rendered by haptic devices. For each type of haptic data to be encoded (i.e. physical properties), different atlases are created (e.g. temperature, friction, etc.).

[0104] Figure 9 A computer graphics 3D scene is illustrated that includes an oven, which is an object with haptic properties. Figure 9 A 3D scene can be represented as a set of 2D patches packed in an atlas as described with respect to Figure 5 to Figure 7 Haptic data can be associated with objects of this scene that have editing tools in the 3D original world or equivalently in the 2D re-projection. In Figure 9 In the example of an oven, the hardness of the oven is obtained as well as its temperature.

[0105] In one embodiment, eleven haptic features are considered. Hardness is defined by the rate-hardness metric and temperature is defined by the temperature metric.

[0106] ID haptic_feature format range resolution 0 none - - - 1 rate-hardness 8-bit 0-10000 N.s-1 / m.s-1 40 N.s-1 / m.s-1 2 contact-area-diffusion-rate 8-bit 0-25.6 N / cm2 0.1 N / cm2 3 local-surface-orientation 24-bit (3x8-bit) 2x0-180° 0.002° 4 local-indentation 8-bit ±5 mm 0.04 mm 5 dynamic-friction 8-bit ±5 0.04 6 static-friction 8-bit ±5 0.04 7 relative-temperature 8-bit ±25.4° 0.2° 8 temperature 8-bit [-50:+75]℃ 0.5℃ 9 dynamic-stiffness 8-bit 0-255 1-indexed-in-table 10 stroke-spectrum-response 8-bit 0-255 1-indexed-in-table 11 stick-slip 8-bit 0-255 1-indexed-in-table

[0107] Dynamic stiffness, stroke spectral response and stickiness do not directly encode haptic values as indices of tables. The IDs correspond to files where the coefficients of autoregressive filters are stored. They model the vibrations measured with the material during a brief contact (dynamic stiffness) or a stroke (stroke spectral response or stickiness transient).

[0108]

[0109]

[0110] Figure 10 The atlas representing the haptic property maps obtained with the process for generating color and / or depth atlases is shown. In the haptic atlas, the « value » / « intensity » of any haptic effect is associated with each pixel of the 2D re-projection view of Figure 6 or Figure 7 .

[0111] The haptic atlas is created in a similar way as the color / depth atlas, but instead of color / depth information, haptic information is stored. The number of haptic atlases created depends on the number of haptic features to store. In the example of Figure 9 and Figure 10 , two types of maps are created: one for temperature (on the right of Figure 10 ) and one for hardness-velocity (on the left of Figure 10 ). As shown, this approach allows to precisely define haptic zones in the scene. Temperature is highest near the fire and hardness is highest at the oven edges.

[0112] The haptic patches and color / depth patches of the haptic and color / depth atlases can or can not be perfectly aligned: if only a few objects have haptic properties, the number of patches contained in the haptic atlas can be very lower than one of the color / depth atlases.

[0113] To encode multiple types of haptic maps, the following syntax format is proposed:

[0114]

[0115] where a is the atlas ID and type is defined as:

[0116] type data 0 texture 1 depth 2 rate-hardness 3 contact-area-diffusion-rate 4 local-surface-orientation 5 local-indentation 6 dynamic-friction 7 static-friction 8 relative-temperature 9 temperature

[0117] Each patch is encoded according to its type:

[0118] type format 0 YUV420P10LE 1 YUV420P10LE 2 YUV400P 3 YUV400P 4 YUV444P 5 YUV400P 6 YUV400P 7 YUV400P 8 YUV400P 9 YUV400P 10 YUV400P 11 YUV400P 12 YUV400P

[0119] In another embodiment, the Atlas params can remain as defined in MPEG-I, but the atlas parameter list can be changed:

[0120]

[0121] In another embodiment, which does not require modification of the current syntax, the haptic atlas is defined separately as a dedicated additional / complementary structure,

[0122]

[0123] where haptic_atlas_type is defined as:

[0124] haptic_atlas_type data 0 rate-hardness 1 contact-area-diffusion-rate 2 local-surface-orientation 3 local-indentation 4 dynamic-friction 5 static-friction 6 relative-temperature 7 temperature 8 dynamic-stiffness 9 stroke-spectrum-response 10 stick-slip

[0125] Figure 11 A method 110 for encoding haptic features associated with the material of a 3D object of a 3D scene for volume rendering is shown. At step 111, a haptic map is associated with an object of the 3D scene to be encoded for volume rendering. The haptic map is generated and associated with the object embodying each haptic feature type. In the present application, up to eleven features are proposed. At step 112, a depth atlas is generated according to the process described with respect to Figure 5 to Figure 7 The color atlas can also be generated at the same time. According to the principles of the present application, a haptic atlas is generated using a similar process for each haptic map type. At step 113, each generated atlas is encoded into a data stream associated with corresponding metadata, for example following one of the syntax formats proposed in the present application.

[0126] Figure 12 A method 120 for decoding haptic features associated with the material of a 3D object of a 3D scene for volume rendering is shown. At step 121, a data stream is obtained and from the stream, a depth atlas is decoded according to the principles described with respect to Figure 5 to Figure 7 The color atlas can also be decoded at the same time. From the stream corresponding to the haptic type, a haptic atlas is decoded. If the object embeds several haptic types, a haptic atlas is decoded for each of these haptic types. At step 122, the object is de-projected from the depth atlas. If a color atlas exists, the color information is associated with the object by de-projecting the color atlas. At step 123, the haptic information is de-projected onto the object by using a similar process. Each decoded haptic atlas is de-projected, thus the haptic features are associated with the 3D object.

[0127] In another embodiment; the list of cameras for haptic content can be different from the list of cameras for visual content. Indeed, the visual space and the haptic space can be different. For example, the back of an object can not be seen when it can be touched, or the required resolution can be less than the required resolution for color or depth information.

[0128] In this case, a table of the proposed syntax is replicated to define another camera list (i.e., haptic camera rig), where each camera of the list shares its syntax with the current syntax.

[0129] The implementations described herein can be implemented in, for example, a method or a process, an apparatus, a computer program product, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method or device), the implementation of the discussed features can also be implemented in other forms (for example, a program). An apparatus can be implemented in, for example, appropriate hardware, software, and firmware. The methods can be implemented in, for example, an apparatus that is specifically adapted to be a processing device, such as, for example, a processor, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as, for example, a smartphone, a tablet, a computer, a mobile phone, a portable / personal digital assistant ("PDA"), and other devices that facilitate the communication of information between end users.

[0130] Implementations of various processes and features described herein can be embodied in a variety of different equipment or applications, particularly equipment or applications associated with data encoding, data decoding, view generation, texture processing, and other processing of images and related texture information and / or depth information. Examples of such equipment include an encoder, a decoder, a post-processor processing output from a decoder, a pre-processor providing input to an encoder, a video encoder, a video decoder, a video codec, a web server, a set-top box, a laptop, a personal computer, a cell phone, a PDA, and other communication devices. As should be apparent, the equipment can be mobile, even if implemented in a stationary device.

[0131] Additionally, the methods can be implemented by instructions executed by a processor, and such instructions (and / or data values produced by implementations) can be stored on a processor-readable medium such as, for example, an integrated circuit, a software carrier, or another storage device such as, for example, a hard disk, a compact diskette ("CD"), an optical disk, such as, for example, a DVD, generally referred to as a digital universal disk or digital video disk, random access memory ("RAM"), or read-only memory ("ROM"). The instructions can form an application program embodied on a processor-readable medium. The instructions can be, for example, in hardware, firmware, software, or a combination. The instructions can be found in, for example, an operating system, a separate application, or a combination of the two. The processor can be characterized, therefore, as being configured to perform a process in response to the processor being made to execute the instructions. The processor can also be considered to be entirely hardware-based, even if the processor comprises a software part or a firmware part. The processor-based device or apparatus can be configured to perform one or more processes. The processor can be configured to perform a process by, for example, executing one or more sets of instructions. A set of instructions can be retrieved by the processor from a storage device or computer-readable medium. During retrieval, the instructions can be in the form of electrical signals or other physical signals. The processor can process the electrical or other physical signals, e.g., digitalize the signals, demodulate the signals, and / or decode the signals. A set of instructions can be retrieved by the processor from a storage device or computer-readable medium. During retrieval, the instructions can be in the form of electrical signals or other physical signals. The processor can process the electrical or other physical signals, e.g., digitalize the signals, demodulate the signals, and / or decode the signals. The processor can be configured to store the processed signals in a memory.

[0132] It will be apparent to those skilled in the art that the specific implementations can produce various signals formatted to carry information that can be, for example, stored or transmitted. The information can include, for example, instructions for performing a method, or data created by one of the specific implementations. For example, a signal can be formatted to carry as data the rules for writing or reading the syntax of a described implementation, or to carry as data the actual syntax-values written by a described implementation. Such a signal can be formatted, for example, as an electromagnetic wave (for example, using a portion of the spectrum that is visible, or that is not visible to humans) or a baseband signal. The formatting can include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information that the signal carries can be, for example, analog or digital information. The signal can be transmitted over a variety of different wired or wireless links, as is known. The signal can be stored on a processor-readable medium.

[0133] A number of implementations have been described. Nevertheless, it will be understood that numerous modifications can be made. For example, elements of different implementations can be combined, supplemented, modified, or removed to produce other implementations. Additionally, one of ordinary skill will understand that other structures and processes can be substituted for those disclosed and the resulting implementations will perform at least substantially the same function(s) in at least substantially the same way(s) to achieve at least substantially the same result(s). Accordingly, these and other implementations are contemplated by this application.

Claims

1. An encoding method, comprising: - Obtain a 3D scene, wherein at least one point of the 3D scene includes a depth component and at least one component related to tactile features of a tactile type; - Project the depth components of the points in the 3D scene onto the depth patch image; - Project at least one component of the points in the 3D scene that is related to the haptic feature onto the haptic patch screen; - Package the depth patch image into a depth atlas image and package the haptic patch image into a haptic atlas image; as well as - Encode the depth atlas image and the tactile atlas image having the tactile type.

2. The method according to claim 1, wherein the tactile type belongs to a set of tactile types, the set of tactile types including rate-hardness, contact area expansion rate, dynamic stiffness, local surface orientation, local indentation, stroke spectral response, dynamic friction, static friction, stick-slip, temperature and relative temperature.

3. The method according to claim 1 or 2, wherein at least one point of the 3D scene comprises multiple components respectively associated with tactile features of different tactile types, and packaging the tactile patch image into a tactile atlas image comprises: For each different tactile type, the tactile patch image is packaged into a tactile atlas image.

4. An encoding device, comprising a processor, the processor being configured to: - Obtain a 3D scene, wherein at least one point of the 3D scene includes a depth component and at least one component related to tactile features of a tactile type; - Project the depth components of the points in the 3D scene onto the depth patch image; - Project at least one component of the points in the 3D scene that is related to the haptic feature onto the haptic patch screen; - Package the depth patch image into a depth atlas image and package the haptic patch image into a haptic atlas image; and - Encode the depth atlas image and the tactile atlas image having the tactile type.

5. The device according to claim 4, wherein the tactile type belongs to a group of tactile types, the group of tactile types including rate-hardness, contact area expansion rate, dynamic stiffness, local surface orientation, local indentation, stroke spectral response, dynamic friction, static friction, stick-slip, temperature, and relative temperature.

6. The device according to claim 4 or 5, wherein at least one point of the 3D scene comprises multiple components respectively associated with tactile features of different tactile types, and packaging the tactile patch image into a tactile atlas image comprises: For each different tactile type, the tactile patch image is packaged into a tactile atlas image.

7. A decoding method, comprising: - Decode a depth atlas image and at least one haptic atlas image with an associated haptic type, the depth atlas image including a packaged depth patch image and the haptic atlas image including a packaged haptic patch image associated with the haptic type, the depth patch image and the haptic patch image being projections of points in a 3D scene; as well as - The pixels in the haptic atlas image are deprojected by relating the value of each pixel in the haptic atlas image to a point in the 3D scene in relation to the value of the pixel in the depth atlas image.

8. The method of claim 7, further comprising rendering a haptic effect based on values ​​of the haptic atlas image associated with a point in the 3D scene and based on the distance from the point to the virtual camera used to render the 3D scene.

9. The method according to claim 7 or 8, wherein the tactile type belongs to a set of tactile types, the set of tactile types including rate-hardness, contact area expansion rate, dynamic stiffness, local surface orientation, local indentation, stroke spectral response, dynamic friction, static friction, stick-slip, temperature, and relative temperature.

10. The method of claim 7, wherein decoding the depth atlas image and at least one tactile atlas image comprises: Decode multiple tactile atlas images that are associated with tactile features of different tactile types.

11. A decoding device, comprising a processor configured to: - Decoding a depth atlas image and at least one haptic atlas image having an associated haptic type, the depth atlas image comprising a packaged depth patch image and the haptic atlas image comprising a packaged haptic patch image associated with the haptic type, the depth patch image and the haptic patch image being projections of points in a 3D scene; and - The pixels in the haptic atlas image are deprojected by relating the value of each pixel in the haptic atlas image to a point in the 3D scene in relation to the value of the pixel in the depth atlas image.

12. The device of claim 11, wherein the processor is further configured to render a haptic effect based on the value of the haptic atlas image associated with a point in the 3D scene and based on the distance from the point to the virtual camera used to render the 3D scene.

13. The device according to claim 11 or 12, wherein the tactile type belongs to a set of tactile types, the set of tactile types including rate-hardness, contact area expansion rate, dynamic stiffness, local surface orientation, local indentation, stroke spectral response, dynamic friction, static friction, stick-slip, temperature, and relative temperature.

14. The device according to claim 11 or 12, wherein decoding the depth atlas image and at least one tactile atlas image comprises: Decode multiple tactile atlas images that are associated with tactile features of different tactile types.

Images