Method and apparatus for delivering volumetric video content

Abstract

To provide a method, a device and a data stream for signaling and decoding information representing guiding constraints in a volumetric video.SOLUTION: In a 3D scene 50, a data stream includes metadata associated with video data representing a volumetric video. The metadata includes data representing a viewing bounding box, data representing a curved path 51 in the 3D space of the volumetric video, and a set of viewing orientation ranges 53 representing at least one viewing direction range associated with a point on the curved path.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present principles generally relate to the domain of three-dimensional (3D) scenes and volumetric video content. The present document is also understood in the context of encoding, formatting, and decoding of data representing textures and geometry of 3D scenes for rendering volumetric content on an end-user device, such as a mobile device or a head-mounted display (HMD). In particular, the present principles relate to signaling and decoding of information representing navigation constraints in volumetric video. [Background technology]

[0002] This section is intended to introduce the reader to various aspects of the art, which may be related to various aspects of the present disclosure that are described and / or claimed below. This discussion is believed to be helpful in providing the reader with background information to facilitate a better understanding of the various aspects of the present principles. As such, it should be understood that these statements are to be read in this light, and not as admissions of prior art.

[0003] In recent years, an increasing amount of wide field of view content (up to 360°) has become available. Such content is potentially not fully viewable by a user looking at the content on an immersive display device such as a head mounted display, smart glasses, a PC screen, a tablet, a smartphone, etc. This means that at a given moment, the user may only be viewing parts of the content. However, the user may typically be navigated through the content by various means such as head movement, mouse movement, touch screen, voice, etc. It is usually desirable to encode and decode this content.

[0004] Immersive video, also called 360° flat video, allows the user to see everything around him / her by rotating the user's head around a point of gaze. The rotation allows only a three-degree-of-freedom (3DoF) experience. Even if 3DoF video is sufficient for an initial omnidirectional video experience, for example using a head-mounted display device (HMD), 3DoF video can quickly frustrate viewers who expect more degrees of freedom, for example by experiencing parallax. Moreover, 3DoF also allows the user to translate the user's head in three directions, as well as rotate it, which can cause dizziness, since translation is never reproduced in a 3DoF video experience.

[0005] The wide-field content may be, among others, a three-dimensional computer graphic image scene (3D CGI scene), a point cloud, or an immersive video. Many conditions may be used to design such an immersive video: for example, a virtual reality (VR) video, a 360 video, a panoramic video, a 4π stereoscopic video, an immersive video, an omnidirectional video, or a wide-field video.

[0006] Volumetric video, also known as six degrees of freedom (6DoF) video, is an alternative to 3DoF video. When watching 6DoF video, besides rotation, the user can also translate his / her head, and even his / her body, within the content being watched to experience parallax and even volume. Such video considerably deepens the sense of immersion and depth of the scene, and prevents dizziness by providing consistent visual feedback during head translation. The content is created by dedicated sensors, allowing the color and depth of the target scene to be recorded simultaneously. The use of color camera devices combined with photogrammetry techniques is a way to perform such recording, even though technical difficulties remain.

[0007] While 3DoF video includes a sequence of images resulting from the unmapping of texture images (e.g., spherical images encoded according to latitude / longitude projection mapping or equirectangular projection mapping), 6DoF video frames embed information from several viewpoints. These video frames can be considered as a temporal sequence of point clouds resulting from three-dimensional capture. Two types of volumetric videos can be considered as dependent on the viewing state. The first type (i.e., full 6DoF) allows completely free navigation in the video content, whereas the second type (a.k.a. 3DoF+) restricts the user viewing space to a limited volume called the viewing bounding box, allowing limited translational movement of the head, and parallax experience. This second context is a valuable trade-off between the free navigation state and the passive viewing state of a seated viewer. Between the 3DoF+ and 6DoF experiences, it is possible to define the 4DoF+ case as intermediate between 3DoF+ and 6DoF, where the user's displacement is constrained along a curved (1D) path in the 3D scene, with limited horizontal, vertical and depth translation around each path sample. The user can move in a kind of tunnel along a path that guarantees good visual quality. If the user moves the virtual camera out of the tunnel, data to reconstruct a good quality 3D scene may be lost.

[0008] It may be necessary to limit the range of positions allowed for a user's virtual camera and viewing orientations while navigating a given volumetric video content. Otherwise, the user may "leave" the 3D scene when requesting a viewport that cannot be fully rendered due to lack of available visual data. Thus, a solution is needed to inform an end user device of the preferred, preferred, and / or allowed viewer (virtual) positions and viewing orientations when consuming a given volumetric video content. Summary of the Invention

[0009] The following presents a simplified summary of the present principles in order to provide a basic understanding of some aspects of the present principles. This summary is not an exhaustive summary of the present principles. It is not intended to identify key or critical elements of the present principles. The following summary is merely intended to present some aspects of the present principles in a simplified form as a prelude to the more detailed description provided below.

[0010] The present principles relate to a method and device for signaling information representative of guiding limitations in a volumetric video, the method including encoding metadata within a data stream including video data representative of said volumetric video. The metadata may include: - data representing a viewing bounding box; - data representing a curved path in 3D space of said volumetric video; and - data representing at least one viewing direction range associated with a point on said curved path.

[0011] According to another aspect, a second method and a second device are provided for decoding information representing a guiding restriction in a volumetric video. The second method includes decoding metadata from a data stream including video data representing the volumetric video. The metadata includes: - data representing a viewing bounding box; - data representing a curved path in 3D space of said volumetric video; and - data representing at least one viewing direction range associated with a point on said curved path. According to another general aspect of at least one embodiment, there is provided a data stream including video data and associated metadata generated according to any of the described encoding embodiments or variations.

[0012] According to another general aspect of at least one embodiment, a non-transitory computer-readable medium is provided that includes data content generated according to any of the described encoding embodiments or variations.

[0013] According to another general aspect of at least one embodiment, a computer program product is provided that includes instructions that, when executed by a computer, cause the computer to perform any of the described decoding embodiments or variations. [Brief description of the drawings]

[0014] The present disclosure will be better understood and other particular features and advantages will become apparent on reading the following description, which makes reference to the accompanying drawings, in which:

[0015] [Figure 1] 1 illustrates a three-dimensional (3D) model of an object and points of a point cloud corresponding to the 3D model, in accordance with a non-limiting embodiment of the present principles. [Diagram 2] 1 shows a non-limiting example of encoding, transmission, and decoding of data representing a sequence of 3D scenes, in accordance with a non-limiting embodiment of the present principles. [Diagram 3] 9 shows an exemplary architecture of a device that may be configured to implement the methods described in connection with FIGS. 7 and 8, in accordance with a non-limiting embodiment of the present principles. [Figure 4] 1 illustrates an example embodiment of a syntax for a stream when data is transmitted via a packet-based transmission protocol, in accordance with a non-limiting embodiment of the present principles. [Diagram 5] The concept of constrained navigation within a virtual 3D scene is illustrated diagrammatically by depicting a curved path and a spherical bounding box and viewing orientation range at a given position along the path, in accordance with a non-limiting embodiment of the present principles. [Figure 5b]1 shows an example of a collection camera arrangement with five converging cameras arranged on a circular arc, with a tubular viewing space inferred from the collection camera positions and orientations, in accordance with a non-limiting embodiment of the present principles. [Figure 6] 1 illustrates the specific case of a circular path for navigating around a 3D scene with a target object at a central location, in accordance with a non-limiting embodiment of the present principles. [Figure 7] 1 diagrammatically illustrates a method 70 for signaling information representative of navigation limitations in a volumetric video, in accordance with a non-limiting embodiment of the present principles; [Figure 8] 8 diagrammatically illustrates a method 80 for decoding information representative of guiding constraints in a volumetric video, in accordance with a non-limiting embodiment of the present principles; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] The present principles are described more fully below with reference to the accompanying drawings, in which examples of the present principles are shown. However, the present principles may be embodied in many alternative forms and should not be construed as being limited to the examples set forth herein. Thus, while the present principles are susceptible to various modifications and alternative forms, specific examples thereof are shown by way of example in the drawings and described in detail herein. However, there is no intention to limit the present principles to the particular forms disclosed, but rather, it should be understood that the present disclosure covers all modifications, equivalents, and alternatives falling within the spirit and scope of the present principles as defined in the claims.

[0017] The terminology used herein is for the purpose of describing particular examples only and is not intended to limit the present principles. As used herein, the singular forms "a," "an," and "the" are intended to include the plural unless the context clearly indicates otherwise. It will be further understood that the terms "comprises," "comprising," "including," and / or "comprising" as used herein specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Furthermore, when an element is referred to as "responsive to" or "connected" to another element, it may be directly responsive or connected to the other element, or intervening elements may be present. In contrast, when an element is referred to as "directly responsive to" or "directly connected" to another element, there are no intervening elements. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items and may be abbreviated as " / ."

[0018] In this specification, the terms "first", "second", etc. may be used to describe various elements, but it is understood that these elements should not be limited by these terms. These terms are used only to distinguish one element from another element. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element, without departing from the teachings of the present principles.

[0019] Some diagrams include arrows on communication paths to indicate a primary direction of communication, however, it should be understood that communication may occur in a direction opposite to that depicted arrow.

[0020] Some examples are described with reference to block diagrams and operational flow charts, with each block representing a circuit element, module, or portion of code that includes one or more executable instructions for implementing a specified logical function. It should also be noted that in other implementations, the functions noted in the blocks may occur in a different order than the order noted. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved.

[0021] References herein to "by one example" or "in one example" mean that a particular feature, structure, or characteristic described in connection with the example may be included in at least one implementation of the present principles. The appearances of the phrases "by one example" or "in one example" in various places in the specification do not necessarily all refer to the same example, nor do separate or alternative examples necessarily mutually exclude other examples.

[0022] Reference numerals appearing in the claims are by way of example only and shall have no limiting effect on the scope of the claims. Although not expressly described, the present examples and variations may be used in any combination or subcombination.

[0023] FIG. 1 shows a three-dimensional (3D) model 10 of an object and points of a point cloud 11 corresponding to the 3D model 10. The 3D model 10 and the point cloud 11 may correspond to possible 3D representations of an object in a 3D scene, for example, including other objects. The model 10 may be a 3D mesh representation, and the points of the point cloud 11 may be vertices of the mesh. The points of the point cloud 11 may also be points distributed on the surface of the faces of the mesh. The model 10 may also be represented as a splatted version of the point cloud 11, and the surface of the model 10 is created by splatting the points of the point cloud 11. The model 10 may be represented by a number of different representations, such as voxels or splines. FIG. 1 shows that a point cloud may be defined with a surface representation of a 3D object, and a surface representation of a 3D object may be generated from the points of the cloud. As used herein, projecting the points of a 3D object (by the extended points of the 3D scene) onto an image is equivalent to projecting any representation of this 3D object, for example, a point cloud, a mesh, a spline model or a voxel model.

[0024] A point cloud may, for example, be represented in memory as a vector-based structure, where each point has its own coordinates in the reference frame of the viewpoint (e.g., three-dimensional coordinates XYZ, or solid angle and distance from / to the viewpoint (called depth)) and one or more attributes, also called components. Examples of components are color components, which may be represented in various color spaces, for example, RGB (red, green, and blue) or YUV (Y is the luma component and UV are the two chrominance components). A point cloud is a representation of a 3D scene, including objects. The 3D scene can be viewed from a given viewpoint or from various viewpoints. Point clouds may be obtained by many methods, for example: From the capture of real objects, taken with a camera arrangement optionally complemented by an active depth sensing device, From capturing virtual / synthetic objects, photographed by a virtual camera device in the modeling tool, From a mix of both real and virtual objects.

[0025] 2 shows a non-limiting example of encoding, transmission, and decoding of data representing a sequence of 3D scenes. The encoding format may, for example, be compatible with 3DoF, 3DoF+, and 6DoF decoding simultaneously.

[0026] A sequence of 3D scenes 20 is captured. Since the sequence of pictures is a 2D video, the sequence of 3D scenes is a 3D (also called volumetric) video. The sequence of 3D scenes can be provided to a volumetric video rendering device for 3DoF, 3DoF+, or 6DoF rendering and display.

[0027] The sequence of 3D scenes 20 is provided to an encoder 21. The encoder 21 receives as input a 3D scene or a sequence of 3D scenes and provides a bitstream representative of the input. The bitstream may be stored in a memory 22 and / or an electronic data medium and transmitted over a network 22. The bitstream representative of the sequence of 3D scenes may be read from the memory 22 and / or received from the network 22 by a decoder 23. The decoder 23 receives the bitstream as input and provides the sequence of 3D scenes, for example in a point cloud format.

[0028] The encoder 21 may comprise several circuits that implement several steps. In a first step, the encoder 21 projects each 3D scene into at least one 2D picture. A 3D projection is any method of mapping three-dimensional points onto a two-dimensional plane. The use of this type of projection is particularly widespread in computer graphics, engineering, and drafting, since most current methods for displaying graphical data are based on a two-dimensional medium of planes (pixel information from multiple bit planes). The projection circuit 211 provides at least one two-dimensional frame 2111 for the sequence of 3D scenes 20. The frame 2111 includes color and depth information representing the 3D scene projected in the frame 2111. In a variant, the color and depth information are encoded in two separate frames 2111 and 2112.

[0029] The metadata 212 is used and updated by the projection circuitry 211. The metadata 212 includes information about the projection operation (e.g., projection parameters) as well as information about how color and depth information is organized within the frames 2111 and 2112, as described in connection with Figures 5-7.

[0030] A video encoding circuit 213 encodes the sequence of frames 2111 and 2112 as a video. The pictures of the 3D scene 2111 and 2112 (or a sequence of pictures of a 3D scene) are encoded into a stream by the video encoder 213. The video data and metadata 212 are then encapsulated into a data stream by a data encapsulation circuit 214.

[0031] The encoder 213 is compliant with the following encoders, for example: -JPEG, Specification ISO / CEI 10918-1 UIT-T Recommendation T.81, https: / / www.itu.int / rec / T-REC-T.81 / en, -AVC, also known as MPEG-4 AVC or h264. Specified in both UIT-T H.264 and ISO / CEI MPEG-4 Part 10 (ISO / CEI 14496-10), http: / / www.itu.int / rec / T-REC-H.264 / en; HEVC (whose specification can be found on the ITU website, Recommendation T, H-series, h265, http: / / www.itu.int / rec / T-REC-H.265-201612-I / en); -3D-HEVC (an extension to HEVC, the specification of which can be found on the ITU website, Recommendation T, H-series, h265, http: / / www.itu.int / rec / T-REC-H.265-201612-I / en Annexes G and I), -VP9, developed by Google, or -AV1 (AOMedia Video 1) developed by the Alliance for Open Media.

[0032] The data stream is stored by a decoder 23, for example in a memory accessible via a network 22. The decoder 23 comprises various circuits implementing the various steps of decoding. The decoder 23 receives as input the data stream generated by the encoder 21 and provides a sequence of 3D scenes 24 to be rendered and displayed by a volumetric video display device such as a head mounted device (HMD). The decoder 23 obtains the stream from a source 22. For example, the source 22 belongs to a set including: - local memory, such as video memory or RAM (or Random Access Memory), Flash memory, ROM (or Read Only Memory), hard disk, -Storage interfaces, e.g. interfaces to mass storage, RAM, flash memory, ROM, optical disks or magnetic supports; a communication interface, such as a wired interface (e.g. a bus interface, a wide area network interface, a local area network interface) or a wireless interface (such as an IEEE 802.11 interface or a Bluetooth interface), and A user interface, such as a graphical user interface, that allows a user to input data.

[0033] The decoder 23 comprises a circuit 234 for extracting data encoded in the data stream. The circuit 234 receives the data stream as input and provides metadata 232 corresponding to the metadata 212 encoded in the stream and a two-dimensional video. The video is decoded by a video decoder 233, which provides a sequence of frames. The decoded frames include color and depth information. In a variant, the video decoder 233 provides two sequences of frames, one including color information and the other including depth information. The circuit 231 uses the metadata 232 to backproject the color and depth information from the decoded frames to provide a sequence of 3D scenes 24. The sequence of 3D scenes 24 corresponds to a sequence of 3D scenes 20, which may suffer from a loss of precision associated with encoding and video compression as a 2D video.

[0034] Fig. 3 shows an example architecture of a device 30 that may be configured to implement the methods described in relation to Fig. 7 and Fig. 8. The encoder 21 and / or the decoder 23 of Fig. 2 may implement this architecture. Alternatively, each circuit of the encoder 21 and / or the decoder 23 may be a device according to the architecture of Fig. 3, linked to each other, for example, via their bus 31 and / or via an I / O interface 36.

[0035] The device 30 comprises the following elements linked together by a data and address bus 31: a microprocessor 32 (or CPU), for example a DSP (or Digital Signal Processor); -ROM (or read-only memory) 33, -RAM (or Random Access Memory) 34, -Storage interface 35, an I / O interface 36 for receiving data sent by an application, and A power source, e.g. a battery.

[0036] According to an example, the power source is external to the device. In each memory of the above memories, the word "register" used in this specification may correspond to a small area (a few bits) or a very large area (e.g., the entire program or a large amount of received or decoded data). The ROM 33 includes at least programs and parameters. The ROM 33 may store algorithms and instructions for implementing the technology according to the present principles. When switched on, the CPU 32 uploads the program in the RAM and executes the corresponding instructions.

[0037] The RAM 34 contains in registers the programs executed by the CPU 32 and uploaded after the device 30 is switched on, contains in registers the input data, contains in registers intermediate data for different states of the method, and contains in registers other variables used to execute the method.

[0038] The implementations described herein may be implemented, for example, in a method or process, an apparatus, a computer program product, a data stream, or a signal. Even when discussed only in the context of a single form of implementation (e.g., discussed only as a method or device), the implementations of the features discussed may also be implemented in other forms (e.g., programs). An apparatus may be implemented, for example, in appropriate hardware, software, and firmware. The methods may be implemented, for example, in an apparatus, for example, in a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, for example, computers, mobile phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate communication of information between end users.

[0039] According to an example, the device 30 is configured to implement the method described in relation to Figures 7 and 8 and belongs to a set comprising: -Mobile devices, -Communication devices, -Gaming devices, - a tablet (or tablet computer), -Laptop, -Still camera, -Video camera, - coding chip, A server (eg, a broadcast server, a video-on-demand server, or a web server).

[0040] FIG. 4 shows an example of an embodiment of the syntax of a stream when data is transmitted via a packet-based transmission protocol. FIG. 4 shows an exemplary structure 4 of a volumetric video stream. The structure resides in a container that organizes the stream into individual syntax elements. The structure may include a header section 41, which is a set of data common to every syntax element of the stream. For example, the header section includes a portion of metadata about the syntax elements, describing the nature and role of each of them. The header section may also include a portion of the metadata 212 of FIG. 2, for example, the coordinates of a central viewpoint used to project points of the 3D scene into frames 2111 and 2112. The structure includes a payload that includes an element of syntax 42 and at least one element of syntax 43. Syntax element 42 includes data representing color and depth frames. The images may have been compressed according to a video compression method.

[0041] Syntax 43 elements are part of the payload of the data stream and may contain metadata about how frames of Syntax 42 elements are encoded, for example, parameters used to project and pack points of a 3D scene into frames. Such metadata may be associated with each frame of video, or with groups of frames (also known as Groups of Pictures (GoPs) in video compression standards).

[0042] Figure 5 illustrates the concept of constrained navigation in a virtual 3D scene diagrammatically by depicting one curved path, as well as the spherical bounding box and viewing orientation range at a given position along this path. Navigation constraints are determined per content during the content creation stage, with the added benefit of potentially adding subjective (e.g., artistic) constraints to the objective constraints of the 3D geometry.

[0043] In accordance with the present principles, an arrangement of high-level syntax elements that explicitly describe a subset of viewing positions and orientations associated with a given volumetric video content representing a 3D scene 50. The information signaled includes the following elements: - a curved path 51 (or a set of curved paths) in 3D space, - Base bounding box volume 52, - a set of viewing orientation ranges 53 indexed by position along the curved path The combination of these elements describes a virtual displacement along a path that is suitable, preferred, and / or acceptable for good quality 3D scene reconstruction. At each location along the path, small translational movements within a bounding box are allowed and the viewing orientation is restricted within a given angular range. This can be described as a 4DoF+ virtual guidance "tunnel".

[0044] According to a first embodiment, a set of curvilinear guidance paths is defined by a set of 3D points with the following syntax and semantics: aligned(8)class NavigationPathSet() { ViewingBoundingBox; unsigned int(8)num_paths; for(n=0;n <num_paths;n++){ unsigned int(32)num_points[n]; for(i=0;i <num_points[n];i++){ unsigned int(32)X[n][i]; unsigned int(32)Y[n][i]; unsigned int(32)Z[n][i]; signed int(32) phi_min[n][i]; signed int(32) phi_max[n][i]; signed int(32)theta_min[n][i]; signed int(32)theta_max[n][i]; } } } aligned(8)class ViewingBoundingBox() { unsigned int(8)shape_type; unsigned int(32)first_dimension; unsigned int(32)second_dimension; unsigned int(32)third_dimension; }

[0045] num_paths specifies the number of curvilinear navigation paths defined for navigating the content. A value of 0 indicates that the entire 3D space can be navigated.

[0046] num_points specifies the number of 3D points sampled along the curve guidance path.

[0047] X[n][i], Y[n][i], Z[n][i] are fixed-point values ​​(e.g., 16.16 values) that define the 3D coordinates of the ith sample along the nth path in the global coordinate system of the 3D scene. The points are ordered according to the abscissas of the curve along the path. The nth path is thus defined between two 3D end points (X[n][0], Y[n][0], Z[n][0]) and (X[n][num_points[n]-1], Y[n][num_points[n]-1], Z[n][num_points[n]-1]). For example, the curved path may be a piecewise path of straight lines between two consecutive points in the list and from the last point to the first point in the list. In a variant, the curved path may be determined by using a quadratic or cubic Bézier curve, by using three or four consecutive points.

[0048] phi_min[n][i], phi_max[n][i] and theta_min[n][i], theta_max[n][i] are 2 16 The minimum and maximum azimuth and elevation angles that define the viewing orientation at the ith point along the nth path, in degrees. The azimuth angle value can be, for example, -180*2 inclusive. 16 ~180*2 16 The elevation angle is in the range of -1, for example -90*2, inclusive. 16 ~90*2 16 The azimuth and elevation values ​​may be expressed in different units of measure, for example, radians.

[0049] shape_type specifies the shape of the viewing bounding box that the user is allowed to move slightly at a given location along the guided path. A shape_type of 0 indicates a sphere, and the radius of the sphere is specified by first_dimension. This syntax makes it possible to define more complex ellipsoidal or rectangular 3D volume shapes.

[0050] According to another embodiment, the lightweight viewing space 54 specification is obtained by inferring the guiding path 51 from the parameters of the collecting camera associated with the volumetric content. The 3D points that sample the guiding path are the 3D positions of the collecting camera. The viewing direction (center azimuth and elevation) at each sample position is the viewing direction of the collecting camera at that position. Since the collecting external camera parameters are already part of the metadata associated with the volumetric video content, the additional metadata to send to specify the viewing space is reduced to the shape and size of the viewing box at each sample position, and the azimuth and elevation range around the viewing direction. An example of the syntax and associated semantics is as follows: aligned(8)class NavigationPathFromCameraRig() { ViewingBoundingBox; unsigned int(8)num_paths; for(n=0;n <num_paths;n++){ unsigned int(32)num_cams[n]; for(i=0;i <num_cams[n];i++){ unsigned int(32)cam_idx[n][i]; unsigned int(32)phi_range[n][i]; unsigned int(32)theta_range[n][i]; } } }

[0051] num_cams[n] is the size of the subset of acquisition cameras used to sample the nth guidance path. cam_idx[n][i] is the index of the ith camera along the nth guidance path (in the list of all acquisition cameras). phi_range[n][i] and theta_range[n][i] are the range of azimuth and elevation deviation at the ith position along the nth path around the azimuth and elevation of the cam_idx[n][i]th acquisition camera.

[0052] Such a viewing space specification is particularly suited to volumetric content captured with a camera arrangement positioned along the arc of a circle. Figure 5b shows an example of a viewing space 54 associated with 3D content captured from such an arrangement, where the ten cameras are arranged into five converging camera pairs 55 to capture the 3D scene, and the tubular viewing space 54 is specified by sampling five spheroids at the locations of every other camera.

[0053] 6 shows the specific case of a circular path for navigation around a 3D scene with a target object at a central location. In this use case, the user is allowed to walk around the 3D scene 50 on a circular path 61 in a horizontal plane at a given height value in the global coordinate system with an inward field of view 63. Displacements in the bounding box 52 are allowed as in the first embodiment.

[0054] An example of the syntax and semantics may be as follows: aligned(8)class CircularNavigationPath() { ViewingBoundingBox; signed int(32)center_x; signed int(32)center_y; signed int(32)center_z; unsigned int(32)radius; signed int(32)delta_phi; signed int(32) theta_min; signed int(32) theta_max; }

[0055] center_x, center_y, center_z are fixed point values ​​(e.g. 16.16 values) that define the 3D coordinates of the center of the circular path.

[0056] radius is a fixed-point value that defines the radius of the circular path.

[0057] phi_range is 2 16 The angular range, in degrees, that defines the azimuthal viewing orientation at any point on the circular path (relative to the radial direction). The value of delta_phi ranges from 0 to 360*2, inclusive. 16 The range is -1.

[0058] theta_min, theta_max are 2 16 The minimum and maximum viewing elevation angle at any point on the circular path, in degrees. Elevation values ​​are within the limits of -90*2, inclusive. 16 ~90*2 16 The range is as follows.

[0059] More generally, a curvilinear guidance path may be defined by a parametric 3D curve: X=f(s), Y=g(s), Z=h(s), where s is a scalar value. Such a variety of functional parameterizations is preferred in accordance with the present principles. This general approach is particularly adapted for typical 4DoF+ experiences, such as rendering volumetric videos of sporting or concert events. For such 4DoF+ videos, the preferred, preferred, and / or allowed paths have simple shapes that can be parameterized using a small number of parameters, such as, for example, an oval shape for a stadium stand, or a rectangle around a stadium stage or field.

[0060] FIG. 7 diagrammatically shows a method 70 for signaling information representing limitations of navigation in a volumetric video. In step 71, volumetric video data is acquired. Data representing a curved path and a viewing direction range of points of the curved path are acquired simultaneously. The curved path associated with a viewing bounding box represents limitations of navigation in a 3D space including the 3D scene of the volumetric video. According to an embodiment of the present principles, these data may be represented via different data structures, as described in relation to FIG. 5 and FIG. 6. In step 72, the volumetric video data is encoded in a data stream in association with metadata including the limitations of navigation acquired in step 71. In step 73, the data stream encoded in step 72 may be stored in a non-transitory medium or transmitted to a client device.

[0061] According to one embodiment, the proposed restricted guidance path messages are encoded in a dedicated SEI message (Supplementary Enhancement Information) in the video stream. According to another embodiment, the proposed restricted guidance path messages are encoded at the container level using the ISO Base Media File Format. Adding such guidance path messages to the metadata of the volumetric video stream allows the renderer to restrict the virtual guidance to viewing positions and orientations that are consistent with the encoded 3D scene content, thus ensuring the quality of the immersive experience.

[0062] FIG. 8 diagrammatically shows a method 80 for decoding information representative of navigating limitations in a volumetric video. In step 81, a data stream including video data of a volumetric video is obtained from a source. The data stream also includes metadata associated with the volumetric video and a representation of navigating limitations in a 3D space including the 3D scene of the volumetric video. In step 82, the video data and the metadata are decoded. The metadata includes data representing a viewing bounding box, data representing a curved path in the 3D space of the volumetric video, and data representing at least one viewing direction range associated with a point on the curved path. These data may be represented by different data structures according to an embodiment of the present principles, as shown in relation to FIG. 5 and FIG. 6. In step 83, information representing navigating limitations in the 3D space of the 3D scene is retrieved by using the decoded metadata, and this information is used by the renderer. For example, the renderer may warn the user when the user is about to leave a tunnel where good quality rendering is guaranteed. The renderer may also prevent the virtual camera from moving outside the description volume or alter the rendering (e.g., by fading the image) when the user is moving outside the preferred path.

[0063] The implementations described herein may be implemented, for example, in a method or process, an apparatus, a computer program product, a data stream, or a signal. Even when discussed only in the context of a single form of implementation (e.g., discussed only as a method or device), the implementations of the features discussed may also be implemented in other forms (e.g., programs). An apparatus may be implemented, for example, in appropriate hardware, software, and firmware. For example, the methods may be implemented in an apparatus such as, for example, a processor, which broadly refers to a processing device including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices such as, for example, smartphones, tablets, computers, mobile phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate the transfer of information between end users.

[0064] Implementations of the various processes and features described herein may be embodied in a variety of different devices or applications, particularly, for example, devices or applications associated with data encoding, data decoding, view generation, texture processing, and other image processing, and associated texture and / or depth information. Examples of such devices include encoders, decoders, post-processors that process output from the decoders, pre-processors that provide input to the encoders, video coders, video decoders, video codecs, web servers, set-top boxes, laptops, personal computers, mobile phones, PDAs, and other communication devices. As should be clear, the devices may be portable, even mounted on moving vehicles.

[0065] Also, the methods may be implemented by instructions executed by a processor, and such instructions (and / or data values ​​produced by the implementations) may be stored on a processor-readable medium, such as, for example, an integrated circuit, a software carrier, or other storage device, such as, for example, a hard disk, a compact diskette ("CD"), an optical disk (such as, for example, a DVD, often referred to as a digital versatile disk or digital video disk), a random access memory ("RAM"), or a read-only memory ("ROM"). The instructions may form an application program tangibly embodied on the processor-readable medium. The instructions may be in, for example, hardware, firmware, software, or a combination. The instructions may be found, for example, in an operating system, a separate application, or a combination of the two. Thus, a processor may be characterized as both a device configured to perform a process, and a device that includes a processor-readable medium (such as a storage device) having instructions for performing a process. Furthermore, the processor-readable medium may store data values ​​produced by the implementations in addition to or in place of instructions.

[0066] As will be apparent to one skilled in the art, implementations may generate various signals formatted to carry information, which may be stored or transmitted, for example. Information may include, for example, instructions to perform a method or data generated by one of the described implementations. For example, a signal may be formatted to convey rules for writing or reading syntax of the described embodiments as data, or may be formatted to convey actual syntax values ​​described by the described embodiments as data. Such a signal may be formatted, for example, as an electromagnetic wave (e.g., using a radio frequency portion of the spectrum) or as a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information that the signal carries may be, for example, analog or digital information. The signal may be transmitted over a wide variety of different wired or wireless links, as is known. The signal may be stored in a processor-readable medium.

[0067] Several implementations have been described. Nevertheless, it will be understood that various modifications may be made. For example, elements of different implementations may be combined, supplemented, modified, or deleted to produce other implementations. Also, those skilled in the art will understand that other structures and processes may be substituted for the disclosed structures and processes, with the resulting implementations performing at least substantially the same function(s) and in at least substantially the same manner(s) to achieve at least substantially the same result(s) as the disclosed implementations. Accordingly, these and other implementations are contemplated by the present application.

Claims

1. 1. A method of signaling information representative of a navigation restriction in a volumetric video, the method comprising: encoding metadata in a data stream including video data representative of the volumetric video, the metadata comprising: - data representing a viewing bounding box; - data representing a curvilinear path in the 3D space of said volumetric video; - data representing at least one viewing direction range associated with a point on said curved path.

2. The method of claim 1 , wherein the data representing a curved path comprises a list of 3D points, the points in the list being associated with at least one range of viewing directions.

3. The method of claim 1 , wherein the data representing a curved path includes parameters representing a parametric 3D curve, and data representing at least one viewing direction range is associated with an origin of the curved path.

4. The method of any one of claims 1 to 3, wherein the metadata comprises data representing at least two curved paths in the 3D space of the volumetric video.

5. The method of claim 1 , wherein the data representing a curved path comprises a list of references to camera parameters associated with at least one range of viewing directions.

6. 1. A device for signaling information representative of a navigation restriction in a volumetric video, the device comprising: a processor configured to encode metadata in a data stream including video data representative of the volumetric video, the metadata comprising: - data representing a viewing bounding box; - data representing a curvilinear path in the 3D space of said volumetric video; - data representing at least one viewing direction range associated with a point on said curved path.

7. The device of claim 6 , wherein the data representing a curved path comprises a list of 3D points, the points in the list being associated with at least one range of viewing directions.

8. The device of claim 6 , wherein the data representing a curved path includes parameters representing a parametric 3D curve, and wherein data representing at least one viewing direction range is associated with an origin of the curved path.

9. A device according to any one of claims 6 to 8, wherein the metadata comprises data representing at least two curved paths in the 3D space of the volumetric video.

10. The method of claim 6 , wherein the data representing a curved path comprises a list of references to camera parameters associated with at least one range of viewing directions.

11. 1. A method for decoding information representative of a navigating constraint in a volumetric video, the method comprising: decoding metadata from a data stream including video data representative of the volumetric video, the metadata comprising: - data representing a viewing bounding box; - data representing a curvilinear path in the 3D space of said volumetric video; - data representing at least one viewing direction range associated with a point on said curved path.

12. The method of claim 11 , wherein the data representing a curved path comprises a list of 3D points, the points of the list being associated with at least one range of viewing directions.

13. The method of claim 11 , wherein the data representing a curved path includes parameters representing a parametric 3D curve, and data representing at least one viewing direction range is associated with an origin of the curved path.

14. A method according to any one of claims 11 to 13, wherein the metadata comprises data representing at least two curved paths in the 3D space of the volumetric video.

15. The method of claim 11 , wherein the data representing a curved path comprises a list of references to camera parameters associated with at least one range of viewing directions.

16. 1. A device for decoding information representing a navigating restriction in a volumetric video, the device comprising: a processor configured to decode metadata in a data stream including video data representing the volumetric video, the metadata comprising: - data representing a viewing bounding box; - data representing a curvilinear path in the 3D space of said volumetric video; - data representing at least one viewing direction range associated with a point on said curved path.

17. The device of claim 16 , wherein the data representing a curved path comprises a list of 3D points, the points in the list being associated with at least one range of viewing directions.

18. The device of claim 16 , wherein the data representing a curved path includes parameters representing a parametric 3D curve, and wherein data representing at least one viewing direction range is associated with an origin of the curved path.

19. A device according to any one of claims 16 to 18, wherein the metadata comprises data representing at least two curved paths in the 3D space of the volumetric video.

20. The device of claim 16 , wherein the data representing a curved path comprises a list of references to camera parameters associated with at least one range of viewing directions.

21. A data stream comprising video data representative of the volumetric video and metadata representative of navigation constraints in the volumetric video, the metadata comprising: - data representing a viewing bounding box; - data representing a curvilinear path in the 3D space of said volumetric video; - data representing at least one viewing direction range associated with a point on said curved path.

22. 22. The data stream of claim 21, wherein the data representing a curved path comprises a list of 3D points, the points of the list being associated with at least one range of viewing directions.

23. 22. The data stream of claim 21, wherein the data representing a curved path includes parameters representing a parametric 3D curve, and data representing at least one viewing direction range is associated with an origin of the curved path.

24. A data stream according to any one of claims 21 to 23, wherein the metadata comprises data representing at least two curved paths in the 3D space of the volumetric video.

25. The method of claim 21 , wherein the data representing a curved path comprises a list of references to camera parameters associated with at least one range of viewing directions.

26. A non-transitory computer readable medium comprising data content generated according to the method of any one of claims 1 to 5 or by the apparatus of any one of claims 6 to 11 for reproduction using a processor.

27. A computer program product comprising instructions which, when said program is executed by a computer, cause said computer to carry out the method according to any one of claims 1 to 5.

28. A computer program product comprising instructions which, when said program is executed by a computer, cause said computer to carry out the method according to any one of claims 11 to 15.

Images