Method for transmitting and rendering 3d scene, method for generating patch, and corresponding devices and computer programs

Abstract

The invention discloses a method and apparatus for transmitting and rendering a 3D scene. The method for transmission includes: dividing a space into m angular sectors, the m angular sectors each corresponding to an angular distance from a viewport, and dividing the space into n depth ranges; obtaining (11) at least one first patch generated from a first view of the 3D scene, said at least one first patch comprising a texture component and a depth component; obtaining (12) at least one atlas generated from at least one second view of the 3D scene, the at least one atlas is constructed by packing together at least one second patch generated for at least one point of one of the second views that is invisible in another view of the 3D scene and that belongs to the same one of the m angular sectors and to the same one of the n depth ranges, at least one of m or n is greater than or equal to 2, the at least one second patch comprising a texture component and a depth component, where each of the at least one first patch and the at least one second patch is based on at least one of sector and depth; generating (13) a first subset of streams comprising m 'convection and a second subset of streams comprising m' x n 'convection; and transmitting (14) the first subset of streams and the second subset of streams to the terminal.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of European Patent Application No. 19305939.1, filed on July 15, 2019, the contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the field of video processing, and more specifically to the field of volumetric video content. This disclosure provides a technique for adaptively transmitting a representation of a 3D scene to a terminal by taking into account at least one terminal-based delivery standard. Such adaptive transmission can be used to enhance the rendering of the 3D scene, for example, for immersive rendering on terminals such as mobile or head-mounted displays (HMDs).

[0004] The present disclosure may be applicable to any application that must deliver volumetric content, particularly 3DoF+ video content. Background Art

[0005] This section is intended to introduce various aspects of the art that may be related to various aspects of the present disclosure described and / or claimed below. This discussion helps provide background information to facilitate a better understanding of the various aspects of the present disclosure. Therefore, it should be understood that these statements should be read in this light, and not as admissions of prior art.

[0006] Immersive video (also known as 360° planar video) allows users to see everything around them by rotating their heads around a stationary viewpoint. Rotation only allows for a three-degree-of-freedom (3DoF) experience. Even though 3DoF video is sufficient for a first-time omnidirectional video experience (e.g., using an HMD), it can quickly become frustrating for viewers who expect more freedom (e.g., by experiencing parallax). Furthermore, 3DoF can also cause motion sickness because users not only rotate their heads but also translate them in three directions, movements that are not reproduced in a 3DoF video experience.

[0007] Volumetric video, also known as 6 degrees of freedom (6DoF) video, is an alternative to 3DoF video. When watching 6DoF video, users can pan their head, and even their body, within the content they are watching, in addition to rotating, and experience parallax and even volume. This type of video significantly increases immersion and the perception of scene depth, and prevents motion sickness by providing consistent visual feedback during head panning.

[0008] An intermediate approach between 3DoF and 6DoF, called 3DoF+, has also been proposed. This video-based approach (disclosed, for example, in WO2019 / 055389) involves transmitting volumetric input information as a combination of color and depth patches. Each patch is generated by a continuous spherical 2D projection / mapping of a subsection of the original 3D scene.

[0009] Basically, this decomposition peels / decomposes the scene into: (1) a central patch containing the part of the scene visible from the main central viewpoint; and (2) peripheral patches embedded with supplementary information that is not visible from the central viewpoint.

[0010] In order to transmit 3DoF+ video content, the following two video frames are defined: (1) a color frame that carries both the texture of the central patch and the texture of the peripheral patches to carry disparity information; and (2) a depth frame that carries both the depth of the central patch and the depth of the peripheral patches to carry disparity information.

[0011] To limit the amount of decoder context, color and depth frames have a fixed size corresponding to the size of the center patch (e.g., 4K pixels × 2K pixels) plus an additional room size to carry disparity information from the source viewpoint in all 360° directions.

[0012] However, packing disparity information into fixed-size frames is sufficient for simple scenes without many hidden objects, but can be inefficient for transmitting complex scenes where many hidden objects require a large amount of data for peripheral video patches and disparity information. In addition, existing 3DoF+ technologies suffer from latency when rendering 3D scenes. For example, this can occur when an HMD user quickly turns their head in one direction. According to existing technologies, the rendering terminal must wait for color frames before displaying anything, and wait for depth frames for volume rendering. Summary of the Invention

[0013] Therefore, there is a need for a new technique for transmitting 3D scenes that overcomes at least one of the shortcomings of known techniques.

[0014] According to one aspect of the present disclosure, a method for transmitting a representation of a 3D scene to a terminal is disclosed. The method includes: dividing a space into m angular sectors, each of the m angular sectors corresponding to an angular distance from a viewport, and dividing the space into n depth ranges; obtaining at least one first patch generated from a first view of the 3D scene, the at least one first patch including a texture component and a depth component; obtaining at least one atlas generated from at least one second view of the 3D scene, the at least one atlas being constructed by packing together at least one second patch generated for at least one point of one of the second views that is not visible in another view of the 3D scene and belongs to the same angular sector in the m angular sectors and the same depth range in the n depth ranges, at least one of m or n. Greater than or equal to 2, the at least one second patch includes a texture component and a depth component, wherein each of the at least one first patch and the at least one second patch is based on at least one of a sector and a depth; generating the following items according to at least one terminal-based delivery standard: a first stream subset, the first stream subset comprising m′ pairs of streams from the one or more first patches, m′ being the entirety or a subset of m angular sectors; and a second stream subset, the second stream subset comprising m′×n′ pairs of streams from the at least one atlas, wherein m′≤m and n′≤n, each pair of streams comprising a stream for transmitting a texture component and a stream for transmitting a depth component, and transmitting the first stream subset and the second stream subset to the terminal.

[0015] According to the present disclosure, it is therefore possible to transmit only a subset of streams for transmitting depth components and texture components to a terminal, taking into account at least one terminal-based delivery standard.

[0016] More specifically, for at least one second view, points (or voxels) of the second view that are not visible in another view (the first view or the other second view) may be identified, and the depth ranges and / or angular sectors to which these points belong may be determined. Therefore, second patches obtained from these points that can be used to convey disparity information may be grouped in atlases, with at least one atlas for each depth range and / or each angular sector.

[0017] In this way, only the disparity information that is "useful" to the terminal (user) can be transmitted, rather than all disparity information. For example, only the disparity information corresponding to the terminal user's viewpoint can be transmitted, or only the disparity information corresponding to the minimum depth range from the user's viewpoint can be transmitted, especially when the available bandwidth of the communication channel with the terminal is limited.

[0018] Therefore, at least one embodiment of the present disclosure aims to solve the problem of fixed-size frames according to the prior art. In fact, only useful disparity information can be transmitted, thereby solving the problem of complex scenes or heterogeneous scenes, where some sectors of the 360° space have poor disparity information while other sectors have a large amount of disparity information, which may not be suitable for the additional room size.

[0019] At least one embodiment of the present disclosure is also intended to address the latency issue in rendering. In fact, only useful disparity information can be transmitted, thereby achieving fast rendering.

[0020] According to another embodiment, a corresponding device for transmitting a representation of a 3D scene to a terminal is disclosed. Such a device may be particularly suitable for implementing the method for transmitting a representation of a 3D scene described above. For example, such a device is a server.

[0021] The present disclosure also discloses a method for rendering a 3D scene on a terminal. The method includes: dividing a space into m angular sectors, each of the m angular sectors corresponding to an angular distance from a viewport, and dividing the space into n depth ranges; receiving a first stream subset and a second stream subset generated according to at least one terminal-based delivery standard, the first subset including m′ pairs of streams generated from at least one first patch and the second subset including m′×n′ pairs of streams generated from at least one atlas, each pair of streams including a stream for transmitting a texture component and a stream for transmitting a depth component, m′ being the entirety or a subset of the m angular sectors and n′ being the entirety or a subset of the n depth ranges, the at least one first patch being generated from a first view of the 3D scene and including a texture component and a depth component. degree component, the at least one atlas being generated from at least one second view of the 3D scene and constructed by packing together at least one second patch generated for at least one point of one of the second views that is not visible in the other view of the 3D scene and belonging to the same angular sector of m angular sectors and the same depth range of n depth ranges, at least one of m or n being greater than or equal to 2, the at least one second patch comprising a texture component and a depth component, where m′≤m and n′≤n, wherein each of the at least one first patch and the at least one second patch is based on at least one of a sector and a depth; and constructing a representation of the 3D scene from the first subset of streams and the second subset of streams.

[0022] In particular, such a method may be implemented for rendering a 3D scene transmitted by means of a method for transmitting a representation of a 3D scene as described above.

[0023] As already mentioned, the method according to at least one embodiment allows for fast rendering of 3D scenes, since the terminal may only receive "useful" disparity information.

[0024] According to another embodiment, a corresponding terminal for rendering a 3D scene is disclosed. Such a terminal (also referred to as a rendering device) may be particularly suitable for implementing the method for rendering the above-mentioned 3D scene. For example, such a device may be an HMD, a mobile phone, a tablet computer, etc.

[0025] The present disclosure also discloses a method for generating a patch representing a 3D scene. The method comprises: obtaining a first view of a 3D scene from a first view; generating at least one first patch from the first view, the at least one first patch comprising a texture component and a depth component; obtaining at least one second view of the 3D scene from at least one second viewpoint; and spatially partitioning the 3D scene into m angular sectors, each corresponding to a distance from a given viewport, and into n depth ranges, wherein for at least one of the second views, the method further comprises: identifying at least one point of the second view that is not visible in another view of the 3D scene; determining a depth range to which the at least one point belongs; for at least one of the m angular sectors and for at least one of the n depth ranges, at least one of m or n being greater than or equal to 2, generating at least one second patch from the second view for points belonging to the angular sector and the depth range, the at least one second patch comprising a texture component and a depth component, wherein each of the at least one first patch and the at least one second patch is based on at least one of a sector and a depth; and constructing at least one atlas by packing together at least one of the second patches generated for points belonging to the same angular sector and the same depth range.

[0026] In particular, this method may be implemented to generate patches and atlases obtained by the method for transmitting a representation of a 3D scene as described above.

[0027] According to a first embodiment, the method for generating patches and the method for transmitting a representation of a 3D scene may be implemented by the same device (eg a server).

[0028] According to a second embodiment, the method for generating patches and the method for transmitting a representation of a 3D scene may be implemented by two different devices that can communicate by wire or wirelessly according to any communication protocol.

[0029] Therefore, a corresponding device for generating a patch representing a 3D scene according to a second embodiment is disclosed.Such a device may be particularly suitable for implementing a method for generating a patch representing a 3D scene as described above.

[0030] Another aspect of the present disclosure relates to at least one computer program product downloadable from a communication network and / or recorded on a computer-readable and / or processor-executable medium, the at least one computer program product comprising software code adapted to execute a method for transmitting a representation of a 3D scene, a method for rendering a 3D scene, or a method for generating a patch representing a 3D scene, wherein the software code is adapted to perform at least one step of the above-mentioned method.

[0031] In addition, another aspect of the present disclosure relates to a non-transitory computer-readable medium including a computer program product recorded thereon and capable of being executed by a processor, the computer program product including program code instructions for implementing a method for transmitting a representation of a 3D scene, a method for rendering a 3D scene, or a method for generating a patch representing the previously described 3D scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The present disclosure will be better understood and explained by the following embodiments and execution examples, but is in no way limiting, with reference to the accompanying drawings, in which:

[0033] Figure 1 is a flow chart illustrating a method for transmitting a representation of a 3D scene according to an embodiment of the present disclosure;

[0034] Figure 2 is a flowchart illustrating a method for generating a patch representing a 3D scene according to an embodiment of the present disclosure;

[0035] Figure 3 is a flowchart illustrating the main steps of a method for processing a 3D scene according to an embodiment of the present disclosure;

[0036] Figure 4 shows the position of a camera for generating a patch according to the prior art;

[0037] Figure 5 Examples of patches generated according to prior art stripping techniques are given;

[0038] Figure 6A and Figure 6B Give examples of patches generated according to the present disclosure;

[0039] Figure 7 Shows an example of depth-first representation;

[0040] Figure 8 Examples of sector and depth-first representations are shown; and

[0041] Figure 9is a block diagram of an apparatus for implementing at least one of a method for generating a patch representing a 3D scene, a method for transmitting a representation of a 3D scene, or a method for rendering a 3D scene according to at least one embodiment of the present disclosure.

[0042] In the figures, the blocks represented are purely functional entities which do not necessarily correspond to physically separate entities, that is, they can be developed in the form of software, hardware, or implemented in one or more integrated circuits, including one or more processors. DETAILED DESCRIPTION

[0043] It should be understood that the drawings and descriptions of the present disclosure have been simplified to illustrate elements relevant to a clear understanding of the present disclosure, while many other elements found in a typical transmission or rendering device have been eliminated for clarity.

[0044] The general principles of the present disclosure will now be discussed.

[0045] The present disclosure proposes a technique for volumetric data organization and associated terminal-dependent delivery mode (eg, viewport-dependent).

[0046] According to at least one embodiment, this technology provides progressive rendering on the terminal, thereby reducing latency by delivering the first basic elements for real-time volume rendering.

[0047] This technique relies on a new approach to constructing patches containing disparity information (volumetric data), allowing patches to be constructed based on the viewpoint (e.g., of a real or virtual camera) and / or the point position in space (i.e., the position of the point / voxel in the 3D scene from the viewpoint): the farther away, the less important. The criterion for prioritizing volumetric data elements (point positions) can be depth (distance from the viewpoint), angular sector (distance from the center of the delivered viewport), or a combination of both. For example, the client can first download the video information required for a basic flat 360° rendering, and further download data to improve the disparity experience based on the available throughput.

[0048] In accordance with at least one embodiment, volumetric data is thus organized in a list of video frames that may be of the same size (e.g., 4K) but with different patch arrangements, allowing rendering of each sector of 360° space and each distance to the source viewpoint (e.g., near to far).

[0049] The volumetric data may be contained in a variable list of patches, with the contents of the patches being distributed over the transmission of consecutive video frames for a given spatial sector.

[0050] To enable switching from one viewpoint to another while optimizing the amount of received data, volumetric content can be segmented into chunks of fixed duration. The chunks stored on the server show a three-level organization: per time interval, per sector, and per depth (i.e., level of detail) relative to the source viewpoint. Thanks to this approach, the terminal (or client application) can retrieve data in a prioritized order: first, the necessary video information for flat 360° rendering, and then, depending on the available throughput, data for improved parallax experiences. The priority of this data retrieval can be proportional to the proximity of the user's position to the scene. This means that video patches and associated metadata corresponding to more objects can only be used if network resources are sufficient.

[0051] Now combine Figures 1 to 3 At least one embodiment of the present disclosure is presented.

[0052] Figure 1 The main steps implemented by a device (e.g., a server) for transmitting a representation of a 3D scene are schematically shown in FIG. According to this embodiment, a server (10) obtains (11) at least one first patch, the first patch comprising a texture component and a depth component. Such one or more first patches (also referred to as master patches or center patches) may be generated by a first view (also referred to as primary view or source view) of the 3D scene captured from a first viewpoint (by a real camera or a virtual camera). The first view may be a projected representation of the 3D scene.

[0053] The server also obtains (12) at least one atlas. Such one or more atlases may be generated from at least one second view of the 3D scene obtained from at least one second viewpoint (by a real camera or a virtual camera). More specifically, for one of the second views (and advantageously for each of the second views), at least one second patch (also called peripheral patch) may be generated. In order to reduce the amount of data that has to be transmitted, such second patches may be generated only for points of the second view that are not visible in the first view or in a second view captured from another viewpoint. Such second patches may be packed or grouped together in the atlas taking into account the angular sectors and / or depth ranges to which the corresponding points belong. In this way, several angular sectors centered around one of the viewpoints and / or several depth ranges originating from one of the viewpoints may be taken into account, and at least one atlas for each angular sector and / or each depth may be constructed and obtained by the server. For example, the first depth range corresponds to a distance between 0 and 50 cm from one of the viewpoints, the second depth range corresponds to a distance between 50 cm and 1 m from one of the viewpoints, the third depth range corresponds to a distance between 1 m and 2 m from one of the viewpoints, and the fourth depth range corresponds to a distance greater than 2 m.

[0054] It may be noted that the steps for obtaining the first patch and obtaining the atlas may be implemented simultaneously or consecutively in any order.

[0055] After obtaining the first patch and the atlas, the server may generate (13) the following streams according to at least one terminal-based delivery standard: (1) a first stream subset comprising m′ pairs of streams from the one or more first patches; and (2) a second stream subset comprising m′×n′ pairs of streams from the one or more atlases, where m′≤m and n′≤n, each pair of streams comprising a stream for transmitting a texture component and a stream for transmitting a depth component.

[0056] For example, if the bandwidth of the communication channel between the server and the terminal is very large, there is no need to transmit only a subset of the streams: m' can be equal to m and n' can be equal to n. On the other hand, if the bandwidth is limited, m' can be equal to 1 and n' can be equal to n, or m' can be equal to m and n' can be equal to 1, or other combinations.

[0057] The server may then transmit (14) or deliver the first subset of streams and the second subset of streams to the terminal.Thus, the first patch and the second patch are transmitted in different frames.

[0058] For example, the terminal-based delivery criteria may be selected from the group consisting of: bandwidth available on a communication channel between the terminal and the server, at least one angular sector observed by a user of the terminal, capabilities of the terminal, and requests received from the terminal.

[0059] The generation of the stream and the transmission of the stream may be performed periodically and / or after the at least one terminal-based delivery standard is changed.

[0060] In this way, the stream to be transmitted to a terminal can be generated to suit the terminal. Specifically, the stream generation can be changed over time to adapt the content carried by the stream to the terminal and, for example, to the terminal user's viewpoint. Stream generation can be determined by the server, for example, after analyzing available bandwidth, or based on a request from the terminal.

[0061] According to at least one embodiment, the server obtains all first patches generated from the first view and all atlases for each angular sector and each depth range generated from all second views of the 3D scene.

[0062] In this way, the server can have complete knowledge of the 3D scene and can generate only streams useful to the terminal based on at least one terminal delivery standard. Specifically, the server can generate a first stream set containing m pairs of streams from all first patches, and a second stream set containing m×n pairs of streams from all atlases, each pair of streams including a stream for transmitting a texture component and a stream for transmitting a depth component.

[0063] According to a first embodiment, the first patch and the second patch and the corresponding atlas may be generated by such a server. In this first embodiment, the steps of obtaining (11) the first patch and obtaining (12) the atlas may correspond to the steps of generating the first patch and generating the atlas.

[0064] According to a second embodiment, the first patch and the second patch and the corresponding atlas may be generated by another device for generating patches and then transmitted to the server. In this second embodiment, the steps of obtaining (11) the first patch and obtaining (12) the atlas may correspond to the steps of receiving the first patch and receiving the atlas.

[0065] Figure 2 The main steps for generating a first patch and an atlas implemented by an apparatus for generating patches according to this second embodiment are shown. According to this embodiment, such an apparatus for generating patches (20) may include a memory (not shown) associated with at least one processor, the processor being configured to: obtain (21) a first view of a scene from a first viewpoint; generate (22) at least one first patch from the first view, the at least one first patch comprising a texture component and a depth component; and obtain (23) at least one second view of the scene from at least one second viewpoint. For at least one second view of the scene (and advantageously for each second view), at least one processor is further configured to: identify (24) at least one point of the second view that is not visible in another view of the 3D scene (the first view or another second view); determine (25) a depth range to which the at least one point belongs, wherein for at least one of m angular sectors centered on one of the viewpoints and for at least one of n depth ranges originating from one of the viewpoints, at least one of m or n is greater than or equal to 2; generate (26) at least one second patch for points belonging to the angular sector and the depth range, the at least one second patch comprising a texture component and a depth component; and construct (27) at least one atlas by packing together at least one (preferably all) second patches generated for points belonging to the same angular sector and the same depth range.

[0066] According to the first or second embodiments, a first patch may be generated by projecting a first view of the 3D scene onto a 2D representation. For example, such a 2D projection may be an equirectangular projection (ERP) or a cubic projection, such as that proposed in the Omnidirectional Media Format (OMAF) standard currently being developed by the Moving Picture Experts Group (MPEG). Other 3D to 2D projection representations may also be used. For more complex projections, a rectangle in the projected image may map to a more complex 3D region than an angular sector, but advantageously a one-to-one correspondence between the tile and the point cloud sub-portion may be ensured.

[0067] According to at least one embodiment, descriptive data describing the organization of the first and second stream subsets can also be transmitted from the server to the terminal. This descriptive data can be transmitted in a manifest file before transmitting the first and second stream subsets. This descriptive data can be transmitted offline on a dedicated channel in response to a request from the terminal, or can be previously stored in the terminal, downloaded from the server when the present disclosure is first used, and so on.

[0068] For example, the description data may include: (1) the number of available depth ranges and their values; (2) the number of available angular sectors and their locations; (3) the resolution of one or more atlases for each stream of the second subset, and whether the atlases are packed together in a GOP; (4) the average bit rate per GOP and per stream of the second subset of streams. The description data may also include the location of patches within the 3D scene, for example, in spherical coordinates. The terminal may use the description data to select and decode the stream and render the 3D scene.

[0069] Figure 3 The main steps implemented by a terminal for rendering a 3D scene are schematically shown in .According to this embodiment, a terminal (30) receives (31) a first subset of streams and a second subset of streams generated according to at least one terminal-based delivery standard.

[0070] For example, the terminal-based delivery criteria may be selected from the group consisting of: bandwidth available on a communication channel with the device for transmitting the representation of the 3D scene, at least one angular sector observed by the terminal user, capabilities of the terminal, and a request sent by the terminal.

[0071] Such a subset of flows can be represented by Figure 1 The server 10 shown in FIG. For example, a terminal may send a request to the server to receive only the disparity information that is useful to the terminal. In a variant, the server may analyze terminal-based delivery criteria (e.g., the communication channel between the server and the terminal or the terminal user's location / viewpoint) and select the streams that must be transmitted. For example, the server may only provide patches that correspond to the user's field of view.

[0072] The first stream subset may include m′ pairs of streams generated from at least one first patch and the second stream subset may include m′×n′ pairs of streams generated from at least one atlas, each pair of streams including a stream for transmitting a texture component and a stream for transmitting a depth component. The at least one first patch may be generated from a first view of a 3D scene. The at least one atlas may be generated from at least one second view of the 3D scene and may be constructed by packing together at least one second patch generated for at least one point of one of the second views that is not visible in another view of the 3D scene and belongs to the same angular sector of m angular sectors and the same depth range of n depth ranges, where m′≤m and n′≤n, and at least one of m or n is greater than or equal to 2. The at least one first patch and the at least one second patch may each include a texture component and a depth component.

[0073] The terminal may then construct (32) and render a representation of the 3D scene from the first subset of streams and the second subset of streams.

[0074] According to at least one embodiment, the second subset of streams may include at least an atlas constructed for a minimum depth range originating from the end user's viewpoint.

[0075] According to at least one embodiment, the second subset of streams may include at least a set of atlases constructed for an angular sector centered about the end user's viewpoint.

[0076] According to these embodiments, the viewpoint of the terminal (or terminal user) may first be determined by the terminal, the server or another device. If determined by the terminal, the terminal may send a request to the server to obtain a second stream subset taking into account the viewpoint.

[0077] Thus, the manner of generating disparity patches according to at least one embodiment of the present disclosure may allow for scalable delivery of volumetric video, and thus allow for reaching a larger audience or improving the experience for the same transmission cost.

[0078] According to at least one embodiment, this approach may allow for the transmission of the same 3DoF+ content over heterogeneous networks with varying bandwidth capabilities, as each terminal may adjust the amount of disparity information retrieved from the server based on its network characteristics.

[0079] According to at least one embodiment, this approach can also aim to provide fast first render (low latency) on devices by enabling progressive rendering and displaying disparity information in order of importance and receipt.

[0080] Detailed descriptions of embodiments of the present disclosure will be described below.

[0081] First, several embodiments of the present disclosure will be discussed in the context of 3DoF+.

[0082] Now, we briefly discuss the generation of patches according to the prior art. To explain the difference between the prior art and the present disclosure, the following is a hint about the 3DoF+ technique for generating patches according to the prior art.

[0083] As mentioned in the prior art section, 3DoF+ has been developed to enrich immersive video experiences with parallax. Volumetric input information (e.g., volumetric video) can be decomposed into several components: the color / depth of a projected representation of a 360° scene viewed from a central point (also known as the first patch or central patch); the color / depth patches of the portion of the scene displayed by natural head displacement, also known as the second patch or peripheral patch; and metadata containing information used to utilize the patches.

[0084] Basically, the components of the volumetric video can be generated by, for example, capturing a 360° scene by a rig of N 360° cameras; generating a point cloud from the N camera captures; introducing four virtual cameras, three of which are placed at the three vertices of a tetrahedron concentric with the central observation point where the camera C0 is located, such as Figure 4 As shown; generate a projection representation with texture and depth, such as from the center camera ( Figure 4 The scene is seen by camera C0 on the vertex, forming two video streams C0 (color) and D0 (depth), where this projected representation can be obtained by any 3D to 2D projection, such as equirectangular projection (ERP) or cubic projection (CMP); a peeling process that generates color / depth patches for points not seen by the previous camera, where for each camera placed on a vertex ( Figure 4 cameras C1, C2 and C3 in the image processing unit), which can be done in an iterative manner; packing the central color / depth patch and the peripheral color / depth patches generated in the previous step in a rectangular patch atlas, where the packing algorithm provides the patch positions on the GOP and metadata is generated accordingly; encoding the atlas using a conventional HEVC video codec, where the depth atlas and the color atlas can first be feathered and quantized in a dedicated manner respectively to be sufficiently robust to encoding artifacts and to optimize the overall bit rate.

[0085] like Figure 5 As shown, the points captured by camera C0 can be placed in the first patches C0.10, C0.I1, and C0.I2, where they are collected by neighboring points. Therefore, patches can be defined by sets of neighboring points. Patches can be segmented based on size criteria.

[0086] The peeling process can then deliver the peripheral patch. Figure 5 As shown, points captured by camera C1 that are not seen by camera C0 are placed in a second patch C1.I0, C1.I1 where they are collected by neighboring points. This process can be implemented iteratively for each camera C1, C2, and C3.

[0087] A dedicated packing algorithm can then place the patches in the color and depth atlases in a GOP-consistent manner (patch positions remain unchanged within a GOP / IntraPeriod). The atlases can then be encoded using the traditional HEVC video codec. For each patch, an additional metadata set can be provided that specifies the information needed to recover the volumetric scene (patch position / size, projection parameters). The entire stream is thus completely video-based and compatible with existing video streaming pipelines.

[0088] Generating a patch according to the present disclosure will be described below.

[0089] According to this disclosure, a new algorithm is proposed for generating patches per viewpoint based on the angular sector of the central viewpoint and / or the distance from the central viewpoint of the angular sector. This technique aims to distinguish points based on their location. In a global scale, the farthest points may require less texture or depth accuracy.

[0090] More specifically, the components of the volumetric video may be generated as disclosed in the previous section, but may also be generated by considering the depth ranges and / or angular sectors to which the points of the point cloud belong.

[0091] According to a first example, the capture from the center camera (C0) delivering the first view (reference Figure 2 21) is not modified according to the prior art. It still provides a projected representation of the 3D scene, such as an equirectangular representation with color and depth (refer to Figure 2 22).

[0092] According to a second example, the capture from the central camera (C0) may be modified according to prior art techniques. For example, the first patches are defined according to the depth range or angular sector to which they belong.

[0093] The second patch is obtained by capturing points from various cameras (e.g., C1, C2 and / or C3 can deliver a second view) (reference Figure 2 23) to show points that were masked by previous captures (ref. Figure 2 24). It should be noted that the cameras C0 to C3 according to the present disclosure can be real cameras or virtual cameras or a combination thereof. In addition, the number of cameras is not limited to the four cameras disclosed in the prior art.

[0094] According to the present disclosure, second patches can be defined by the depth range or angular sector to which they belong, rather than (or in addition to) being defined by neighboring points. The depth here can be the distance from the central viewport (i.e., the location of C0) or the distance from the capture point (i.e., the location of C1, C2, or C3). The second method with respect to the capture point is more relevant because the depth determined from the capture point may be equivalent to the depth seen by the user visualizing the volumetric content. In the same way, the angular sector can be centered on the central viewport or any capture point.

[0095] Figure 6A and Figure 6B Two examples are shown that take into account the depth of a point in the second patch generation, thus allowing patches to be used adaptively based on the depth of the point they represent. This distance is taken into account when constructing the patch. All points of a patch must belong to the same depth range. This allows patches to be constructed based on the depth of the observed point, enabling them to be selected accordingly for optimal delivery.

[0096] according to Figure 6A In the first example shown, the space is divided into three regions D0, D1, and D2, which correspond to three different depth ranges (also called distance ranges) from a central camera C0.

[0097] Therefore, according to the present disclosure, two patches C1.D0.I0 and C1.D0.I1 as well as one patch C1.D1.I0 and one patch C1.D2.I0 (C i Indicates the corresponding camera, D j represents the corresponding depth range, and I k represents the patch index within the considered depth range), and according to Figure 5 The prior art shown generates only one patch C1.I0 and one patch C1.I1.

[0098] according to Figure 6B In the second example shown, the space is divided into three areas D0, D1, D2 corresponding to three different depth ranges (also called distance ranges) from the capture camera C1.

[0099] In this case, five patches C1.D0.10, C1.D0.11, C1.D1.I0, C1.D1.I1, and C1.D2.I0 (C i Indicates the corresponding camera, D j represents the corresponding depth range, and I k represents the patch index within the considered depth range), and according to Figure 5 The prior art shown generates only one patch C1.I0 and one patch C1.I1.

[0100] Thus, if the second patch is defined by grouping adjacent points according to the depth range to which they belong, five patches C1.D0.I0, C1.D0.I1, C1.D1.I0, C1.D1.I1, and C1.D2.I0 may be generated according to the present disclosure. In a variant, if the second patch is not defined by adjacent points but rather according to the depth range or angular sector to which they belong, three patches C1.D0, C1.D1, and C1.D2 may be generated.

[0101] Of course, the number and size of depth ranges are not limited to Figure 6A and Figure 6B Those shown.

[0102] Once patches are constructed, they may be packed into an atlas with other patches of the same depth range (even if the depth is from another viewpoint).

[0103] Once all patches / atlases for each depth and / or each sector are generated, they are stored in the memory of the device used to generate the patches for later use.

[0104] Such per-depth and / or per-angular-sector patching according to the present disclosure may allow privileging of the nearest volumetric data or viewport-based volumetric data when the available throughput is insufficient to deliver all content.

[0105] For example, when the furthest patches are not delivered, inpainting techniques can limit the impact of missing parts of the scene. Available throughput is dedicated to the closest objects, which optimizes rendering.

[0106] Before playing the content, the player / device used for rendering can instantiate and configure a fixed number of video decoders without having to reconfigure them during use, even though the amount of data in the atlas may change over time.

[0107] The following description discusses the delivery of patches.

[0108] According to the present disclosure, a new algorithm for delivering patches, ie transmitting a representation of a 3D scene, is also proposed.

[0109] This transmission is adaptive and depends on at least one terminal-based delivery criterion. According to at least one embodiment, this patch delivery algorithm aims to optimize the user experience based on available network and terminal resources.

[0110] In other words, the device for transmitting the representation of the 3D scene may select some patches / atlases to be transmitted from all patches / atlases previously generated and stored by the device for generating patches. As mentioned, the device for generating patches and the device for transmitting the representation of the 3D scene may be the same device, such as a server.

[0111] Different approaches for adaptive volumetric content delivery with the goal of optimizing bitrate and player resources are disclosed below.

[0112] The following description will first discuss depth-based patch delivery.

[0113] According to a first example, the texture component and the depth component of the projected representation of the 3D scene (the first patch) may be delivered entirely from the device for transmitting the representation of the 3D scene (eg, the server 10 ) to the device for rendering the 3D scene (eg, the terminal 30 ).

[0114] If the first patches are generated by sector and / or by depth in a device for generating patches, they may all be transmitted to the server 10, and the server 10 may connect or merge the first patches to cover one 360° angular sector.

[0115] In the same way, if the second patches are generated by sector and / or by depth in the device for generating patches, they may all be transmitted to the server 10, and the server 10 may connect or merge the second patches to cover one 360° angular sector.

[0116] In this depth-based approach, the content may be organized in 2+(n×2) streams as follows: a first stream set comprising a pair of streams to transmit texture components and depth components of a first patch, respectively; a second stream set comprising n pairs of streams to transmit texture components and depth components of n atlases generated for second patches associated with n depth range levels and metadata associated with the atlases, respectively.

[0117] For example, the first stream set may carry a center patch of size W×H, where W and H may depend on the visual quality defined by pixels per degree (PPD). For example, a 4K×2K frame provides a quality of 4K / 360°=11 PPD.

[0118] These atlases can be grouped together in a group of pictures (GOP). A GOP may have the same duration for all streams and may not always contain frame numbers.

[0119] A manifest can describe the organization of different flows.

[0120] For example, the manifest indicates: for each stream associated with the depth range d=1..n, the number n of available depth ranges and their values, the resolution Wd×Hd of the atlas carried by the stream; and for each stream associated with the depth range d=1..n, for each GOP index t, the average bit rate Ratet,d.

[0121] The value of the atlas resolution Wd×Hd can be defined as, for example: at least equal to the average number of points (i.e., pixels) per patch per second for depth range d; or at least equal to the maximum number of points (i.e., pixels) per patch per second for depth range d.

[0122] In the latter case, there may be exactly one atlas frame per rendered video frame.

[0123] As already mentioned, the manifest may be transmitted offline at the start of content distribution (in the same or a dedicated channel) or via any suitable means (such as an explicit request from the client (terminal) to the server).

[0124] If there is no bandwidth limitation, the server may transmit the first stream set (including one pair of streams) and the second stream set (including n pairs of streams) to the terminal.

[0125] In a variant, for each depth range d, knowing the necessary bandwidth Ratet,d, the terminal can select a first subset of streams and a second subset of streams. For example, as discussed above, the projected representation of the 3D scene (the first patch) can be delivered entirely to the terminal, and the first subset of streams can be identical to the first set of streams. The second subset of streams includes n′ pairs of streams, where n′≤n. The number of atlas streams to be downloaded is selected based on at least one terminal-based criterion, such as available bandwidth or terminal capabilities.

[0126] The stream corresponding to the nearest depth may be downloaded first.

[0127] Rendering can be decoupled from the complete reception of all streams and can start immediately after the first atlas stream is completed. This allows for dynamic progressive rendering. The first level of detail from the first atlas stream (for d=1) is rendered first with the lowest latency, and is gradually completed by receiving the next streams to be processed (for d=2...n').

[0128] In the absence of sectorization (i.e., with one 360° angular sector), patches retrieved by the rendering device may be prioritized by depth index, where the smallest index corresponds to the shortest distance from the viewpoint, as in Figure 7 shown.

[0129] According to at least one embodiment, the number n of available depth ranges for the same content can vary over time. For example, the number can be reduced to one most of the time (e.g., by merging atlases generated on the server side for different depth ranges), and can increase during periods when the scene becomes more complex. In this case, the player's adaptive behavior may allow it to select only the most basic depth atlas based on its available bandwidth.

[0130] The following description will first discuss viewport-based patch delivery.

[0131] According to a second example, the texture component and the depth component of the projected representation (first patch) of the 3D scene can be partially delivered from the device for transmitting the representation of the 3D scene (e.g., server 10) to the device for rendering the 3D scene (e.g., terminal 30) in a viewport-based manner.

[0132] In practice, volumetric content can require a large amount of data to be delivered, so it is not always compatible with existing networks where bandwidth may be limited. Therefore, such content is often delivered in parts based on the viewport.

[0133] For example, high-quality content (e.g., 8K 3DoF+ content or more for full scene representation) can be tiled in m angular sectors ([Θil, Θi2] for longitude and [Θil, Θi2] for latitude). Centered on the central viewport (i.e., position C0) or any capture point (i.e., positions C1, C2, and C3). The second method regarding the capture point is more relevant because the angular sector observed from the capture point may be equivalent to the angular sector seen by the user visualizing the volumetric content.

[0134] For each sector, a set of streams carrying volumetric data corresponding to that sub-portion of the scene is exposed.

[0135] In this viewport-based approach, the content can be organized in 2+(n×2) streams as follows: a first set of m pairs of streams to transmit texture components and depth components of a first patch for m sectors, respectively; a second set of m×n pairs of streams to transmit texture components and depth components of n atlases generated for a second patch associated with n depth range levels and metadata associated with the atlases, respectively, for m sectors.

[0136] If there is no bandwidth limitation, the server can transmit the first stream set (containing m pairs of streams) and the second stream set (containing m×n pairs of streams) to the terminal. In this case, if the available bandwidth is sufficient to deliver all scenes to the player, the depth-based delivery is actually a viewport-based delivery, where m=1 (for example, only one sector).

[0137] In a variant, the client may select a first subset of m′ pairs of streams and a second subset of m′×n′ pairs of streams, where m′≤m and n′≤n, the number of streams to be downloaded being selected according to at least one terminal-based criterion, such as available bandwidth or terminal capabilities.

[0138] On the rendering device, for each time interval (GOP), the next viewport and the sector covering this next viewport can be predicted. Therefore, the terminal can download only the stream related to this part from the server during the next GOP duration. This operation can be repeated for each GOP.

[0139] In another embodiment, for over-provisioning purposes, in addition to the stream associated with the next predicted viewport, the terminal may also download a supplementary stream to cover adjacent portions of the predicted viewport.

[0140] According to at least one embodiment, the atlas can be defined by depth and angular sectors. In this case, the priority of the atlas can be defined based on two parameters: the depth from the user's position; and the angle with the user's gaze direction. Figure 8 The priority of the atlas that the player wants to retrieve based on the location of the point represented by the atlas is shown. Figure 8 As shown, the atlas obtained for the minimum depth index (depth 1) and the angular sector corresponding to the user viewpoint (S0) can be retrieved first. Then, the atlas obtained for the immediately higher depth index (depth 2) and the angular sector corresponding to the user viewpoint (S0) can be retrieved, as well as the atlas obtained for the minimum depth index (depth 1) and the angular sector adjacent to the angular sector corresponding to the user viewpoint (S1, S-1), and so on.

[0141] Of course, the number and size of depth ranges and angular sectors are not limited to Figure 8 Specifically, the size of the depth range of each angular sector may be different from depth range to depth range and from sector to sector.

[0142] As with depth-based delivery methods, a single manifest can describe the organization of the different streams. To benefit from sectorization and provide patch positions to the client, the manifest can also include the patch positions within the 3D scene, for example in spherical coordinates. Since patches may represent volumes rather than points (having texture and depth components), their positions can be represented as a single coordinate indicating the center point of the patch (e.g., the center of gravity), or as a set of spherical coordinates of the volume element containing the patch. / size

[0143] Finally, it should be noted that both depth-based and viewport-based patch delivery methods can be used in combination.

[0144] The following description discusses the device.

[0145] Figure 9 Examples of a device for generating a patch representing a 3D scene, a device for transmitting a representation of a 3D scene, or a device for rendering a 3D scene according to at least one embodiment of the present disclosure are schematically shown.

[0146] The device for generating a patch representing a 3D scene may include, for example, a non-volatile memory 93G (e.g., a read-only memory (ROM) or a hard disk), a volatile memory 91G (e.g., a random access memory or RAM), and at least one processor 92G. The non-volatile memory 93G may be a non-transitory computer-readable carrier medium. It may store executable program code instructions that are executed by the processor 92G to implement the method described above in its various embodiments.

[0147] Specifically, the processor 92G is configured to perform the following processes: obtain a first view of a 3D scene from a first viewpoint; generate at least one first patch from the first view, the at least one first patch including a texture component and a depth component; and obtain at least one second view of the 3D scene from at least one second viewpoint. For at least one second view in the second viewpoint, the processor 92G is further configured to perform the following processes: identify at least one point of the second view that is not visible in another view of the 3D scene; determine a depth range to which the at least one point belongs; for at least one angular sector among m angular sectors and for at least one depth range among n depth ranges, at least one of m or n is greater than or equal to 2, generate at least one second patch from the second view for points belonging to the angular sector and the depth range, the at least one second patch including a texture component and a depth component; and construct at least one atlas by packing together at least one of the second patches generated for points belonging to the same angular sector and the same depth range.

[0148] Upon initialization, the aforementioned program code instructions may be transferred from the non-volatile memory 93G to the volatile memory 91G for execution by the processor 92G. The volatile memory 91G may also include registers for storing variables and parameters required for such execution.

[0149] The device for transmitting a representation of a 3D scene may include, for example, a non-volatile memory 93T (e.g., read-only memory (ROM) or a hard disk), a volatile memory 91T (e.g., random access memory or RAM), and at least one processor 92T. The non-volatile memory 93T may be a non-transitory computer-readable carrier medium. It may store executable program code instructions that are executed by the processor 92T to enable the method described above in its various embodiments.

[0150] Specifically, the processor 92T can be configured to perform the following process: obtain at least one first patch generated from a first view of a 3D scene, the at least one first patch including a texture component and a depth component; obtain at least one atlas generated from at least one second view of the 3D scene, the at least one atlas being constructed by packing together at least one second patch generated for at least one point of one of the second views that is not visible in another view of the 3D scene and belongs to the same angular sector among m angular sectors and the same depth range among n depth ranges, at least one of m or n being greater than or equal to 2, the at least one second patch including a texture component and a depth component; generate a first subset of m′ pairs of streams from the one or more first patches and a second subset of m′×n′ pairs of streams from the one or more atlases according to at least one terminal-based delivery standard, where m'≤m and n′≤n, each pair of streams including a stream for transmitting a texture component and a stream for transmitting a depth component, and transmit the first stream subset and the second stream subset to the terminal.

[0151] Upon initialization, the aforementioned program code instructions may be transferred from the non-volatile memory 93T to the volatile memory 91T for execution by the processor 92T. The volatile memory 91T may also include registers for storing variables and parameters required for such execution.

[0152] The apparatus for rendering a 3D scene may include, for example, a non-volatile memory 93R (e.g., a read-only memory (ROM) or a hard disk), a volatile memory 91R (e.g., a random access memory or RAM), and at least one processor 92R. The non-volatile memory 93R may be a non-transitory computer-readable carrier medium. It may store executable program code instructions that are executed by the processor 92R to implement the methods described above in their various embodiments.

[0153] Specifically, processor 92R may be configured to receive a first stream subset and a second stream subset generated according to at least one terminal-based delivery standard, the first stream subset comprising m′ pairs of streams generated from at least one first patch and the second stream subset comprising m′×n′ pairs of streams generated from at least one atlas, each pair of streams comprising a stream for transmitting a texture component and a stream for transmitting a depth component, the at least one first patch being generated from a first view of a 3D scene and comprising a texture component and a depth component, the at least one atlas being generated from at least one second view of the 3D scene and constructed by packing together at least one second patch generated for at least one point of one of the second views that is not visible in another view of the 3D scene and belongs to the same angular sector of m angular sectors and the same depth range of n depth ranges, at least one of m or n being greater than or equal to 2, the at least one second patch comprising a texture component and a depth component, where m′≤m and n′≤n. Processor 92R may be further configured to construct a representation of the 3D scene from the first stream subset and the second stream subset.

[0154] Upon initialization, the aforementioned program code instructions may be transferred from the non-volatile memory 93R to the volatile memory 91R for execution by the processor 92R. The volatile memory 91R may also include registers for storing variables and parameters required for such execution.

[0155] The method according to at least one embodiment of the present disclosure may be equally well implemented by: (1) executing a set of program code instructions executed by a reprogrammable computing machine (such as a PC-type device, a DSP (digital signal processor), or a microcontroller). Such program code instructions may be stored on a detachable (e.g., floppy disk, CD-ROM, or DVD-ROM) or non-detachable non-transitory computer-readable carrier medium; or (2) a dedicated machine or component (such as an FPGA (field programmable gate array), an ASIC (application-specific integrated circuit), or any dedicated hardware component).

[0156] In other words, the present disclosure is not limited to a purely software-based implementation in the form of computer program instructions, but the present disclosure can also be realized in the form of hardware or in any form combining hardware parts and software parts.

[0157] The flowcharts and / or block diagrams in the figures illustrate possible configurations, operations, and functions of the systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of code, which includes one or more executable instructions for implementing a specified logical function.

[0158] It should also be noted that in some alternative implementations, the functions marked in the blocks may not appear in the order marked in the figure. For example, two blocks shown in succession may actually be executed substantially simultaneously, or the blocks may sometimes be executed in the reverse order, or the blocks may be executed in an alternative order depending on the functions involved. It should also be noted that each block of the block diagram and / or flowchart illustration, and the combination of blocks in the block diagram and / or flowchart illustration, may be implemented by a dedicated hardware-based system that performs the specified function or action, or a combination of dedicated hardware and computer instructions. Although not explicitly described, embodiments of the present invention may be employed in any combination or subcombination.

Claims

1. A method for rendering a 3D scene on a terminal, the method comprising: A list is received at the terminal, wherein the list includes: a reference to at least one first data stream available from a server, the at least one first data stream comprising a center patch content of a 3D scene, the center patch content comprising a portion of the scene visible from a center viewpoint; references to a plurality of second data streams available from a server, the plurality of second data streams comprising disparity patch content of the 3D scene, wherein the disparity patch content comprises a portion of the 3D scene visible from a second non-central viewpoint; and association of each second data stream with an angular sector; requesting at least one available first data stream from the server; requesting from the server a subset of available second data streams selected based at least on an angular sector associated with at least one available second data stream; and A 3D scene is rendered using center patch content from the requested first data stream and disparity patch content from the selected subset of the requested available second data stream.

2. The method according to claim 1, further comprising: A requested first data stream and a requested subset of available second data streams are received at a terminal. 3 . The method of claim 1 , wherein the manifest further specifies an association of each second data stream with a depth range.

4. The method according to claim 3, wherein: A subset of the available second data streams is selected based on the depth ranges and angular sectors associated with the available second data streams.

5. The method according to claim 3, wherein: The subset of available second data streams is selected based on a priority of the depth ranges, second data streams associated with closer depth ranges being given higher priority than second data streams associated with further depth ranges.

6. The method according to claim 1, wherein The associated angular sector comprises an angular range within the end user's field of view.

7. The method according to claim 1, wherein The subset of available second data streams is selected based on at least one of available bandwidth on a communication channel between the terminal and the server and capabilities of the terminal.

8. The method of claim 1, wherein the manifest further comprises an association of each second data stream with a time interval.

9. The method according to claim 1, wherein: The terminal is a head-mounted display.

10. The method of claim 1, wherein the disparity patch content comprises a depth patch and a texture patch.

11. The method according to claim 1, wherein Each second data stream carries disparity patch content in the form of a patch atlas.

12. A method for generating a 3D scene at a server, the method comprising: Transmit a manifest, wherein the manifest includes: a reference to at least one first data stream, the at least one first data stream comprising a center patch content of a 3D scene, the center patch content comprising a portion of the scene visible from a center viewpoint; a reference to a second plurality of data streams comprising disparity patch content of a 3D scene, wherein the disparity patch content comprises a portion of the 3D scene visible from a second non-central viewpoint; and association of each second data stream with an angular sector; receiving a request for at least one available first data stream; receiving a request for a subset of available second data streams selected based at least on an angular sector associated with the at least one available second data stream; and The at least one available first data stream and a subset of the available second data stream are transmitted to enable rendering of a 3D scene.

13. The method of claim 12, further comprising wherein the manifest further specifies an association of each second data stream with a depth range.

14. The method according to claim 12, wherein: The associated angular sector comprises an angular range within the end user's field of view.

15. The method of claim 12, wherein the manifest further comprises an association of each second data stream with a time interval. The method of claim 12 , wherein the disparity patch content comprises a depth patch and a texture patch.

17. The method according to claim 12, wherein: Each second data stream carries disparity patch content in the form of a patch atlas.

18. A terminal comprising: Receiver; transmitter; and processor; wherein the receiver is configured to receive a manifest, wherein the manifest comprises: a reference to at least one first data stream available from a server, the at least one first data stream comprising a center patch content of a 3D scene, the center patch content comprising a portion of the 3D scene visible from a center viewpoint; references to a plurality of second data streams available from a server, the plurality of second data streams comprising disparity patch content of the 3D scene, wherein the disparity patch content comprises a portion of the 3D scene visible from a second non-central viewpoint; and association of each second data stream with an angular sector; wherein the transmitter is configured to request at least one available first data stream from the server; wherein the transmitter is further configured to request from the server a subset of available second data streams selected based at least on an angular sector associated with the at least one available second data stream; and Wherein the processor is configured to render a 3D scene using center patch content from the requested first data stream and disparity patch content from a selected subset of the requested available second data stream.

Images