Method and apparatus for encoding and decoding volumetric video.

Abstract

Methods, apparatus, and streams for encoding and decoding scenes (such as point clouds) in the context of chunk-based transmission of volumetric video content are disclosed. The invention relates to a method for partitioning an atlas image into regions and packing them into chunks, each region being associated with a sector of the scene's space, and obtaining the chunk by projecting only points included in the associated sector. The atlas and associated metadata are encoded in the data stream; during rendering, a set of sectors is selected based on the user's current field. Only pixels of the region associated with the selected sector are accessed to generate a viewport image representing the scene viewed from the current viewpoint.

Description

1. Technical Field

[0001] The principles of this invention generally relate to the domain of three-dimensional (3D) scenes and volumetric video content. This document is also understood in the context of encoding, formatting, and decoding data representing the textures and geometry of 3D scenes for rendering volumetric content on end-user devices such as mobile devices or head-mounted displays (HMDs). 2. Background Technology

[0002] This section aims to introduce the reader to various aspects of the art that may relate to the aspects of the inventive principles described and / or claimed below. This discussion is believed to help provide the reader with background information to facilitate a better understanding of the various aspects of the inventive principles. Therefore, it should be understood that these statements should be interpreted in this light, rather than as an admission of prior art.

[0003] Immersive video (also known as 360° planar video) allows users to see everything around them by rotating their heads around a stationary viewpoint. Rotation only allows for a 3-degree-of-freedom (3DoF) experience. Even if 3DoF video is sufficient for first-time omnidirectional video experiences (e.g., using a head-mounted display (HMD device)), it can quickly become frustrating for viewers expecting more freedom (e.g., by experiencing parallax). Furthermore, 3DoF can cause dizziness because users never just rotate their heads, but also translate them in three directions, movements that are not reproduced in a 3DoF video experience.

[0004] In this context, large field-of-view content can be three-dimensional computer graphics scenes (3D CGI scenes), point clouds, or immersive videos. Many terms can be used to design such immersive videos: for example, virtual reality (VR), 360, panoramic, 4π spherical, immersive, omnidirectional, or large field of view.

[0005] Volumetric video (also known as 6DoF video) is an alternative to 3DoF video. When watching 6DoF video, in addition to rotation, users can pan their head and even their body within the content being viewed, experiencing parallax and even volume. This type of video significantly increases immersion and perception of scene depth, and prevents motion sickness by providing consistent visual feedback during head panning. The content is created using dedicated sensors, allowing for the simultaneous recording of color and depth of the scene of interest. Even though technical challenges remain, using color camera equipment incorporating photogrammetry is another way to perform this recording.

[0006] While 3DoF video comprises a sequence of images derived from the demapping of textured images (e.g., spherical images encoded according to latitude / longitude projection maps or isometric projection maps), 6DoF video frames embed information from multiple viewpoints. They can be viewed as a temporal series of point clouds generated by 3D capture. Two types of volumetric video can be considered depending on the viewing conditions. The first (i.e., full 6DoF) allows for completely free navigation within the video content, while the second (aka 3DoF+) restricts the user's viewing space to a finite volume called the viewing bounding box, thus allowing for limited head translation and parallax experience. This second case represents a valuable trade-off between free navigation and the passive viewing conditions of a seated audience.

[0007] Techniques for encoding volumetric video are based on projecting a 3D scene onto multiple 2D images (called tiles), which are then packaged into an atlas that can be further compressed using conventional video coding standards (e.g., HEVC). During decoding, it is not always necessary or expected to use every tile of the atlas to prepare the rendering of the 3D scene in the current viewport. In fact, some tiles include information about points not visible from the current viewpoint, and for a given portion of the 3D scene to be rendered, some tiles include more accurate information than other tiles that include information about the same portion of the scene when viewed from the current viewpoint.

[0008] The success of emerging technologies such as immersive media (MPEG Immersive Video-MIV based on the 3DoF+ method) often depends on their widespread availability on everyday devices such as embedded low-end user devices like digital TVs, set-top boxes, tablets, or wireless HMDs. However, these devices have limited GPU resources, and the immersive technology must also be compatible with these types of devices. One possible approach is to generate two types of content: a downgraded immersive content for low-end user devices and another for a richer immersive experience on high-end users' PCs, such as gamers. This approach requires storing several versions of the same content and knowing the type of client device used to transmit the adapted version. Another approach is based on tiling. However, this approach has the drawback of introducing latency into the streaming process, where the server must stream tiles corresponding to the user's current viewpoint direction. This results in motion photon latency (i.e., the time between the moment the user moves in a direction and the moment the renderer receives the content from the server and displays it on the screen).

[0009] The lack of a data stream format for encoding volumetric video allows the decoder to select a subset of chunks within the acquired atlas for access and use by the rendering engine, depending on the user's current viewpoint position and orientation. 3. Summary of the Invention

[0010] The following is a simplified overview of the principles of the invention to provide a basic understanding of some aspects of these principles. This summary is not a broad overview of the principles of the invention and is not intended to identify key or essential elements of the invention. The following summary presents only some aspects of the principles of the invention in a simplified form as a preface to the more detailed description that follows.

[0011] In the context of chunk-based representation of volumetric video content, this principle relates to a method for decoding a volumetric scene and generating a viewport image visible from the user's current viewpoint. The method includes:

[0012] - Obtain source images divided into regions, which are associated with sectors in the scene space;

[0013] - Select at least one sector based on the current field of view of a virtual camera controlled by the user and located in that space of the scene; and

[0014] - Access pixels of a region of the source image associated with at least one selected sector to generate a viewport image of the scene for the user's field of view.

[0015] In this implementation, the source images are packaged tiled images. Each tiled image is only a portion of a region and is a projection of points of the scene included in the sectors associated with that region. For volumetric video as a sequence of source images, the association between regions and sectors can change over time.

[0016] In another implementation, a sector in the scene space is associated with two regions. The first region comprises a tiled image of the projection points of the sector visible from a given viewpoint, such as the center viewpoint of a 3DoF+ viewing area. The second region comprises tiled images of the projections of the same sector's points not visible from that given viewpoint, and these tiled images are referred to as peripheral tiles.

[0017] The source image can be obtained from a data stream that includes metadata that associates each region of the source image with sectors of the scene space. In variants, the metadata is obtained from a different source.

[0018] The principles of the present invention also relate to a device including a processor configured to implement the above-described methods.

[0019] The invention also relates to a method for generating images divided into regions, each region being associated with a sector of scene space. Points of the scene included in a sector of the scene are projected into pixels of the region associated with that sector.

[0020] In this implementation, the image is a packaged tiled image. Each tiled image belongs to a unique region and is a projection of points of the scene included in a sector associated with that region.

[0021] In another implementation, a sector in the scene space is associated with two regions. The first region comprises a tiled image of the projection points of the sector visible from a given viewpoint, such as the center viewpoint of a 3DoF+ viewing area. The second region comprises tiled images of the projections of the same sector's points not visible from that given viewpoint, and these tiled images are referred to as peripheral tiles.

[0022] This method can further include encoding the image and metadata that associates each region of the source image with sectors of the scene space in the data stream. In a variant, the metadata is encoded in a separate data stream.

[0023] The principles of this invention also relate to a device including a processor configured to implement the above-described method. The principles of this invention also relate to a data stream generated by the method. 4. Description of the attached drawings

[0024] This disclosure will be better understood, and further specific features and advantages will emerge after reading the following description and referring to the accompanying drawings, in which:

[0025] - Figure 1 A three-dimensional (3D) model of an object according to a non-limiting embodiment of the principles of the present invention and points of a point cloud corresponding to the 3D model are shown;

[0026] - Figure 2 Non-limiting examples of encoding, transmitting, and decoding data representing a sequence of 3D scenes according to a non-limiting embodiment of the principles of the present invention are shown;

[0027] - Figure 3 The illustration shows a non-limiting embodiment of the invention that can be configured to achieve the following: Figure 11 and Figure 12 An exemplary architecture of the device for the described method;

[0028] - Figure 4 Examples of embodiments of the syntax of a stream when transmitting data via a packet-based transport protocol are shown, according to a non-limiting embodiment of the principles of the present invention;

[0029] - Figure 5 An example of a block-based three-dimensional (3D) scene representation according to a non-limiting embodiment of the principles of the present invention is shown;

[0030] - Figure 6 An example of a texture atlas including points of a 3D scene, according to a non-limiting embodiment of the principles of the present invention, is shown;

[0031] - Figure 7 The illustration shows a non-limiting embodiment based on the principles of the invention, including... Figure 6 An example of a 3D scene atlas containing depth information of points;

[0032] - Figure 8 This illustrates an implementation scheme for sectorization of the 3D space in a 3D scene.

[0033] - Figure 9 A first layout of a sectorized atlas according to the principles of the present invention is shown;

[0034] - Figure 10 A second layout of the sectorized atlas according to the principles of the present invention is shown;

[0035] - Figure 11 A third layout of a sectorized atlas prepared for cube mapping according to the principles of the present invention is shown;

[0036] - Figure 12 A method 120 for generating a viewport image of a scene encoded in an atlas image is shown according to a non-limiting embodiment of the principles of the present invention;

[0037] - Figure 13 A method 130 for encoding a scene is shown according to a non-limiting embodiment of the principles of the present invention. 5. Detailed Implementation

[0038] The principles of the invention will be described more fully below with reference to the accompanying drawings, in which examples of the principles of the invention are shown. However, the principles of the invention may be embodied in many alternative forms and should not be construed as limited to the examples set forth herein. Therefore, while the principles of the invention are susceptible to various modifications and alternatives, specific examples are shown by way of example in the drawings and will be described in detail herein. However, it should be understood that there is no intention to limit the principles of the invention to the specific forms disclosed, but rather, this disclosure is intended to cover all modifications, equivalents, and alternatives that fall within the spirit and scope of the principles of the invention as defined by the claims.

[0039] The terminology used herein is for the purpose of describing particular examples only and is not intended to limit the principles of the invention. As used herein, the singular forms “a,” “an,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It will be further understood that, when used in this specification, the terms “comprising” and / or “including” specify the presence of the stated feature, integer, step, operation, element, and / or component, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Furthermore, when an element is referred to as “responding” or “connected” to another element, it may directly respond to or be connected to the other element, or there may be intermediate elements present. Conversely, when an element is referred to as “directly responding” or “directly connected” to another element, there are no intermediate elements present. As used herein, the term “and / or” includes any and all combinations of one or more of the listed related items and may be abbreviated to “ / ”.

[0040] It should be understood that although the terms first, second, etc., may be used herein to describe various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, without departing from the teachings of the principles of the invention, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element.

[0041] Although some illustrations include arrows along the communication path to show the main communication direction, it should be understood that communication can occur in the opposite direction to the arrows depicted.

[0042] Examples of block diagrams and operation flowcharts are described where each box represents a circuit element, module, or code section, the code section including one or more executable instructions for implementing a specified logical function. It should also be noted that in other specific implementations, the functions marked in the boxes may not appear in the order they are marked. For example, two boxes shown consecutively may actually be executed substantially simultaneously, or these boxes may sometimes be executed in reverse order depending on the functions involved.

[0043] The references to "according to an example" or "in an example" in this document mean that a particular feature, structure, or characteristic described in connection with the example may be included in at least one specific embodiment of the principles of the invention. The appearance of the phrases "according to an example" or "in an example" in various places in the specification does not necessarily refer to the same example in all instances, nor is it necessarily a separate or alternative example that is mutually exclusive with other examples.

[0044] The reference numerals appearing in the claims are for illustrative purposes only and do not limit the scope of the claims. Although not explicitly described, these examples and variations may be employed in any combination or sub-combination.

[0045] In the context of tile-based encoding of a 3D scene, the data stream includes metadata, which contains information about each tile of the atlas. Each tile is associated with projection data, which includes parameters of the virtual camera that captured the tile during encoding. During decoding, these parameters of the virtual camera (e.g., pose and 3D-to-2D projection method) are retrieved from the stream to perform a 2D-to-3D inverse conversion.

[0046] The volumetric video is prepared (i.e., analyzed and encoded) for rendering from a viewing area that is part of the 3D space of the 3D scene. Points of the 3D scene visible from any viewpoint within this viewing area are encoded in tiles of the atlas. If the decoded 3D scene is viewed from a viewpoint outside the viewing area, some points will be missing and the 3D scene will be incomplete and / or exhibit strong visual artifacts. In 3DoF rendering mode, the viewing area is limited to points in the 3D space of the 3D scene. In 3DoF+ rendering mode, the viewing area is a finite region surrounding an initial point in the 3D space. In 6DoF, the viewing area is the entire 3D space of the 3D scene.

[0047] The principles of this invention relate to a method, data stream, and device for associating tiles of an atlas with a validity domain that is part of a viewing area. This association is encoded via formatted metadata linked to the atlas in the data stream. Upon decoding, this metadata allows a rendering engine to select a subset of tiles in the atlas that is associated with a validity domain covering the current viewpoint. The validity domain is determined such that these tiles are best suited for the current view among the different tiles encoding these points, because these tiles include information for each visible point in the 3D scene from that viewpoint and the most accurate information for those points.

[0048] Figure 1 A three-dimensional (3D) model 10 of an object and points corresponding to a point cloud 11 of the 3D model 10 are shown. The 3D model 10 and point cloud 11 may, for example, correspond to possible 3D representations of objects in a 3D scene including other objects. Model 10 may be a 3D mesh representation, and the points of point cloud 11 may be vertices of the mesh. The points of point cloud 11 may also be points distributed on the surface of the mesh face. Model 10 may also be represented as a sputtered version of point cloud 11, the surface of which is created by sputtering the points of point cloud 11. Model 10 may be represented by many different representations such as voxels or splines. Figure 1 This demonstrates that a point cloud can be defined using a surface representation of a 3D object, and that a surface representation of a 3D object can be generated from cloud points. As used herein, projecting points of a 3D object (and by extension, points of a 3D scene) onto an image is equivalent to projecting any representation of that 3D object, such as a point cloud, mesh, spline model, or voxel model.

[0049] Point clouds can be represented in memory as, for example, a vector-based structure, where each point has its own coordinates (e.g., 3D coordinates XYZ, or solid angle and distance from / to the viewpoint (also called depth)) and one or more attributes, also called components, in the viewpoint's frame of reference. An example of components is color components, which can be represented in various color spaces, such as RGB (red, green, and blue) or YUV (Y is the luminance component and UV are the two chrominance components). A point cloud is a representation of a 3D scene including objects. The 3D scene can be viewed from a given viewpoint or viewpoint range. Point clouds can be obtained in various ways, such as:

[0050] • Capture of real objects from camera equipment, optionally supplemented by active depth sensing devices;

[0051] • Capture of virtual / composite objects taken by a virtual camera setup within a modeling tool;

[0052] • A mixture of real and virtual objects.

[0053] Figure 2 A non-limiting example of encoding, transmitting, and decoding data representing a sequence of 3D scenes is shown. The encoding format may, for example, be compatible with 3DoF, 3DoF+, and 6DoF decoding simultaneously.

[0054] Obtain a 3D scene sequence 20. Just as a frame sequence is 2D video, a 3D scene sequence is 3D (also known as volumetric) video. The sequence of 3D scenes is provided according to a viewing area where the 3D scene can be viewed during decoding. The 3D scene can be provided to a volumetric video rendering device for 3DoF, 3DoF+, or 6DoF rendering and display.

[0055] A 3D scene sequence 20 can be provided to an encoder 21. The encoder 21 takes a 3D scene or a sequence of 3D scenes as input and provides a bitstream representing that input. The bitstream can be stored in a memory 22 and / or on an electronic data medium, and can be transmitted via a network 22. The bitstream representing the 3D scene sequence can be read from the memory 22 and / or received from the network 22 by a decoder 23. The decoder 23 takes the bitstream input and provides a 3D scene sequence, for example, in point cloud format.

[0056] Encoder 21 may include several circuits implementing several steps. In a first step, encoder 21 projects each 3D scene onto at least one 2D frame. 3D projection is any method of mapping three-dimensional points onto a two-dimensional plane. This type of projection is widely used, especially in computer graphics, engineering, and drafting, because most current methods for displaying graphics data are based on a planar (pixel information from several bit planes) two-dimensional medium. Projection circuitry 211 provides at least one two-dimensional frame 2111 for the sequence of 3D scenes 20. Frame 2111 includes color information and depth information representing the 3D scene projected onto frame 2111. In a variant, the color and depth information are encoded in two separate frames 2111 and 2112.

[0057] Metadata 212 is used and updated by projection circuitry 211. Metadata 212 includes information about projection operations (e.g., projection parameters) and information about how color and depth information is organized within frames 2111 and 2112, such as... Figures 5 to 7 As stated above.

[0058] According to the principles of the present invention, at least one validity domain is determined by projection circuit 211. A validity domain is information describing a portion of the viewing area. A validity domain can represent a combination of connected 3D regions or potentially disconnected connected regions. Projection circuit 211 associates each of its generated blocks with a validity domain. In a variant, projection circuit 211 does not associate some blocks within a block with any validity domain, thus indicating that the validity domain of that block is the entire viewing area. That portion of the 3D space is defined and associated with the block when the information projected onto that block is necessary and accurate for reconstructing the 3D scene from a viewpoint contained within a portion of the 3D space during rendering. The same validity domain can be associated with several blocks. The description of the determined validity domain and the association between blocks and validity domains are added to metadata 212, and this information is then encapsulated in a data stream by data encapsulation circuit 214.

[0059] In an embodiment of the principles of this invention, the validity domain is determined relative to the projection center. The validity domain determined for a given projection center is associated with each block generated based on that projection center, as referenced... Figure 5 As described. The description of the validity field added in metadata 212 is associated with the projection data describing the different projection operations used to generate the atlas tiles.

[0060] The video encoding circuit 213 encodes the sequence of frames 2111 and 2112 into video. The frames 2111 and 2112 of the 3D scene (or the sequence of frames of the 3D scene) are encoded in the stream by the video encoder 213. Then, the video data and metadata 212 are encapsulated in the data stream by the data encapsulation circuit 214.

[0061] Encoder 213 is compatible with, for example, encoders such as:

[0062] -JPEG, specification ISO / CEI 10918-1UIT-T Recommendation T.81, https: / / www.itu.int / rec / T-REC-T.81 / en;

[0063] -AVC, also known as MPEG-4 AVC or h264. It is specified in both UIT-T H.264 and ISO / CEI MPEG-4 Part 10 (ISO / CEI 14496-10), http: / / www.itu.int / rec / T-REC-H.264 / en, HEVC (its specification can be found on the ITU website, T recommendation, H series, h265, http: / / www.itu.int / rec / T-REC-H.265-201612-I / en);

[0064] -3D-HEVC (an extension of HEVC, the specification of which can be found on the ITU website, T recommendation, H series, h265, http: / / www.itu.int / rec / T-REC-H.265-201612-I / en annex G and I);

[0065] - VP9 developed by Google; or

[0066] - AV1 (AOMedia Video 1) was developed by Alliance for Open Media.

[0067] The data stream is stored in a memory accessible by the decoder 23, for example, via network 22. The decoder 23 includes different circuitry implementing various decoding steps. The decoder 23 takes the data stream generated by the encoder 21 as input and provides a sequence 24 of 3D scenes to be rendered and displayed by a volumetric video display device, such as a head-mounted display (HMD). The decoder 23 obtains the stream from the source 22. For example, the source 22 belongs to a group that includes:

[0068] - Local storage, such as video storage or RAM (or random access memory), flash memory, ROM (or read-only memory), hard disk;

[0069] - Storage interfaces, such as interfaces for mass storage devices, RAM, flash memory, ROM, optical discs, or magnetic media;

[0070] - Communication interfaces, such as wired interfaces (e.g., bus interfaces, WAN interfaces, LAN interfaces) or wireless interfaces (e.g., IEEE 802.11 interfaces or...). Interface); and

[0071] - User interfaces that enable users to input data, such as graphical user interfaces.

[0072] Decoder 23 includes circuitry 234 for extracting data encoded in the data stream. Circuitry 234 takes the data stream as input and provides metadata 232 corresponding to metadata 212 encoded in the stream and two-dimensional video. The video is decoded by video decoder 233, which provides a sequence of frames. The decoded frames include color and depth information. A frame is an atlas, i.e., image data comprising a set of chunks packed within the frame. A chunk is image data containing information for retrieving points from a 3D scene for reconstruction. In a variant, video decoder 233 provides two frame sequences, one containing color information and the other containing depth information.

[0073] According to the principles of the present invention, metadata 232 includes at least one validity field associated with at least one chunk of the atlas. The validity field is information representing a portion of the viewing area in the 3D space of the 3D scene.

[0074] Circuit 231 uses metadata 232 to deproject the color and depth information from the decoded frame to provide a 3D scene sequence 24. The 3D scene sequence 24 corresponds to the 3D scene sequence 20, and may lose the accuracy associated with encoding and video compression as 2D video.

[0075] The 3D scene is retrieved from an atlas containing multiple tiles. According to the principles of the invention, circuit 231 deprojects a subset of pixels from the tiles of the atlas. Circuit 231 selects only tiles associated with a validity domain (via metadata 232) surrounding the current rendering viewpoint. In a variant, if a tile is not associated with a validity domain, that tile is always used for deprojection.

[0076] In an embodiment of the principles of the invention, metadata 232 includes a collection of chunked data. Within each chunked data item, a chunk is associated with projection data including parameters for a projection operation used to generate that chunk. Metadata 232 also includes a set of projection data, and the projection data is associated with a validity domain. In this embodiment, circuitry 231 selects chunks for deprojection that are associated with projection data that is itself associated with a validity domain surrounding the current viewpoint.

[0077] Figure 3 This shows that it can be configured to implement about Figure 11 and Figure 12 An exemplary architecture of device 30 for the described method. Figure 2The encoder 21 and / or decoder 23 can implement this architecture. Alternatively, each circuit in the encoder 21 and / or decoder 23 can be based on... Figure 3 Devices with an architecture that are linked together, for example, via their bus 31 and / or via I / O interface 36.

[0078] Device 30 includes the following components connected together via data and address bus 31:

[0079] - Microprocessor 32 (or CPU), which is, for example, a DSP (or digital signal processor);

[0080] -ROM (or read-only memory) 33;

[0081] -RAM (or random access memory) 34;

[0082] - Storage interface 35;

[0083] -I / O interface 36, which is used to receive data to be transmitted from the application; and

[0084] - Power source, such as a battery.

[0085] According to one example, the power supply is external to the device. In each mentioned memory, the term "register" used in the specification can correspond to a small area (a few bits) or a very large area (e.g., the entire program or a large amount of received or decoded data). ROM 33 includes at least the program and parameters. ROM 33 can store algorithms and instructions for executing the technology according to the principles of the invention. When powered on, CPU 32 loads the program from RAM and executes the corresponding instructions.

[0086] RAM 34 includes a program executed by CPU 32 and uploaded after device 30 is turned on, input data in the register, intermediate data in different states of the method in the register, and other variables used to execute the method in the register.

[0087] The specific embodiments described herein may be implemented, for example, in methods or processes, apparatus, computer program products, data streams, or signals. Even if discussed only in the context of a single form of implementation (e.g., discussed only as a method or apparatus), the specific implementation of the discussed features may be implemented in other forms (e.g., programs). Apparatus may be implemented, for example, in suitable hardware, software, and firmware. Methods may be implemented in apparatus (such as, for example, a processor) that generally refers to a processing device, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices, such as, for example, computers, mobile phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate information communication between end users.

[0088] According to the example, device 30 is configured to implement about Figure 11 and Figure 12 The described method belongs to a set that includes the following items:

[0089] -mobile device;

[0090] - Communication equipment;

[0091] -Gaming devices;

[0092] - Tablet PC (or tablet computer);

[0093] - Laptop;

[0094] - Still image camera;

[0095] -Camera;

[0096] - Encoding chip;

[0097] - Servers (such as broadcast servers, video-on-demand servers, or web servers).

[0098] Figure 4 An example of an implementation of the syntax for streams is shown when data is transmitted via a packet-based transport protocol. Figure 4 An exemplary structure 4 for a volumetric video stream is shown. This structure is contained within a container that organizes the stream by syntactic, independent elements. The structure may include a header section 41, which is a set of data common to each syntactic element of the stream. For example, the header section includes metadata about the syntactic elements, describing the properties and roles of each of them. The header section may also include... Figure 2 This is part of the metadata 212, such as the coordinates of the center viewpoint used to project points of the 3D scene onto frames 2111 and 2112. The structure includes a payload comprising syntax element 42 and at least one syntax element 43. Syntax element 42 includes data representing color and depth frames. The image may have been compressed according to a video compression method.

[0099] Syntax element 43 is part of the payload of the data stream and may include metadata about how the frames of syntax element 42 are encoded, such as parameters for projecting points of the 3D scene onto the frames. Such metadata may be associated with each frame or group of frames (also known as a group of frames (GoP) in video compression standards). According to the principles of the invention, the metadata of syntax element 43 also includes at least one validity field associated with at least one chunk of the atlas. The validity field is information representing a portion of the view area in the 3D space of the 3D scene and can be encoded according to different representations and structures. Examples of such representations and structures are provided in this disclosure.

[0100] For example, the metadata includes a collection of at least one valid field. The collected items may be indexed, for example, by letters or by integers. The metadata further includes a projected data collection, which includes at least a reference to one of the valid fields. This reference may be, for example, an index to a valid field in the collection or a pointer to a memory address of that data. In this example, the metadata also includes a chunked data collection, which includes references to chunks and references to a piece of projected data.

[0101] Figure 5 A tiled atlas method is illustrated using four projection centers as an example. The 3D scene 50 includes characters. For example, projection center 51 is a perspective camera, and camera 53 is an orthophoto camera. The camera can also be an omnidirectional camera with, for example, a spherical mapping (e.g., an isorectangular mapping) or a cubic mapping. Based on the projection operations described in the projection data of the metadata, 3D points of the 3D scene are projected onto a 2D plane associated with a virtual camera located at the projection center. Figure 5 In the example, the projection of the points captured by camera 51 is mapped onto block 52 according to perspective mapping, and the projection of the points captured by camera 53 is mapped onto block 54 according to orthophoto mapping.

[0102] Clustering of projected pixels produces multiple 2D tiles, which are packed into a rectangular atlas 55. The organization of tiles within the atlas defines the atlas layout. In an embodiment, two atlases have the same layout: one for texture (i.e., color) information and one for depth information. Two tiles captured by the same camera or by two different cameras may include information representing the same portion of the 3D scene, such as, for example, tiles 54 and 56.

[0103] The packing operation generates chunk data for each generated chunk. Chunk data includes references to the projection data (e.g., an index in the projection data table or a pointer to the projection data (i.e., an address in memory or the data stream)) and information describing the location and size of the chunk within the atlas (e.g., top-left corner coordinates, size, and width in pixels). Chunk data items are added to metadata to be encapsulated in the data stream in association with the compressed data of one or two atlases.

[0104] Figure 6 An example of the layout of an atlas 60 comprising texture information (e.g., RGB or YUV data) of points in a 3D scene, according to a non-limiting embodiment of the principles of the invention, is shown. (See attached image.) Figure 5 The atlas, as explained, is a packaged and segmented image, which is a view obtained by projecting a portion of points from a 3D scene.

[0105] The layout of an atlas refers to the way it organizes blocks on the image plane. Figure 6 In the example, atlas 60 includes a first portion 61 and one or more second portions 62, the first portion including texture information of points visible from the viewpoint in the 3D scene. The texture information of the first portion 61 can be obtained, for example, according to an isometric projection mapping, where an isometric projection mapping is an example of a spherical projection mapping. Figure 6 In the example, the second part 62 is positioned at the left and right boundaries of the first part 61, but the second part can be positioned differently. The second part 62 includes texture information of a portion of the 3D scene that is complementary to the portion visible from the viewpoint. The second part can be obtained by removing points visible from the first viewpoint (whose textures are stored in the first part) from the 3D scene and projecting the remaining points according to the same viewpoint. The latter process can be repeated iteratively to obtain the hidden portion of the 3D scene each time. According to a variant, the second part can be obtained by removing points visible from the viewpoint (e.g., the center viewpoint) from the 3D scene (whose textures are stored in the first part) and projecting the remaining points according to viewpoints different from the first viewpoint, such as one or more second viewpoints from a viewing space centered on the center viewpoint (e.g., a 3DoF rendered viewing space).

[0106] The first part 61 can be viewed as the first large texture tile (corresponding to the first part of the 3D scene), and the second part 62 includes smaller texture tiles (corresponding to the second part of the 3D scene that complements the first part). This type of atlas has the advantage of being compatible with both 3DoF rendering (when only the first part 61 is rendered) and 3DoF+ / 6DoF rendering.

[0107] Figure 7 The illustration shows a non-limiting embodiment based on the principles of the invention, including... Figure 6 The depth information of points in the 3D scene and has the same Figure 6 This is an example of a layout similar to that of Atlas 70. Atlas 70 can be seen as corresponding to... Figure 6 The texture image is a depth image of 60.

[0108] Atlas 70 includes a first portion 71 and one or more second portions 62, the first portion including depth information of points in the 3D scene visible from a central viewpoint. Atlas 70 can be obtained in the same manner as Atlas 60, but contains depth information associated with points in the 3D scene instead of texture information.

[0109] For 3DoF rendering of a 3D scene, only one viewpoint is considered, typically a central viewpoint. The user can rotate their head with three degrees of freedom around this primary viewpoint to view different parts of the 3D scene, but the user cannot move this single viewpoint. The points in the scene to be encoded are those visible from this single viewpoint, and only texture information needs to be encoded / decoded for 3DoF rendering. For 3DoF rendering, it is not necessary to encode points in the scene that are not visible from this single viewpoint because the user cannot access them.

[0110] For 6DoF rendering, users can move their viewpoint to various locations within the scene. In this case, every point in the scene (depth and texture) in the bitstream needs to be encoded, as a user moving their viewpoint may access each point. At the encoding stage, there is no prior means of knowing from which viewpoint the user will be viewing the 3D scene.

[0111] For 3DoF+ rendering, the user can move the viewpoint within a limited space around the central viewpoint. This allows for the experience of parallax. Data representing portions of the scene visible from any point in the viewing space is encoded into the stream, including data representing the 3D scene visible from the central viewpoint (i.e., the first portions 61 and 71). For example, the size and shape of the viewing space can be determined and encoded in the bitstream at the encoding step. The decoder obtains this information from the bitstream, and the renderer restricts the viewing space to the space determined by the obtained information. According to another example, the renderer determines the viewing space based on hardware constraints, such as hardware constraints related to the ability of sensors to detect user movement. In this case, if a point visible from a point within the renderer's viewing space has not yet been encoded in the bitstream at the encoding stage, that point will not be rendered. According to yet another example, data representing each point of the 3D scene (e.g., texture and / or geometry) is encoded in the stream, regardless of the rendering viewing space. To optimize the size of the stream, only a subset of the points in the scene can be encoded, such as a subset of the points visible from the rendering viewing space.

[0112] In existing technologies, after decoding the color and depth atlases, the rendering device performs the reverse operation of 3D rendering. Instead of projecting each pixel of each block of the atlas to reconstruct 3D points, immersive rendering devices reproject the 3D points into the viewport of the user's current visual pyramid.

[0113] This means two types of operations on the rendering device:

[0114] - Memory lookup to retrieve the color and depth values ​​of each pixel in the atlas (an operation limited by GPU memory bandwidth); and

[0115] - Calculate to deproject / reproject each point (very suitable for operations on massively parallel GPU architectures).

[0116] In a typical implementation, the rendering engine pipelines the processing of vertex / fragment shadows for each pixel of the atlas, but this triggers several memory lookups equal to the atlas size. For example, for a 4K HMD that supports up to 15 pixels per degree, the atlas consists of more than 17M pixels (5.3K x 3.3K).

[0117] This approach has a significant drawback: the atlas contains tiles for any orientation (360°x180°), while tiles that belong only to the end-user device's field of view (typically 90°x90℉OV of the HMD) are visible in the current viewport; the rendering engine can then read up to 8 times more pixels than necessary.

[0118] According to the principles of the present invention, in order to perform fast rendering on low-end user devices, the number of memory lookups and deprojections is reduced by reading only a subset of tiles in the atlas that are visible in the current user's field of view; that is, only tiles that appear in the user's view direction are selected.

[0119] Figure 8 An implementation of sectorization of a 3D scene's 3D space is illustrated. In this example, a spherical projection and mapping, such as an equal rectangular projection (ERP), is chosen to project points of the 3D scene onto tiles of an atlas. A sector is a non-intersecting portion of the 3D space of the 3D scene (i.e., another sector that does not overlap). In this particular implementation, a sector is defined by a solid angle; that is, the range (theta, phi) points to a reference point in 3D space (e.g., the center point of a 3DoF+ view frame), where theta and phi are polar coordinates.

[0120] exist Figure 8 In this context, the scene space 80 is divided into eight sectors of equal size. Sectors can have different sizes and do not necessarily cover the entire scene space. The number of sectors is chosen to optimize the coding based on the principles detailed herein. Scene 80 includes several objects or portions thereof, from objects 81a to 87a. Figure 5 As shown, the points in the scene are projected onto the blocks. The method ensures that the pixels in the blocks are projections of points from the same sector, selecting the portion of the object to be projected onto the block. Figure 8 In the example, object 87 has points belonging to two sectors. The points of object 87 are then divided into two parts, 86a and 87a, so that when projected, the points of part 86a and part 87a are encoded in two different chunks. Chunks associated with the same sector are packed into the same region of the atlas. For example, a region is a rectangular area of ​​the atlas image, and a region can pack several chunks. In variations, regions can have different shapes; for example, a region can be defined by an ellipse or a general polygon. Chunks within the same region are projections of points from the same sector. Figure 8 In the example, five of the eight sectors of the scene space comprise points. According to this principle, the atlas image 88 representing the scene comprises five regions 891 to 895. Regions are packaged as blocks of projections of points within the same sector. For example, in... Figure 8 In this context, region 891 includes blocks 83b and 84b corresponding to groups of points 83a and 84a belonging to the same first sector. Groups of points 86a and 87a, even though they are parts of the same object 87, also result in two blocks 86b and 87b because they belong to two separate sectors. Block 86b is packed in region 892, while block 87b is packed in a different region 893. Block 85b is packed in region 894 because the corresponding point in the scene belongs to the second sector, and blocks 81b and 82b corresponding to groups of points 81a and 82a, respectively, are packed in region 895 because the groups of points 81a and 82a are included in the same sector.

[0121] In another implementation, for a sequence of scenes, i.e., over a period of time, i.e., for a sequence of atlas images, such as a set of pictures (GOP) for a codec used to compress video, chunks (and the groups of points used to generate them) are determined to belong to a sector. In this implementation, chunks maintain the same size and the same packing properties across a sequence of atlases. Objects (e.g., object 87) can be moved from one sector to another; for example, portion 86a can be moved sequentially toward the sector of portion 87a. In this example, chunk 87b is large enough to encode each point of object 87, and this chunk is initially filled with the projection of point 87a and finally filled with the projection of point 86a plus 87a. Chunk 86b is large enough to encode point 86a, and this chunk is initially filled with the projection of point 86a and finally filled with values ​​indicating "unused pixels".

[0122] Figure 9 A first layout of a sectorization atlas according to the principles of the present invention is shown. In this example, sectorization follows... Figure 8 The space shown is divided into spherical sections. This spatial division corresponds to geographical orange slices or time zones. To utilize... Figure 6 and Figure 7 The central field of view is divided into n sectors (in the central block). Figure 9 In the example, there are 8 sectors) to obtain 8 regions. In this implementation, each central region includes information corresponding to the same angular amplitude, and therefore includes exactly the same number of pixels when projected onto the ERP image. The peripheral blocks are also classified by sectors with the same angular amplitude (in Figure 9 (There are 8 more in the example), and then they are packed into the outer regions 91 to 98. Unlike the central chunk, the amount of data in the outer regions is different because it depends on the amount of parallax information in a given sector. Figure 9In this model, the outer regions can have different sizes. In the variant, the outer regions 91 to 98 have the same size. Unused pixels can be filled with specific values, such as 0 or 255 for the depth atlas and white, gray, or black for the color atlas.

[0123]

[0124] In a 3DoF+ rendering device, the processor manages a virtual camera located within the 3DoF+ viewing area. The virtual camera defines the user's viewpoint and field of view. The processor generates a viewport image corresponding to that field of view. At any given time, based on the user's viewport orientation (theta... 用户 phi 用户 The renderer selects multiple sectors, at least one sector (e.g., 3 or 4 sectors), and then accesses and processes the same number of central and peripheral areas. Figures 8 to 10 In the example, only 37.5% (3 / 8 sector) or 50% (4 / 8 sector) of the tiles were processed. The number of selected sectors can be dynamically adjusted by the renderer based on its CPU and / or GPU capacity. The number of selected sectors at any given time must at least cover the field of view. Additional sectors can be selected to enhance reliability by rendering peripheral tiles at the boundaries of the current FOV for use with the user's lateral and / or rotational movement.

[0125] Figure 10 A second layout of a sectorized atlas according to the principles of the present invention is shown. In this example, the selected projection and mapping are ERPs. Figure 6 and Figure 7 The central block is divided into 10 regions: such as in Figure 8 The example shows eight regions for the large equatorial region and two regions for the poles. Then, ten atlas regions are dedicated to the outer blocks of ten sectors.

[0126]

[0127] and Figure 8 Compared to the first layout, this example layout has advantages. In fact, in terms of rendering, the atlas region corresponding to the pole is only accessed and projected when the user is looking above or below a given angle (depending on the size of the pole region). Figure 8 In this layout, when the user looks towards the pole, each central block (i.e., each region) must be accessed to obtain the necessary information to generate the viewport image. Figure 9 In this layout, only one polar region and some panoramic regions (e.g., four, depending on the width of the field of view) must be accessed to generate the viewport image. Therefore, in each case, fewer regions must be accessed.

[0128] Figure 11A third layout of a sectorized atlas prepared for cube mapping according to the principles of the present invention is shown. Figure 11 In the example, the scene's 3D space is divided into six pyramidal sectors (or truncated cones) corresponding to the six faces of a cube, centered, for example, at the center point of the viewing bounding box. This example atlas layout can be advantageously used for (theta fov phi fov Field of view (FOV) less than (90°, 90°) (theta) fov ,phi fov This FOV feature ensures that all pixels visible within the FOV belong to the three largest faces of the cube map atlas.

[0129] When an atlas is projected from several cube atlases for multiple center and peripheral cameras (amplitude pyramid vision (90°, 90°)), a sector is defined as the set of 3D points in any camera field of view (center and periphery) visible, for example, in the face of the central cube centered at the main central viewpoint. For the ERP case, the concept of a sector for a peripheral block is relative to its position in the Cartesian coordinate system of the center camera, rather than the peripheral block from which it originates.

[0130]

[0131] At any given time, the renderer selects three regions corresponding to the cubemap projection of the three faces of the cube visible in the current user's visual pyramid. Only the pixels of these three regions are accessed to generate the viewport image.

[0132] The sectorization of scene space must be signaled in the data stream, for example, with metadata associated with a region-based atlas. A possible syntax for such metadata would be to use the following semantic elements:

[0133] Sector identifier (uint(8)): Sector identifier. A sector corresponds to a continuous volume in scene space relative to the coordinate system. For example, (but not limited to) a vertical slice of a sphere. Other possible volumes can be, for example, a visual pyramid (cube mapping), a bounding box, or all pixels visible in a cylindrical slice (all pixels enclosed around the vertical axis at an angle θ). Several camera views can have pixels belonging to the same sector.

[0134] All volumetric data appearing in a given sector i in 3D space are stored in the same compact 2D region R of the atlas. i .

[0135] Sector descriptor: Describes a portion of the scene space relative to a virtual camera containing a segmented subset.

[0136]

[0137]

[0138]

[0139]

[0140]

[0141] The atlas parameter syntax is used to add the Sector_Id field to the metadata of each chunk. This syntax element can include other data relative to the chunk. The Object_Id parameter can also be used, for example, to mark several chunks with different Object_Ids when they share the same Sector_Id. These two attributes are complementary and lateral.

[0142]

[0143]

[0144] Wherein, Sector_Id[a][i] is the value of Sector_Id of the i-th block of the a-th atlas.

[0145] Advanced syntax includes a list of sector descriptors:

[0146]

[0147] The sector parameter list is defined as follows. Num_sectors_minus1 gives the number of different sectors. num_sectors_descriptors_minus1 gives the number of sector_descriptors, and there is one sector_descriptor for each sector_Id.

[0148]

[0149] Atlas_layout_descriptor (optional): This descriptor is optional because the implicit layout of the atlas can be inferred by parsing the Sector_Id information of each block in atlas_param(). This is presented as an alternative syntax that directly expresses the corresponding sector in the atlas. i 2D region {R i The definition of}.

[0150]

[0151] An atlas layout, i.e., region partitioning and tile organization within each region, can be determined for one or more picture groups (GOPs). Atlas images within the same GOP (e.g., 8 or 16 consecutive images from a video stream) can share the same layout. Subsequent GOPs can have different layouts, for example, by having different region partitioning and / or different tile organization within the regions.

[0152] The decoding process can be implemented through the following steps. At each new GOP, the renderer reads the Sector_Id information from the metadata stream to determine the internal layout of the atlas for the duration of the GOP. The renderer reads and calculates, directly or indirectly, the number of sectors N, the definition of sector regions R1...RN in the atlas, and the valid angular sectors (theta, phi) and bounding boxes. The rendering device has a tracking system (e.g., an IMU for an HMD or mouse, or an external tracking system) that returns the position and orientation (theta) of the virtual camera at each frame. 用户 ,phi 用户 ), (x 用户 ,y 用户 ,z 用户 ).

[0153] Each time the screen is refreshed, the renderer recomposes a new viewport frame and retrieves values ​​from the atlas so that:

[0154] - Determine which sectors match the current user pose (position + orientation) and construct a candidate Sector_Id list. This list includes all sectors declared in the metadata, which may appear in the visual pyramid and field of view centered in the user's orientation. FOV, phi FOV )middle.

[0155] - Optionally, this list can be over-provisioned by appending adjacent sectors that increase scene coverage. This over-provisioning ratio can be a recommended value included in the metadata, or it can be dynamically adjusted by the renderer depending on its CPU and / or GPU capacity and the user's current position in the scene. For stationary positions (x... 用户 =0,y 用户 =0,z 用户 =0), you can choose to cover at least the current FOV oversupply ratio. The more users move laterally from the center location (e.g., y 用户 =20cm), the oversupply ratio should be increased as much as possible to include lateral sectors on the FOV boundary. This will essentially trigger a read of one or two adjacent sectors on the periphery of the FOV.

[0156] - Configure shaders using the Sector_Ids list to synthesize the next frame.

[0157] - This causes the shader to read only the relevant pixels in the atlas (i.e., the pixels in the region of the atlas associated with the selected sector) before deprojecting (reconstructing 3D points) and reprojecting (placing them in the viewport of the user's visual pyramid).

[0158] In the prior art, one possible implementation is to allocate k pixel grids M each time the atlas layout changes (e.g., possibly at each GOP). i , grid M i Coverage area R in the map set i All pixels.

[0159] Another possible implementation is based on using a single static universal grid for the entire atlas.

[0160] Figure 12 A method 120 for generating a viewport image of a scene encoded in an atlas image, according to a non-limiting embodiment of the invention, is shown. At step 121, an atlas is obtained from a data stream. The data stream can be read from memory or received from a network. Metadata is obtained simultaneously from the same or different sources. The metadata describes the atlas's division into regions and the regions associated with sectors in the scene space. Pixels in the atlas region are projections of points of the scene located in the sectors associated with the region. At step 122, the renderer selects a set of sectors based on the current viewpoint and field of view of a user-controlled virtual camera. In practice, 3DoF+ rendering devices include means for user-controlled virtual cameras to visualize the scene from a viewpoint in the scene space's field of view. Sectors within the viewer's field of view are selected. In variations, adjacent sectors are selected to anticipate movement of the virtual camera. At step 123, the renderer accesses pixels in the atlas region associated with the selected region to generate a viewport image of the scene as seen from the user's viewpoint. An advantage of this method is that the number of pixels accessed, and therefore the number of points to be projected, is limited to points in the viewing sectors of the scene space.

[0161] Figure 13 A method 130 for encoding a scene according to a non-limiting embodiment of the principles of the present invention is illustrated. At step 131, the scene is divided into non-overlapping sectors. Chunks are obtained by these sectors. A chunk is an image as a projection of points of the scene included in a unique sector. At step 132, a region layout for the atlas is selected. Each region of the layout is associated with a unique sector in the scene space. A sector may be associated with several regions. At this step, chunks are packaged within the regions associated with their respective sectors. The atlas is generated and encoded in a data stream, wherein metadata describes the regions and associates the regions with sectors in the scene space. In variations, the metadata is encoded in different data streams.

[0162] The specific implementations described herein may be implemented, for example, in methods or processes, apparatus, computer program products, data streams, or signals. Even if discussed only in the context of a single form of implementation (e.g., discussed only as a method or apparatus), the specific implementations of the discussed features may also be implemented in other forms (e.g., programs). Apparatus may be implemented, for example, in suitable hardware, software, and firmware. Methods may be implemented in apparatus (such as, for example, a processor) that generally refers to a processing device, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices, such as, for example, smartphones, tablets, computers, mobile phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate communication of information between end users.

[0163] Specific implementations of the various processes and features described herein can be found in a wide variety of devices or applications, particularly those associated with data encoding, data decoding, view generation, texture processing, and other processing of images and related texture and / or depth information. Examples of such devices include encoders, decoders, post-processors that process the output from decoders, pre-processors that provide input to encoders, video encoders, video decoders, video codecs, web servers, set-top boxes, laptops, personal computers, cellular phones, PDAs, and other communication devices. It should be understood that the devices can be mobile, even mounted in mobile vehicles.

[0164] Additionally, the method can be implemented by instructions executed by a processor, and such instructions (and / or data values ​​generated by the implementation) can be stored on a processor-readable medium, such as, for example, an integrated circuit, a software carrier, or other storage device, such as, for example, a hard disk, a compact disk (“CD”), an optical disk (such as, for example, a DVD, commonly referred to as a digital versatile optical disk or digital video optical disk), random access memory (“RAM”), or read-only memory (“ROM”). Instructions can form an application program tangibly embodied on the processor-readable medium. Instructions can be, for example, hardware, firmware, software, or a combination thereof. Instructions can be found, for example, in an operating system, a standalone application, or a combination of both. Thus, a processor can be characterized, for example, as a device configured to execute a process and a device comprising a processor-readable medium (such as a storage device) having instructions for executing the process. Furthermore, in addition to or instead of instructions, the processor-readable medium can store data values ​​generated by the implementation.

[0165] It will be apparent to those skilled in the art that the embodiments may produce various signals formatted to carry, for example, storable or transmissible information. The information may include, for example, instructions for performing a method or data generated by one of the embodiments. For example, the signal may be formatted as data carrying rules for writing or reading the syntax of the described embodiment, or as data carrying actual syntax values ​​written by the described embodiment. Such signals may be formatted as, for example, electromagnetic waves (e.g., using the radio frequency portion of the spectrum) or baseband signals. Formatting may include, for example, encoding a data stream and using a modulated carrier for the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is known, the signal can be transmitted via a variety of different wired or wireless links. The signal may be stored on a processor-readable medium.

[0166] Several specific embodiments have been described. However, it should be understood that many modifications can be made. For example, elements of different embodiments can be combined, supplemented, modified, or removed to produce other embodiments. Furthermore, those skilled in the art will understand that other structures and processes can be replaced with those disclosed, and the resulting embodiments will perform at least substantially the same function in at least substantially the same manner to achieve at least substantially the same results as the disclosed embodiments. Therefore, this application considers these and other embodiments.

Claims

1. A method, the method comprising: - Obtain an image from a data stream divided into a central region and a peripheral region, wherein the central region is a portion of a central block generated by projecting points of a 3D scene based on a central viewpoint, and the peripheral region is packed with peripheral blocks generated by projecting points of the 3D scene based on a viewpoint different from the central viewpoint, the data stream including metadata that associates each central region and each peripheral region of the image with sectors of the space of the 3D scene; - Select at least one sector based on the field of view in the space of the 3D scene; as well as - A viewport image of the 3D scene for the field of view is generated by accessing pixels of the central and peripheral regions of the image that are associated with at least one selected sector in the metadata.

2. The method according to claim 1, wherein, The image is packaged with segmented images, one of which belongs to a region and is a projection of points in the sector associated with the region of the 3D scene.

3. The method according to claim 1 or 2, wherein, The sectors of the space in the 3D scene are associated with two regions, wherein the first region is the central region and the second region is the peripheral region.

4. An apparatus comprising a processor configured to: - Obtain an image from a data stream divided into a central region and a peripheral region, wherein the central region is a portion of a central block generated by projecting points of a 3D scene based on a central viewpoint, and the peripheral region is packed with peripheral blocks generated by projecting points of the 3D scene based on a viewpoint different from the central viewpoint, the data stream including metadata that associates each central region and each peripheral region of the image with sectors of the space of the 3D scene; - Select at least one sector based on the field of view in the space of the 3D scene; as well as - A viewport image of the 3D scene for the field of view is generated by accessing pixels of the central and peripheral regions of the image that are associated with at least one selected sector in the metadata.

5. The device according to claim 4, wherein, The image is packaged with segmented images, one of which belongs to a region and is a projection of points in the sector associated with the region of the 3D scene.

6. The device according to claim 4 or 5, wherein, The sectors of the space in the 3D scene are associated with two regions, wherein the first region is the central region and the second region is the peripheral region.

7. A method, the method comprising: - Generate an image divided into a central region and peripheral regions, wherein the central region is a portion of a central block generated by projecting points of the 3D scene based on a central viewpoint, and the peripheral regions are comprised of peripheral blocks generated by projecting points of the 3D scene based on a viewpoint different from the central viewpoint. Each central region and each peripheral region is associated with a sector of the space of the 3D scene, the sector being a non-intersecting portion of the space defined by a solid angle pointing towards the central viewpoint; wherein points of the 3D scene included in a given sector are projected into pixels of a block image, the block image being comprised of regions associated with the given sector; and - Encode the image and metadata in the data stream that associates each region of the image with a sector of the space of the 3D scene.

8. The method according to claim 7, wherein, The image is packaged with segmented images, one of which belongs to a region and is a projection of points in the sector associated with the region of the 3D scene.

9. The method according to claim 7 or 8, wherein, The sectors of the space in the 3D scene are associated with two regions, wherein the first region is the central region and the second region is the peripheral region.

10. An apparatus comprising a processor configured to: - Generate an image divided into a central region and peripheral regions, wherein the central region is a portion of a central block generated by projecting points of the 3D scene based on a central viewpoint, and the peripheral regions are comprised of peripheral blocks generated by projecting points of the 3D scene based on a viewpoint different from the central viewpoint. Each central region and each peripheral region is associated with a sector of the space of the 3D scene, the sector being a non-intersecting portion of the space defined by a solid angle pointing towards the central viewpoint; wherein... Points in the 3D scene included in the sector are projected into pixels of a tiled image, which is then packaged in a region associated with the sector. as well as - Encode the image and metadata in the data stream that associates each region of the image with a sector of the space of the 3D scene.

11. The device according to claim 10, wherein, The image is packaged with segmented images, one of which belongs to a region and is a projection of points in the sector associated with the region of the 3D scene.

12. The device according to claim 10 or 11, wherein, The sectors of the space in the 3D scene are associated with two regions, wherein the first region is the central region and the second region is the peripheral region.

Images