Encoding and decoding views on volumetric image data

Abstract

An encoding method includes: obtaining (101) an input set of volumetric image data; selecting (103) data from the image data for multiple views such that the multiple views include only a portion of the image data, the selection being based on the visibility of the data from the respective viewpoint in the respective viewing direction and / or within the respective field of view; encoding (105) each of these views into a separate output set (31); and generating (107) metadata indicating these viewpoints. A decoding method includes: determining (121) a desired user viewpoint; obtaining (123) metadata; selecting (125) one or more available viewpoints based on the desired user viewpoint; obtaining (127) one or more image datasets in which one or more available views corresponding to the selected one or more available viewpoints have been encoded; and decoding (129) at least one of the one or more available views.

Description

Technical Field

[0001] The present invention relates to an encoder system for encoding volumetric image data (e.g., point clouds) and a decoder system for decoding encoded volumetric image data.

[0002] The present invention further relates to a method for encoding volumetric image data and a method for decoding encoded volumetric image data.

[0003] The present invention also relates to a computer program product that enables a computer system to perform such a method. Background Technology

[0004] Augmented reality (AR) and virtual reality (VR) offer a compelling set of use cases, such as remotely watching live sports events, sharing and social VR, (serious) gaming, and training and education. Such experiences allow viewers to connect remotely. For a truly immersive experience in both AR and VR, the viewer needs six degrees of freedom (6DoF). That is, when wearing an AR headset or VR display, the viewer should experience changes in the environment as they move their head in all directions (i.e., changing head position forward / backward (surge), up / down (up / down), and left / right (sway) and simultaneously changing orientation by rotation along the lateral (yaw), lateral (pitch), and longitudinal (roll) axes), and more generally, while moving their head while their head can remain stationary relative to their body.

[0005] A volumetric format is needed to describe and thus allow rendering environments that viewers can experience at 6DoF. One aspect of this volumetric format is volumetric video formats, which have been created to describe volumetric environments that change dynamically over a given time. The AR and VR industries are moving towards this format. For example, in the use case mentioned above, volumetric capture models can be used to make the user's image more realistic.

[0006] For example, volumetric image data can include point clouds (PCs), voxels, or volumetric (polygon) meshes. Meshes are used, for example, to describe 3D models in games. Point clouds can be used to describe volumetric objects as a set of points that can then be used in a virtual scene. A point cloud is a type of image data used to describe a volumetric object using a (typically very large) set of three-dimensional... This method represents 3D data as points, where x, y, and z typically refer to Cartesian coordinates, but other formats also exist (e.g., 3D reference points with angles x and y on a sphere of radius z (e.g., [0,0,0])). Depending on the data type represented, each point can have additional attributes assigned to it (e.g., color, reflectivity, surface orientation, timestamp, movement).

[0007] Points in a point cloud are typically considered to have zero volume (in other words, they are generally considered to have no defined size / dimension). Several techniques have been described in the literature for meaningfully rendering such points. One simpler approach is to assign a thickness value to each point before or during rendering. Using this thickness, each point can be represented by a 3D object (e.g., a tiny sphere, voxel, hexagon, or other shape), making it visible and allowing other points behind it to be hidden.

[0008] Point clouds are well-suited as a storage format for the output of a series of measurement and capture devices. In particular, RGB cameras combined with and synchronized with infrared time-of-flight (ToF) sensors (such as Microsoft Kinect) are often used to sense depth and color information, which can be combined and represented as a point cloud. Another technology that has led to the use of point clouds is LiDAR, a technology primarily used in autonomous vehicles and remote sensing.

[0009] A mesh is a 3D structure composed of multiple connected points or vertices. Vertices can be connected and closed to form (planar) surfaces. Graphics cards (GPUs) are typically optimized for rendering large groups of meshes consisting of three or four faces. Increasing the number of vertices can better approximate an object. Meshes can be constructed programmatically and / or defined using 3D modeling software. For example, there are many methods for explicitly storing mesh data, and therefore many public and proprietary formats for this purpose, such as 3DS, OBJ, GLTF, and PLY formats.

[0010] A voxel, or 'volume pixel,' is a data structure used to represent volumetric data. Typically, voxels are defined on a 3D 'voxel grid' composed of cells of similar size. In practice, voxels are used to represent various volumetric measurements and samplings, finding applications in medicine, geospatial fields, and more generally in computer graphics. Thus, voxels commonly appear in the representation of MRI, CT and ultrasound scans, satellite measurements, and topographic height maps.

[0011] When rendering a 3D scene, the amount of content described (e.g., the entire scene) is often more than what is ultimately visible to the user (e.g., the view). This fact is commonly used in graphics pipelines to optimize rendering by first detecting and then ignoring (partial) objects that do not affect the user's view. This type of operation is typically called "culling" (e.g., removing unwanted objects). Two important types of culling are "frustum culling" and "occlusion culling."

[0012] The most straightforward form of culling (frustum culling) removes objects that are not within the view volumes of (multiple) virtual camera objects. In practice, such camera objects are typically modeled using a pyramid (because the virtual screen is square), where the 'near' and 'far' (minimum and maximum drawing distance) planes are cut off, resulting in an object called a 'truncated pyramid'. The frustum culling algorithm involves detecting objects within the truncated pyramid volume and removing (i.e., culling) a portion of all other objects, as these other objects will become invisible.

[0013] Additionally, "occlusion culling" is another commonly used technique for selecting content that is visible only from a given viewpoint. It does not consider rendering all (partial) objects occluded by other objects within the viewer's line of sight. Detecting occlusion in a scene from a given viewpoint is a well-known problem in computer graphics, where numerous algorithms have been developed to attempt to solve the more general 'hidden surface problem' (and its analogues, the 'visible surface problem'), i.e., determining which (partial) surfaces in a scene are invisible (and therefore can be removed without affecting the output of the rendering algorithm) given a particular viewpoint. A wealth of literature is available on this topic, and implementations of such algorithms exist in current state-of-the-art systems.

[0014] Rendering engines (such as Unity and Unreal Engine) typically provide occlusion culling before the rendering phase. The occlusion culling process traverses the scene using a set of virtual cameras, determining the visible portion of the scene from each camera's perspective. Ultimately, only visible objects are rendered. The occlusion culling process accounts for a significant portion of the computation time during content rendering.

[0015] The larger the volumetric image data (e.g., point cloud) provided to the rendering system for rendering, the more processing (including culling) needs to be performed. This results in the rendering system consuming a significant amount of computation time. Therefore, it is desirable to minimize the size of the volumetric image data provided to the rendering system for rendering.

[0016] The Cesium platform, including CesiumJS and Cesium ion, for 3D geospatial data has been developed by a company called Cesium. The technology offered by this platform also reduces the size of the volumetric image data provided to the rendering system for rendering. The Cesium platform achieves this using 3D tiles. Cesium tiles use literal spatial (e.g., geographic) subdivision and aim to efficiently stream (and display) point clouds from large areas. Large areas consist of many point clouds, but with 3D tiles, only the desired data can be requested. A drawback of the Cesium platform is that it is designed for 3D geospatial data, not for augmented reality and / or virtual reality content. An additional drawback is that Cesium tiles are defined by arbitrarily dividing 3D space into segments separate from any viewing location. Therefore, data within a Cesium tile may or may not be visible from a particular viewing location, meaning that clients rendering large point clouds will need to download Cesium tiles containing occlusion data for that rendered viewing location. Summary of the Invention

[0017] The first object of the present invention is to provide an encoder system that limits the amount of volumetric image data to be transmitted.

[0018] A second objective of the present invention is to provide a decoder system that uses limited hardware resources to render volumetric image data.

[0019] A third objective of this invention is to provide a method for encoding volumetric image data that limits the amount of volumetric image data to be transmitted.

[0020] A fourth objective of this invention is to provide a method for decoding encoded volumetric image data, which requires limited hardware resources to render the volumetric image data.

[0021] In a first aspect of the invention, an encoder system includes at least one processor configured to: obtain an input set of volumetric image data; for each of a plurality of views on the volumetric image data, select data from the volumetric image data such that the plurality of views include only a portion of the volumetric image data, the selection being based on the visibility of the data from the respective viewpoint in a respective viewing direction and / or within a respective field of view; encode each of the views into a separate output set of volumetric image data; and generate metadata indicating the plurality of viewpoints.

[0022] By segmenting the input set of volumetric image data into, for example, frustum, pyramid, or conical views, and by performing frustum culling (and optional occlusion culling, typically removing fully occluded data rather than partially occluded data) on these input sets, the encoder system can emit a limited amount of content to the decoder system. This results in the decoder system receiving only a portion of the input set—the portion necessary to decode enough volumetric image data to render the input set from the desired user viewpoint. This limits the amount of volumetric image data the decoder system needs to render content (e.g., augmented reality and / or virtual reality content) without requiring any specific action to limit the amount of volumetric image data to a certain maximum. Due to the limited size of the volumetric image data, the decoder system requires limited / less computational resources (and typically limited / less memory and storage) to render invisible points or perform frustum culling and occlusion culling, and the use of network resources such as bandwidth may also be limited / reduced compared to systems that emit the entire input set of volumetric image data (e.g., an entire point cloud) or a partial set containing occluded data that is useless for rendering (e.g., an arbitrary "cube" of points emitted as tiles by Cesium).

[0023] For example, the shape of the view can be like a truncated pyramid, a prism, or a cone. For example, a single wide view (e.g., a truncated pyramid shape) can be used to render views for both eyes simultaneously. The metadata is associated with the multiple viewpoints and includes 3D positional information. The metadata can describe other characteristics of each view, such as viewing direction and / or field of view. It is beneficial to include the viewing direction and / or field of view in the metadata if the views do not all have the same viewing direction and / or field of view, for example, if one or more of the viewpoints are inside the point cloud and the viewpoints are not associated with a default viewing direction. Multiple output sets can be associated with the same metadata / viewpoint, for example, if multiple output sets represent multiple qualities of the same content. Information specifying which viewpoints and / or viewing directions and / or fields of view to use can be obtained from the input data, for example, as part of or separate from the same input data that makes up the input set. For example, the viewing direction and / or field of view can alternatively be the default viewing direction and / or default field of view. In this specification, encoded volumetric data is also referred to as “volume data,” even if it is only indirect volumetric (i.e., through decoding).

[0024] The default viewing direction and / or field of view are those known at runtime and not provided in the signaling information. Different use cases may use different default values. Default values ​​can dynamically depend on another default value (however, for any set of static default parameters, there is typically only one set of (static + dynamic) default parameters). For example, the encoder and decoder systems can obtain default values ​​by compiling them into the software, by specifying rules for how to calculate / determine the default values ​​(e.g., in a standard), or by having another component determine these values ​​and provide them as input to the encoder and decoder systems.

[0025] The at least one processor can be configured to select the data from the volumetric image data for each of the plurality of views in such a way that, for each corresponding view, all volumetric image data visible from the corresponding viewpoint in the corresponding viewing direction and / or within the corresponding field of view is selected from the volumetric image data. Alternatively, some data visible from the corresponding viewpoint in the corresponding viewing direction can be omitted, for example, if the data does not have a significant impact on rendering, to further limit the total amount of volumetric image data, or simply to reduce coding effort / time. For example, in some viewing directions, two non-overlapping points in the point cloud may significantly overlap during rendering (and one of them may be almost invisible), and therefore it may not be necessary to render both points simultaneously.

[0026] The at least one processor can be configured to select additional data for the plurality of views based on the visibility of the additional data from one or more corresponding additional viewpoints associated with the respective viewpoint. For example, the data and additional data may correspond to adjacent or partially overlapping views. Since the desired user viewpoint is often different from one of the available viewpoints, the decoder system typically requires some adjustments to adjust the decoded view to the desired user viewpoint. Typically, some data visible from the selected available viewpoint is not visible from the desired user viewpoint and is therefore omitted in the rendered view. Furthermore, some data visible from the desired user viewpoint is not visible from the selected available viewpoint. By providing additional data, not just data visible from the respective viewpoint in the corresponding viewing direction and / or within the corresponding field of view in the view, it is still possible to render data visible from the desired user viewpoint but not from the available viewpoints.

[0027] Alternatively, the decoder system can obtain multiple views and merge them. In both cases, the computation time required for frustum culling and occlusion culling and / or rendering invisible data is generally less than when the entire input set is provided to the decoder system for rendering. Additional data for a viewpoint can be selected either by virtually moving that viewpoint or by selecting data near the data already selected for the view. The former leads to optimal results (e.g., no data is missed), but the latter can be performed faster.

[0028] The at least one processor can be configured to specify in the metadata where to obtain the output set of the volumetric image data or a portion thereof. This gives the decoder system complete control over which output sets / views are obtained. For example, the metadata can be specified in a manifest file, such as a Media Presentation Description (MPD) of MPEG-DASH (ISO / IEC 23009-1). In another embodiment, the metadata can be specified in the ISO Basic Media File Format (ISOBMFF) data format, in which each output set of the volumetric image data can be encapsulated into different tracks within the same ISOBMFF (ISO / IEC 14496-12) file.

[0029] The input set of the volumetric image data may include one or more point clouds. Point clouds are relatively easy to segment into views; for example, a subset of the point cloud is valid and requires no further processing. If the volumetric image data includes multiple point clouds, these point clouds constitute a single scene, i.e., they are spatially related. For example, one point cloud may represent a table in a room and another point cloud may represent a chair in the same room. If the volumetric image data includes multiple point clouds, culling can be performed only on subsets of these multiple point clouds. Although 3D meshes can also be segmented into views, this may require creating new polygons to replace the original polygons, for example, because a portion of these original polygons is occluded in a certain view. Other examples of volumetric image data are RGB+D image data and voxels. The input set of the volumetric image data may vary over time, and the at least one processor may be configured to repeatedly select the data from the volumetric image data for each of the views and encode each of the views into a separate bitstream.

[0030] The multiple views can collectively include all the volumetric image data. This allows the user to view all the volumetric image data by changing the user's viewpoint. Alternatively, some data may not be in any view, for example, because it is hidden from every viewpoint.

[0031] In a second aspect of the invention, a decoder system includes at least one processor configured to: determine a desired user viewpoint; obtain metadata associated with encoded volumetric image data, the metadata indicating available viewpoints, each of which corresponds to an available view; select one or more of the available viewpoints based on the desired user viewpoint; obtain one or more volumetric image datasets based on the selected one or more viewpoints, in which one or more available views corresponding to the selected one or more viewpoints have been encoded; decode at least one available view from the one or more volumetric image datasets; and optionally render at least one (or a portion thereof) of the decoded one or more available views. The term viewpoint refers to a location. Multiple views may be possible from a given viewpoint, for example, depending on the viewing direction.

[0032] For example, the decoder system may be a mobile device or part of a mobile device. The decoder system may include one or more devices. For example, the shape of the view may be like a truncated pyramid, a prism, or a cone. The metadata is associated with the plurality of viewpoints and includes 3D positional information. The metadata may describe other characteristics of each view, such as viewing direction and / or field of view. Including viewing direction and / or field of view in the metadata is beneficial if views do not all have the same viewing direction and / or field of view, for example, if one or more viewpoints are inside a point cloud and the viewpoints are not associated with a default viewing direction.

[0033] Preferably, at least one of the one or more available viewpoints is selected in such a way that the corresponding at least one available view is the most similar to the desired user view (corresponding to the desired user viewpoint). The decoder system may first select one or more views in this manner and then further select the view that is closest to the volumetric image to achieve the highest quality. Rendering can occur in parallel with decoding (e.g., a partially decoded point cloud may have already been rendered). In the case of multiple views, each view can be rendered separately.

[0034] Since the desired user viewpoint is often not exactly the same as one of the available viewpoints, decoder systems typically require some adjustments to adjust the decoded view to the desired user viewpoint. To prevent rendering invisible data or reduce the amount of rendered invisible data, decoder systems may perform occlusion culling and / or frustum culling to obtain the adjusted view.

[0035] In this embodiment, the decoder system includes a CPU, a desired viewpoint generator (e.g., an HMD, mouse, or other input device), a GPU, and an optional display. In this embodiment, rendering occurs on the GPU, but the CPU receives the desired viewpoint and instructs the GPU to render it. The decoded image can be stored in memory and then sent to any system that needs the decoded image, such as the GPU, system bus, and / or network interface.

[0036] The at least one processor may be configured to: determine an additional desired user viewpoint; select an additional available viewpoint from the available viewpoints based on the additional desired user viewpoint; obtain an additional volumetric image dataset in which additional available views corresponding to the additional available viewpoints have been encoded; decode the additional available views from the additional volumetric image dataset; merge the decoded additional available views with the at least one decoded available view; and optionally render the merged additional available view after rendering the at least one decoded available view or instead of rendering the at least one decoded available view.

[0037] Blending can be used to prevent inconsistencies in rendering when switching viewpoints. Blending typically involves smoothing the view using known techniques to hide the 'seams' between different views from different viewpoints, which can be introduced by lossy compression artifacts and / or overlap between views. Not only can the expected user viewpoint change from one moment to another, but the available viewpoint can also change every frame or every frame sequence. In both cases, a new selection from the available viewpoint is required.

[0038] The at least one processor may be configured to: obtain an additional volumetric image dataset in which data from one or more related views have been encoded, the one or more related views being associated with the one or more available views; decode at least one of the one or more related views from the additional volumetric image dataset; merge the decoded at least one related view with the decoded at least one available view; and optionally render the merged view.

[0039] The one or more related views can be considered related to the one or more available views in the following circumstances:

[0040] Content creators define it this way (e.g., by creating a group of views). This is helpful in scenarios where the set of user locations is limited. Such scenarios are typically defined as scenes or location maps within one or more scenes.

[0041] They share some common properties (e.g., the difference in viewing direction is less than a constant c, or their relative distance is less than a constant d).

[0042] • Automatically generated views can be grouped by the encoder system. For example, if the encoder system starts with a set of 'anchor' views but needs more views to cover more scenes, the encoder system can define additional views associated with the known views.

[0043] For example, the one or more related views can be adjacent, separate, or overlapping. Related views may partially overlap with the available views, or one of the views may be entirely contained within the other. Since the desired user viewpoint is typically different from one of the available viewpoints, an available view may not include all the data visible from the desired user viewpoint. By obtaining and fusing the related views, all the data visible from the desired viewpoint can be obtained and rendered. In this case, fusing is not only performed when switching viewpoints. Frustum culling and / or occlusion culling can be performed on the fused view.

[0044] In some cases, an additional set (i.e., a relevant view) can be obtained before the first set, for example, because the first image dataset takes longer to acquire than the other sets (e.g., it may contain more points or be more difficult to compress). In this case, the additional set can be decoded and rendered while the first set has not yet been (yet) decoded and rendered. For example, the first set may arrive too late, and the additional set may be the only one available. This allows at least some of what might be more relevant to the user to be displayed, rather than nothing. If, for non-(fully) real-time content, the additional set has already been pre-acquired, for example, based on estimated user movement, then the additional set can also be obtained before the first set.

[0045] The at least one processor can be configured to obtain metadata indicating the available viewpoints and specifying where a volumetric image dataset or a portion thereof is obtained, in which available views corresponding to the available viewpoints have been encoded.

[0046] The metadata may further indicate the viewing direction and / or field of view and / or additional viewpoint configuration for each of the available viewpoints. Examples of additional viewpoint configurations are camera projection type, camera orientation, near / far clipping plane, zoom level, lens shape, speed, acceleration, anti-aliasing level, anti-aliasing type, anisotropic filtering level, gamma correction, contrast, and brightness.

[0047] In a third aspect of the invention, a method for encoding volumetric image data includes: obtaining an input set of volumetric image data; selecting data from the volumetric image data for each of a plurality of views on the volumetric image data, such that the plurality of views include only a portion of the volumetric image data, the selection being based on the visibility of the data from a corresponding viewpoint in a corresponding viewing direction and / or within a corresponding field of view; encoding each of the views into a separate output set of volumetric image data; and generating metadata indicating the plurality of viewpoints. The method can be performed by software running on a programmable device. This software can be provided as a computer program product.

[0048] In addition to multiple views, the entire input set of volumetric image data can also be encoded into a separate output set of volumetric image data. This can be used, for example, to provide a low-quality version of the entire input set of volumetric image data at any given time, thus avoiding the "blank" effect of not being able to display data corresponding to a portion of the scene. If the client has sufficient bandwidth and computing resources after downloading the low-quality version, the client will be able to choose to obtain high-quality available volumetric image data instead of the low-quality available volumetric image data. In this way, the scene can always be rendered, even under conditions of rapid user movement and high network latency.

[0049] In a fourth aspect of the invention, a method for decoding encoded volumetric image data includes: determining a desired user viewpoint; obtaining metadata associated with the encoded volumetric image data, the metadata indicating available viewpoints, each of the available viewpoints corresponding to an available view; selecting one or more of the available viewpoints based on the desired user viewpoint; obtaining one or more volumetric image datasets based on the selected one or more viewpoints, in which one or more available views corresponding to the selected one or more viewpoints have been encoded; decoding at least one available view from the one or more volumetric image datasets; and optionally rendering the at least one available view (or a portion thereof) of the decoded one or more available views. The method can be performed by software running on a programmable device. This software can be provided as a computer program product.

[0050] For example, metadata can be received before rendering the content item associated with it, or the selection of content items and the receipt of metadata can be performed in parallel. Content items can be selected after the desired user viewpoint has been determined. For example, if multiple content items with similar content exist, the content item can be selected based on the desired user viewpoint.

[0051] Furthermore, a computer program for performing the methods described herein is provided, as well as a non-transitory computer-readable storage medium for storing the computer program. The computer program may, for example, be downloaded or uploaded to an existing device, or stored during the manufacture of these systems.

[0052] A non-transitory computer-readable storage medium stores at least a first software code portion configured, when executed or processed by a computer, to perform executable operations including: obtaining an input set of volumetric image data; selecting data from the volumetric image data for each of a plurality of views, such that the plurality of views include only a portion of the volumetric image data, the selection being based on the visibility of the data from the respective viewpoint in a respective viewing direction and / or within a respective field of view; encoding each of the views into a separate output set of volumetric image data; and generating metadata indicating the plurality of viewpoints.

[0053] A non-transitory computer-readable storage medium stores at least a second software code portion configured, when executed or processed by a computer, to perform executable operations including: determining a desired user viewpoint; obtaining metadata associated with encoded volumetric image data, the metadata indicating available viewpoints, each of which corresponds to an available view; selecting one or more of the available viewpoints based on the desired user viewpoint; obtaining one or more volumetric image datasets based on the selected one or more viewpoints, in which one or more available views corresponding to the selected one or more viewpoints have been encoded; decoding at least one available view from the one or more volumetric image datasets; and optionally rendering the at least one available view (or a portion thereof) of the decoded one or more available views. These two software portions can operate independently and generally do not need to operate independently of each other.

[0054] As those skilled in the art will recognize, aspects of the present invention can be implemented as devices, methods, or computer program products. Therefore, aspects of the present invention can take the form of entirely hardware embodiments, entirely software embodiments (including firmware, resident software, microcode, etc.), or combined software and hardware aspects that are generally referred to herein as “circuit,” “module,” or “system.” The functionality described in this disclosure can be implemented as algorithms executed by a computer’s processor / microprocessor. Furthermore, aspects of the present invention can take the form of computer program products implemented on one or more computer-readable media having, for example, computer-readable program code implemented thereon, stored thereon.

[0055] Any combination of one or more computer-readable media can be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any suitable combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connection devices having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the context of this invention, a computer-readable storage medium can be any tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0056] Computer-readable signal media may include propagated data signals having computer-readable program code implemented therein (e.g., in baseband or as part of a carrier wave). Such propagated signals may take any of a variety of forms, including, but not limited to, electromagnetic, optical, or any suitable combination thereof. Computer-readable signal media may be any computer-readable medium that is not a computer-readable storage medium and can communicate, propagate, or transmit programs (for use by or in conjunction with an instruction execution system, apparatus, or device).

[0057] The program code implemented on a computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, fiber optic, cable, RF, etc., or any suitable combination thereof. The computer program code for performing the operations of various aspects of the invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java™, Smalltalk, C++, functional programming languages, hardware description languages, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0058] The aspects of the invention will now be described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, specifically a microprocessor or central processing unit (CPU), to produce a machine that, when executed via the processor of the computer or other programmable data processing apparatus or other device, creates means for implementing the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams.

[0059] These computer program instructions may also be stored in a computer-readable medium that can instruct a computer, other programmable data processing apparatus, or other device to function in a particular manner, such that the instructions stored in the computer-readable medium produce manufactured articles comprising the instructions, which implement the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0060] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus or other equipment to cause a series of operational steps to be performed on the computer, other programmable apparatus or other equipment to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide for implementing the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0061] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function(s). It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a non-linear order. For example, depending on the function involved, two consecutively shown blocks may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order. It will also be noted that each block of the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented by a system based on dedicated hardware or a combination of dedicated hardware and computer instructions that performs a particular function or action. Attached Figure Description

[0062] These and other aspects of the invention will become apparent by way of example with reference to the accompanying drawings, in which:

[0063] Figure 1These are block diagrams of embodiments of the encoder system and the decoder system;

[0064] Figure 2 This is a flowchart of an embodiment of the encoding method and a first embodiment of the decoding method;

[0065] Figure 3 An example of cone culling is shown;

[0066] Figure 4 Examples of frustum culling and occlusion culling are shown;

[0067] Figure 5 An example of a stereo frustum is shown;

[0068] Figure 6 An example of point cloud and three viewpoints is shown;

[0069] Figure 7 It shows the relationship with Figure 6 One of the viewpoints corresponding to the view;

[0070] Figure 8 An example of a user viewpoint is shown;

[0071] Figure 9 This is a flowchart of the second embodiment of the decoding method;

[0072] Figure 10 This is a flowchart of the third embodiment of the decoding method; and

[0073] Figure 11 This is a block diagram of an exemplary data processing system for performing the methods of the present invention.

[0074] Corresponding elements in the accompanying drawings are indicated by the same reference numerals. Detailed Implementation

[0075] Figure 1 An embodiment of an encoder system (computer 21) and an embodiment of a decoder system (mobile device 1) are illustrated. Computer 21 includes a receiver 23, a transmitter 24, a processor 25, and a storage device 27. Processor 25 is configured to acquire an input set of volumetric image data, such as including one or more point clouds, and to select data from the volumetric image data for each of a plurality of views, such that the plurality of views (e.g., each of these views) includes only a portion of the volumetric image data, the selection being based on the visibility of the data from the respective viewpoint in the respective viewing direction and / or within the respective field of view. The term viewpoint refers to a location. There may be multiple views from a given viewpoint, for example, depending on the viewing direction.

[0076] Processor 25 is further configured to encode each of these views into a separate volumetric image data output set and generate metadata indicating multiple viewpoints. The metadata is associated with multiple viewpoints and may include 3D positional information. The metadata may describe other characteristics of each view, such as viewing direction and / or field of view. Multiple output sets may be associated with the same metadata / viewpoint, for example, if multiple output sets represent multiple qualities of the same content. Information specifying which viewpoints and / or viewing directions and / or fields of view to use can be obtained from the input data, for example, from the same input data that includes the input sets. For example, the viewing direction and / or field of view may alternatively be a default viewing direction and / or default field of view. Figure 1 In the example, computer 21 provides output sets and metadata to server computer 13.

[0077] The default viewing direction and / or field of view are those known at runtime and not provided in the signaling information. Different use cases may use different default values. Default values ​​can dynamically depend on another default value (however, for any set of static default parameters, typically only one set of (static + dynamic) default parameters exists). For example, the encoder and decoder systems can obtain default values ​​by compiling them into the software, by specifying rules for how to calculate / determine the default values ​​(e.g., in a standard), or by having another component determine these values ​​and provide them as input to the encoder and decoder systems.

[0078] Mobile device 1 includes transceiver 3, transmitter 4, processor 5, memory 7, camera 8, and display 9. Processor 5 is configured to, for example, use camera 8 to determine a desired user viewpoint and obtain metadata associated with encoded volumetric image data from server computer 13 via medium 11 (e.g., a computer network, such as the Internet). The metadata indicates available viewpoints, each corresponding to an available view. Processor 5 is further configured to: select one or more available viewpoints based on the desired user viewpoint; obtain one or more volumetric image datasets from server computer 13 based on the selected one or more viewpoints, in which one or more available views corresponding to the selected one or more available viewpoints have been encoded; and decode at least one available view from the one or more volumetric image datasets.

[0079] Mobile device 1 can select available viewpoints in such a way that the corresponding view is the most similar to the desired user view (corresponding to the desired user viewpoint). Alternatively, mobile device 1 can first select one or more views in this manner and then further select the view closest to the object to achieve the highest quality. Rendering can occur in parallel with decoding (e.g., a partially decoded point cloud may have already been rendered). In the case of multiple views, each view can be rendered separately.

[0080] exist Figure 1 In one embodiment, the decoder system is a mobile device and renders at least one of one or more decoded available views. In an alternative embodiment, the decoder system is a different device (e.g., a PC), is part of a device (e.g., a mobile device), or comprises multiple devices. Figure 1 In the example, mobile device 1 obtains metadata and volumetric image data from server computer 13. Alternatively, mobile device 1 can obtain metadata and volumetric image data from storage 7.

[0081] exist Figure 1 In the embodiment of the mobile device 1 shown, the mobile device 1 includes a processor 5. In alternative embodiments, the mobile device 1 includes multiple processors. The processor 5 of the mobile device 1 may be a general-purpose processor (e.g., from ARM or Qualcomm) or a dedicated processor. The processor 5 of the mobile device 1 may run an operating system such as Android or iOS. For example, the display 9 may include an LCD or OLED display panel. For example, the display 9 may be a touchscreen. For example, the processor 5 may use the touchscreen to provide a user interface. The memory 7 may include one or more memory cells. The memory 7 may include, for example, solid-state memory. For example, the camera 8 may include a CCD or CMOS sensor.

[0082] For example, receiver 3 and transmitter 4 can use one or more wireless communication technologies, such as Wi-Fi (IEEE 802.11), to communicate with other devices. In alternative embodiments, multiple receivers and / or multiple transmitters are used instead of a single receiver and a single transmitter. Figure 1 In the illustrated embodiment, a separate receiver and a separate transmitter are used. In an alternative embodiment, receiver 3 and transmitter 4 are combined into a transceiver. The mobile device 1 may include other components typical of mobile devices, such as a battery and power connector. The invention can be implemented using a computer program running on one or more processors.

[0083] exist Figure 1In the embodiment of computer 21 shown, computer 21 includes a processor 25. In alternative embodiments, computer 21 includes multiple processors. The processor 25 of computer 21 may be a general-purpose processor (e.g., from Intel or AMD) or a dedicated processor. For example, the processor 25 of computer 21 may run a Windows- or Unix-based operating system. Storage device 27 may include one or more memory units. For example, storage device 27 may include one or more hard disks and / or solid-state drives. For example, storage device 27 may be used to store operating systems, applications, and application data.

[0084] For example, receiver 23 and transmitter 24 can communicate with other devices using one or more wired and / or wireless communication technologies such as Ethernet and / or Wi-Fi (IEEE 802.11). In alternative embodiments, multiple receivers and / or multiple transmitters are used instead of a single receiver and a single transmitter. Figure 1 In the illustrated embodiment, a separate receiver and a separate transmitter are used. In an alternative embodiment, receiver 23 and transmitter 24 are combined into a transceiver. Computer 21 may include other components typical of a computer, such as a power connector. The invention can be implemented using a computer program running on one or more processors.

[0085] Figure 2 An embodiment of a method for encoding volumetric image data and a first embodiment of a method for decoding encoded volumetric image data are illustrated. Step 101 includes obtaining an input set of volumetric image data. For example, the input set of volumetric image data may include one or more point clouds or may include 3D meshes, RGB+D image data, or voxels. A point cloud is a set of 3D points, where each point may or may not have associated attributes (e.g., color, brightness). The points of a point cloud may have associated 3D normal vectors.

[0086] For example, point clouds can be provided as '.ply' files. Known techniques can be used to parse this file and store it in RAM. For instance, the stored point cloud can be copied from RAM to GPU memory as part of a GPU algorithm. Point clouds can be generated from one or more RGB+D inputs (e.g., captured by an RGB+D sensor). If there are more than one single RGB+D input, points can be fused to improve the smoothness of the point cloud.

[0087] If volumetric image data comprises multiple point clouds, these point clouds form a single scene, meaning they are spatially related. For example, one point cloud might represent a table in a room, and another point cloud might represent a chair in the same room. If volumetric image data comprises multiple point clouds, culling can be performed only on a subset of these multiple point clouds.

[0088] The input set of volumetric image data can vary over time, and at least one processor can be configured to repeatedly select data from the volumetric image data for each of these views and encode each of the views into a separate bitstream.

[0089] Step 103 includes selecting data from the volumetric image data for each of a plurality of views on the volumetric image data, such that the plurality of views (e.g., each of these views) include only a portion of the volumetric image data, the selection being based on the visibility of the data from the respective viewpoint in the respective viewing direction and / or within the respective field of view. Figure 2 In one embodiment, step 103 includes sub-steps 111 and 113.

[0090] Step 111 involves defining multiple viewpoints for the input set and performing initial data selection for the views corresponding to these viewpoints. Viewpoints can be selected such that the corresponding views collectively include all volumetric image data. This allows the user to view all volumetric image data by changing their viewpoint. Alternatively, some data (e.g., points) may not be included in any view, for example, because it is hidden from every viewpoint or simply to reduce coding effort / time.

[0091] It can accept a set of viewpoints as input. For example, the following JSON schema specifies the data structure that can be used to specify these viewpoints:

[0092]

[0093] In this example, the location is specified in the same units as the input point cloud. The orientation is specified as the Tait-Bryan angle in radians. The projection type is selected from a list (e.g., orthogonal, perspective, cylindrical, spherical, or a projection type disclosed in WO 2018 / 215502A1). The camera field of view is specified in degrees. The near and far clipping planes are specified in the same units and coordinate space as the input point cloud. The oversampling rate is specified as the number of subsamples per pixel.

[0094] You can initially select all volumetric image data from the volumetric image data that are visible from the corresponding viewpoint in the corresponding viewing direction and / or within the corresponding field of view for each view.

[0095] Alternatively, some data that is visible from the corresponding viewpoint in the corresponding viewing direction can be omitted, for example, if the data does not have a significant impact on rendering or is simply to reduce coding effort / time. For instance, in some viewing directions, two non-overlapping points in the point cloud may significantly overlap during rendering (and one of them may be almost invisible), and therefore it may not be necessary to render both points simultaneously.

[0096] In the first implementation of this step, the viewpoint and view are determined as follows. First, a virtual sphere is constructed at point c (e.g., point c could be the centroid of all points) within a point cloud space of radius r. Based on the desired coverage level, a position on this sphere is selected as the viewpoint. The view is rotated such that the viewpoint looks towards point c (e.g., the center of the virtual sphere). To increase coverage, the process can be repeated for different values ​​of {r, c}.

[0097] "Coverage level" can be defined as an abstract metric that includes how many points, for example:

[0098] 1. The ratio of included points to excluded points (for example, a minimum coverage of 80% for all points can be defined).

[0099] 2. Include the absolute number of points (for example, a minimum coverage level of 100,000 points can be defined).

[0100] 3. The above combinations, such as a minimum of 40% or 100,000 points.

[0101] Definition 2) is particularly useful when aiming to achieve a specific target bandwidth because the desired level of coverage can be set to that value (the number of points is related to bandwidth usage). However, Definition 2) does not take into account the accuracy of this representation (e.g., the boundaries do not account for the relative loss of information).

[0102] In a first variant of this first implementation (i.e., the second implementation), c remains constant, while r gradually decreases by a constant amount e in multiple constant step sizes n.

[0103] In a second variant (i.e., a third embodiment) of the first implementation, when generating a suitable set of viewpoints and views, a large set of views V is first generated to achieve a minimum coverage level c. Next, views that achieve a specific desired coverage level d (here, c = d) can be determined. A subset of . The result may be more efficient because |V′|≤|V|.

[0104] The advantage of this second variant is that in some scenarios, it is less costly to generate a large set of views first, which provides higher accuracy than needed / desired (e.g., due to bandwidth limitations). In this second variant, viewpoints are then discarded based on their contribution to the coverage level (e.g., if two views overlap significantly (covering similar points), removing one or the other will not significantly reduce the coverage level, but will preserve the encoding / emission of the entire view).

[0105] In the fourth embodiment of this step, camera attributes are specified on a viewpoint-by-view basis.

[0106] Optionally, step 111 includes selecting additional data for multiple views, the selection being based on the visibility of the additional data from one or more corresponding additional viewpoints associated with the respective viewpoint. For example, the data and additional data may correspond to adjacent or partially overlapping views.

[0107] Since the desired user viewpoint is often different from one of the available viewpoints, decoder systems typically require some adjustments to align the decoded view to the desired user viewpoint. This adjustment can be determined from the obtained dataset by providing additional data—not just data visible from the corresponding viewpoint in the corresponding viewing direction and / or field of view within the view—that is, data visible from the desired user viewpoint but not from the available viewpoints.

[0108] Additional data for a viewpoint can be selected either by virtually moving the viewpoint or by selecting data near the data already selected for the view. The former leads to optimal results (e.g., no data is missed), but the latter can be performed faster. In an alternative embodiment, this additional data is provided as a separate dataset from the view (but associated with the view).

[0109] Step 113 includes culling the initial data selection for the view. For example, the shape of the view can be like a truncated pyramid, a prism, or a cone. Figure 3 An example of frustum culling is shown. In this example, the view from viewpoint 41, which has frustum 43, is culled. Figure 3 The diagram shows five points in the point cloud: points 51-54 and 56. Of these five points, only points 53 and 54 are located inside the view frustum 43. Point 51 is located to the left of the view frustum 43, point 52 is located to the right of the view frustum 43, and point 56 is located behind the view frustum 43. Points 51, 52, and 56 are therefore discarded, i.e., removed from the initial selection.

[0110] Figure 4 Examples of frustum culling and occlusion culling are shown. Figure 4 In the example, an additional occlusion point 55 is shown. This point 55 is not visible from viewpoint 41 and is therefore culled, i.e., removed from the initial selection. As previously described, occlusion culling and frustum culling are known techniques.

[0111] A single wide view (such as a truncated cone shape) can be used to render views for both eyes simultaneously. This is in Figure 5 As shown in [the image]. Figure 5 In the example, it represents the relationship with Figure 3 and Figure 4 Together with the visual cone 43, the second visual cone 45 forms a stereoscopic visual cone.

[0112] For example, rasterization or ray tracing can be used to perform culling. In a first implementation of this step, the point cloud is rendered using the setpoint size of occlusion detection and then the culled point cloud is reconstructed based on the color and depth buffers.

[0113] Alternatively, the point cloud can be rendered point-by-point, and a list of points to include or to exclude can be maintained. In the second embodiment of this step, after initializing the list of points to include, rasterization is used to render the point cloud. If a point is determined to be invisible during rendering, it is included in the exclusion list. If a point is determined to be visible, it is not included in the exclusion list. The culled point cloud consists of all points from the original point cloud, except those in the exclusion list.

[0114] In the third implementation of this step, after initializing the list of points to be included, ray tracing is used to render the point cloud point by point. If a point is determined to be visible during ray tracing, it is included in the output list. If a point is determined to be invisible, it is not included in the output list. The culled point cloud consists of all points in the output list. The advantage of ray tracing is that occlusion can be obtained 'freely' as part of the visibility detection in the ray tracing algorithm, and ray tracing can be relatively fast due to recent hardware acceleration technologies for ray tracing in GPUs (such as RTX in Nvidia GPUs).

[0115] Figure 6 An example of point cloud 71 and three viewpoints 73-75 is shown. In this example, point cloud 71 represents a person. Figure 6 The field of view and viewing direction of viewpoint 73 are shown. See also Figure 7 View 83 includes data visible from viewpoint 73 in the field of view: a (strict) subset 81 of point cloud 71.

[0116] Step 105 involves encoding each of these views into a separate volumetric image data output set. For example, the resulting output set 31 can be provided to a server computer. Encoding can simply involve creating a file for each output set and including selected data in the file, or it can involve compressing the selected data. Point cloud compression techniques are described in “Emerging MPEG Standards for Point Cloud Compression” by Sebastian Schwarz et al., published in the IEEE Journal on Emerging and Selected Topics in Circuits and Systems, Volume 9, Issue 1, March 2019. An example of this technique is MPEG video-based point cloud compression (V-PCC) for dynamic content. Multiple point clouds can be encoded in parallel.

[0117] In addition to multiple views, the entire input set of volumetric image data can also be encoded into a separate output set of volumetric image data. This can be used, for example, to provide a low-quality version of the entire input set of volumetric image data at any given time, thus avoiding the "blank" effect of not being able to display data corresponding to a portion of the scene. If the client has sufficient bandwidth and computing resources after downloading the low-quality version of the volumetric image, the client will be able to choose to obtain high-quality, usable volumetric image data instead of the low-quality, usable volumetric image data. In this way, volumetric images can always be rendered, even under conditions of rapid user movement and high network latency.

[0118] Step 107 includes generating metadata 33. The metadata indicates multiple viewpoints. Figure 2 In one embodiment, step 107 is performed after step 111. In an alternative embodiment, step 107 is performed in parallel with step 111. Metadata can specify the location of the viewpoint, but it can also specify where the output set of volumetric image data or a portion of the output set of volumetric image data is obtained. This gives the decoder system complete control over which output sets / views are obtained. For example, metadata can be specified in a manifest file, such as a Media Presentation Description (MPD) of MPEG-DASH (ISO / IEC 23009-1). In another embodiment, metadata can be specified in the ISO Basic Media File Format (ISOBMFF) data format, in which each output set of volumetric image data can be encapsulated into different tracks within the same ISOBMFF (ISO / IEC 14496-12) file.

[0119] The following provides an example of an MPD manifest:

[0120]

[0121] In this example, different perspectives of the same point cloud are encoded into separate adapter sets (the third adapter set is the synchronized audio track). The client can parse this MPD and select the desired adapter set for streaming. Based on the URL in the representation within the adapter set, the client is able to download the selected point cloud via HTTP for rendering. Multiple periods can be used to allow parameters to be changed within a given time interval.

[0122] In this example, the 'CameraLocation' tag defines the position of the camera corresponding to the viewpoint, while the 'CameraOrientation' tag defines the camera's rotation. Since views are encoded and stored independently, they may need to be blended before rendering. For this purpose, the 'Offset' tag is included, which specifies the transformation matrix to be applied to that particular view before rendering.

[0123] Step 121 includes obtaining metadata 33 associated with the encoded volumetric image data. Figure 2 In one embodiment, metadata is associated with a content item (e.g., a football match X against Y), and step 121 includes selecting the content item before obtaining the associated metadata, for example, in response to user input. In an alternative embodiment, metadata may be received before selecting a content item, or selection of a content item may not be necessary.

[0124] As described with respect to step 107, metadata 33 indicates available viewpoints, and each of the available viewpoints corresponds to an available view. The metadata may further indicate the viewing direction and / or field of view and / or additional viewpoint configurations for each of the available viewpoints. Examples of additional viewpoint configurations are camera projection type, camera orientation, near / far clipping planes, zoom level, lens shape, speed, acceleration, anti-aliasing level, anti-aliasing type, anisotropic filtering level, gamma correction, contrast, and brightness.

[0125] Step 123 involves determining the desired user viewpoint. The user can request a view from a new viewpoint by using a controller (e.g., in the case of virtual reality) or by moving his / her head or mobile device, or by moving around (e.g., in the case of augmented reality). This desired viewpoint does not necessarily match one of the available viewpoints.

[0126] In an alternative embodiment, steps 121 and 123 are performed in a different order. Figure 2In one embodiment, a content item is selected before the desired user viewpoint is determined. In an alternative embodiment, a content item can be selected after the desired user viewpoint has been determined. For example, if multiple content items with similar content exist, a content item can be selected based on the desired user viewpoint. In this alternative embodiment, step 123 is typically performed before step 121.

[0127] Step 125 includes selecting one or more of the available viewpoints based on the desired user viewpoint. Figure 8 An example of the desired user viewpoint is shown: User Viewpoint 91. Figure 8 Three available viewpoints are further shown 73-75.

[0128] In a first embodiment of this step, step 125 includes determining the view corresponding to the user viewpoint 91 and the views corresponding to the available viewpoints 73-75, comparing the overlap between the user view and each of the available views, and selecting the available viewpoint corresponding to the available view with the greatest overlap.

[0129] In the second embodiment of this step, step 125 includes determining the distance between the user's viewpoint 91 and each of the available viewpoints 73-75, and selecting the nearest available viewpoint. Therefore, the viewing direction is ignored. This second embodiment is advantageous when it is known that all viewpoints point to the point cloud. For both embodiments, [further details will be provided]. Figure 8 Select available viewpoint 73 from the examples.

[0130] As an extension of these two implementation methods, a priority list of viewpoints can be created, ordered according to their desirability. The view corresponding to the best matching viewpoint in the list will then be obtained first, but if time permits, more available views can be obtained based on the priority list of viewpoints (e.g., second best, third best, etc.).

[0131] Step 127 includes obtaining one or more sets of volumetric image data 31 based on one or more selected viewpoints, in which one or more available views corresponding to the selected one or more available viewpoints have been encoded.

[0132] In the first implementation of this step, the volumetric image dataset is a file published by a server computer using MPEG DASH. Each file may include a subset of point clouds and the files may be streamed to multiple client / decoder systems.

[0133] In the second implementation of this step, the server computer streams volumetric image data, such as files including views, via a media streaming protocol (e.g., RTP). As is common in such streaming scenarios, SIP can be used to negotiate the transmission of the stream before streaming begins. To implement this, it may be necessary to define new SDP message portions that can be used to signal different viewpoints. These new SDP messages have been transmitted and received in steps 107 and 121, respectively. For example, the following message portions can be defined for streaming views on a point cloud, which can be included in each viewpoint:

[0134] …

[0135] m = pointcloud <port>RTP / AVP 99

[0136] a=rtpmap:<ply RTP payload type>VPCC / <clock rate>

[0137] a=viewpoint:<viewpoint id> <x> <y> <z> <yaw> <pitch> <roll>

[0138] …

[0139] In this example, 'viewpoint id' is a session-unique integer that identifies the viewpoint, 'x', 'y', and 'z' are floating-point numbers representing the viewpoint's position, and 'yaw', 'pitch', and 'roll' are floating-point numbers representing the corresponding rotation of the associated camera.

[0140] The following provides an example of a complete SDP message. SDP messages are used to negotiate view streaming on a point cloud. In this example, it is assumed that the RTP payload type of the point cloud is '2019':

[0141] Alice proposes sending a point cloud with three viewpoints:

[0142] v = 0

[0143] o=alice 28908445262890844526 IN IP4 host.atlanta.example.com

[0144] s =

[0145] c=IN IP4host.atlanta.example.com

[0146] t=0 0

[0147] m=audio 49170 RTP / AVP 0 8 97

[0148] a = rtpmap: 0 PCMU / 8000

[0149] a = rtpmap: 8 PCMA / 8000

[0150] a = rtpmap: 97 iLBC / 8000

[0151] m=pointcloud 51372 RTP / AVP 31 32

[0152] a = sendonly

[0153] a = rtpmap:2019 VPCC / 90000

[0154] a=viewpoint: 0 0.1 0 5 30 0 0

[0155] a=viewpoint: 1 0 3 2 340 210 30

[0156] a=viewpoint: 2 3 0 2 140 10 240

[0157] Bob answered the request to receive the viewpoint with id 1.

[0158] v = 0

[0159] o=bob 2808844564 2808844564 IN IP4 host.biloxi.example.com

[0160] s =

[0161] c=IN IP4 host.biloxi.example.com

[0162] t=0 0

[0163] m = audio 49172 RTP / AVP 99

[0164] a = rtpmap: 99iLBC / 8000

[0165] m=pointcloud 51374 RTP / AVP 99

[0166] a = rtpmap:2019 VPCC / 90000

[0167] a=viewpoint: 1 0 3 2 340 210 30

[0168] According to the SIP protocol, the negotiation is now complete, and Alice can begin transmitting the corresponding view to Bob. Once Bob wants to receive views corresponding to different viewpoints, he can renegotiate using SDP.

[0169] In the third implementation of this step, streaming is implemented using WebRTC with the same type of signaling as in the second implementation, but then JavaScript Session Establishment Protocol (JSEP) is used.

[0170] In the fourth embodiment of this step, the view is included in an ISOBMFF container. For example, the ISOBMFF container may include one or more point clouds.

[0171] The goal is to obtain at least one view in step 127, but multiple views can be obtained if sufficient time allows. During streaming, the client can use a deadline to determine if there is enough time to acquire more data. These views can be used to provide a better approximation of the original image data. An extended version of the selection algorithm can be used to determine exactly which additional views to acquire (e.g., when viewpoints are sorted by similarity, the client can acquire a second similar viewpoint, a third similar viewpoint, etc.), as described with respect to step 125.

[0172] Acquiring a random additional viewpoint can also aid subsequent processing, such as when the client needs to display the next frame and no new frame data is available. When the current point cloud is one of a set of multiple consecutive point clouds (e.g., animated point clouds) and the client misses the deadline for the next point cloud, an additional view can be used to display the current point cloud from a different angle than the initially acquired view. The current point cloud and the next point cloud can also be referred to as the current frame and the next frame of the point cloud.

[0173] Additional views can have different qualities than the main view. Client-based quality selection mechanisms, such as ABR or BOLA algorithms, can be used, similar to DASH. If the user has multiple viewpoints, such as one viewpoint per eye, multiple viewpoints can be streamed simultaneously (e.g., in a single stream / video). For some point clouds, this can potentially improve encoding efficiency. These streams can be spatially arranged as a sequence of one or more images (e.g., in a quilted pattern or side-by-side), making them directly usable by holographic and / or light field displays.

[0174] Step 129 involves decoding at least one available view from one or more available views in one or more volumetric image datasets. The client typically uses a state-of-the-art decoder to decode the obtained views(s) according to their format. For example, in embodiments that encode point clouds using V-PCC, a state-of-the-art V-PCC decoder is typically used to decode the views(s). Multiple views / point clouds can be decoded in parallel.

[0175] Optional step 131 includes rendering at least one of the decoded available views. Known techniques can be used to render the views(s), such as point cloud rendering techniques. Because frustum culling has already been performed, the data size is significantly reduced in most cases. Therefore, even with conventional state-of-the-art rendering, the total workload required when using these methods is reduced. However, if the obtained available views are larger than the desired user view, additional frustum culling can be performed in step 131.

[0176] Since the desired user viewpoint is often different from one of the available viewpoints, the decoder system typically requires some adjustments to align the decoded view to the desired user viewpoint. Therefore, performing occlusion culling in step 131 may be beneficial. Figure 2 In one embodiment, occlusion culling is also performed by the encoder system in step 113 (a sub-step of step 103). In an alternative embodiment, the encoder system does not perform occlusion culling in step 103.

[0177] If occlusion culling is not performed, the decoder system uses common overdraw algorithms to ensure that occluded objects (e.g., points) are not rendered in front of visible objects. For example, this can be done by acquiring all objects and calculating their relative distance to the camera for each object. The resulting list of distances is then sorted, and objects are drawn in order from farthest to nearest. Steps 121 or 123 are repeated after step 131, and so on. Figure 2 The method continues as shown. In the next iteration of step 123, another desired user viewpoint is determined.

[0178] about Figure 2 The described implementations involve, but are not limited to, the use of point clouds. The same or similar implementations may involve the use of different types of volumetric data.

[0179] Several options exist to ensure sufficient data is obtained to fill the desired user field of view at the desired user viewpoint, which is offset from the available viewpoint, including:

[0180] • The field of view of the available view is set to be larger than the user's field of view. This has already been described with respect to step 111 and will also be discussed regarding... Figure 10 Describe it.

[0181] • It can obtain one or more related views and merge them with the main view. This will be about Figure 9 Describe it.

[0182] Figure 9 A second embodiment of a method for decoding encoded volumetric image data is shown. In this second embodiment, one or more related views are obtained and merged with the main view.

[0183] Step 121 includes obtaining metadata associated with the encoded volumetric image data. Step 123 includes determining the desired user viewpoint. Step 125 includes selecting available viewpoints based on the desired user viewpoint.

[0184] Step 127 is performed after step 125. Step 127 includes obtaining a volumetric image dataset based on a selected viewpoint, in which available views corresponding to the selected available viewpoint have been encoded. Step 129 includes decoding the available views from the volumetric image dataset.

[0185] In addition to step 127, step 141 is performed after step 125. Step 141 includes obtaining an additional volumetric image dataset in which data from one or more related views has been encoded. These one or more related views are associated with available views. For example, the one or more related views can be adjacent, separate, or overlapping.

[0186] One or more related views can be considered related to one or more available views in the following circumstances:

[0187] Content creators define it this way (e.g., by creating a group of views). This is helpful in scenarios where the set of user locations is limited. Such scenarios are typically defined as scenes or location maps within one or more scenes.

[0188] They share some common properties (e.g., the difference in viewing direction is less than a constant c, or their relative distance is less than a constant d).

[0189] • Automatically generated views can be grouped by the encoder system. For example, if the encoder system starts with a set of 'anchor' views but needs more views to cover more scenes, the encoder system can define additional views associated with the known views.

[0190] The relevant view may partially overlap with the available view, or one of the two views may be completely contained within the other. Step 143 is performed after step 141. Step 143 includes decoding at least one relevant view from one or more relevant views from another volumetric image dataset.

[0191] Step 145 is performed after steps 129 and 143. Step 145 includes merging at least one decoded relevant view with a decoded available view. This can be performed after step 145. Figure 2 Step 131 Figure 9 (Not shown in the image). In this case, the blended view is rendered in step 131. Frustum culling and / or occlusion culling may be performed on the blended view.

[0192] Figure 10 A third embodiment of a method for decoding encoded volumetric image data is shown. Figure 9 In this embodiment, multiple views are always (or alternatively, most of the time) merged. Figure 10 In some embodiments, this is generally unnecessary because the field of view of the available view is set to be larger than the user's field of view to allow the obtained view to be adjusted to the desired user viewpoint. However, when the user changes their desired viewpoint, for example by moving their head, by moving around, or by using a controller, another (e.g., closer) viewpoint may become preferred over the current viewpoint.

[0193] In this scenario, the desired transition is to another available view. To achieve this transition, it may be beneficial to render both views together during the transition. These two views are typically rendered simultaneously using the same algorithm. When rendering multiple views from multiple viewpoints, point fusion (denoising, averaging, smoothing, deduplication, and / or removal) can be performed to ensure a smooth transition between multiple views.

[0194] Step 121 includes obtaining metadata associated with the encoded volumetric image data. Step 123 includes determining the desired user viewpoint. Step 125 includes selecting one or more available viewpoints based on the desired user viewpoint.

[0195] If the desired user viewpoint is the same as an available viewpoint, or if the available viewpoint corresponding to an available viewpoint includes all data visible from the desired user viewpoint, then only that viewpoint is selected. Therefore, all data visible from the desired user viewpoint is obtained without needing to obtain multiple views (as a whole). Step 171 is performed after step 125.

[0196] Step 171 includes checking whether view data for at least one of the selected viewpoints has not yet been obtained. If so, steps 127 and 129 are performed for one of the selected viewpoints. Step 127 includes obtaining a volumetric image dataset based on the selected viewpoint, in which available views corresponding to the selected available viewpoint have been encoded. Step 129 includes decoding the available view from the volumetric image dataset.

[0197] If it is determined in step 171 that views corresponding to all selected viewpoints have been obtained, then step 175 is executed. Step 173 is executed after steps 127 and 129. Step 173 includes checking whether there are still selected viewpoints for which view data has not yet been obtained. If yes, step 171 is repeated. If not, then step 175 is executed.

[0198] Step 175 includes checking whether multiple viewpoints have been selected and whether multiple views have been obtained and decoded. If not, proceed to step 177. Optional step 177 includes rendering a single decoded view. After step 177, repeat step 123 to obtain another desired user viewpoint, and then proceed as follows. Figure 10 The method continues as shown.

[0199] If it is determined in step 175 that multiple viewpoints have been selected and multiple views have been obtained and decoded, then step 181 is performed next. Step 181 includes merging the multiple decoded available views. Optional step 183 includes rendering the merged available views. After step 183, step 123 is repeated to obtain another desired user viewpoint, and then... Figure 10 The method continues as shown. In an alternative embodiment, even if multiple viewpoints have been selected and multiple views have been obtained, only a subset may be used, and if only one of the selected views is used, step 177 may be performed.

[0200] The fusion in step 181 typically involves smoothing the view using known techniques to hide the 'seams' between different views from different viewpoints (these seams may be introduced due to lossy compression artifacts and / or overlap between views). Not only can the desired user viewpoint change from one moment to another, but the available viewpoint can also change every frame or every frame sequence. In both cases, a new selection from the available viewpoint is required.

[0201] To prevent rendering invisible data or reduce the amount of rendered invisible data, occlusion culling and / or frustum culling can be performed in steps 177 and 183, which has the advantage of maintaining the existing rendering pipeline. Frustum culling may be faster than existing techniques because frustum culling is already performed in the encoder system. The same applies if occlusion culling is performed in the encoder system.

[0202] Figure 11 The description shows that the reference can be executed. Figure 2 , Figure 9 and Figure 10 A block diagram of an exemplary data processing system describing the method.

[0203] like Figure 11 As shown, the data processing system 400 may include at least one processor 402 coupled to a memory element 404 via a system bus 406. Thus, the data processing system can store program code within the memory element 404. Furthermore, the processor 402 can execute program code accessed from the memory element 404 via the system bus 406. In one aspect, the data processing system may be implemented as a computer suitable for storing and / or executing program code. However, it should be understood that the data processing system 400 may be implemented in the form of any system including a processor and memory capable of performing the functions described herein.

[0204] Memory element 404 may include one or more physical memory devices, such as, for example, local memory 408 and one or more mass storage devices 410. Local memory may refer to random access memory or other non-persistent memory(s) typically used during the actual execution of the program code. Mass storage devices may be implemented as hard disk drives or other persistent data storage devices. Processing system 400 may also include one or more cache memories (not shown) that provide temporary storage for at least some of the program code to reduce the number of times the program code must be retrieved from mass storage device 410 during execution.

[0205] The input / output (I / O) devices, depicted as input device 412 and output device 414, may optionally be coupled to the data processing system. Examples of input devices may include, but are not limited to, keyboards, pointing devices such as mice, 3DoF or 6DoF tracking controllers, etc. Examples of output devices may include, but are not limited to, monitors or displays (e.g., HMD or AR stereo displays), speakers, etc. The input and / or output devices may be coupled to the data processing system directly or through an intermediate I / O controller.

[0206] In embodiments, the input device and the output device can be implemented as a combined input / output device (in... Figure 11 (Seen in the middle by the dashed lines surrounding input device 412 and output device 414). An example of such a combined device is a touch-sensitive display, sometimes also called a "touchscreen display" or simply a "touchscreen". In such embodiments, input to the device can be provided by the movement of a physical object (such as, for example, a stylus or a user's finger) on or near the touchscreen display.

[0207] Network adapter 416 may also be coupled to the data processing system to enable it to couple to other systems, computer systems, remote network devices, and / or remote storage devices via an intermediate private or public network. The network adapter may include a data receiver for receiving data transmitted from the systems, devices, and / or networks to the data processing system 400, and a data transmitter for transmitting data from the data processing system 400 to the systems, devices, and / or networks. Modems, cable modems, and Ethernet cards are examples of different types of network adapters that can be used with the data processing system 400.

[0208] like Figure 11 As depicted, memory element 404 can store application program 418. In various embodiments, application program 418 may be stored in local memory 408, one or more mass storage devices 410, or separately from local memory and mass storage devices. It should be understood that data processing system 400 may further execute an operating system capable of facilitating the execution of application program 418. Figure 11 (Not shown in the image). The application program 418, implemented in the form of executable program code, can be executed by the data processing system 400 (e.g., by the processor 402). In response to executing the application program, the data processing system 400 can be configured to perform one or more operational or method steps described herein.

[0209] Various embodiments of the present invention can be implemented as a program product for use with a computer system, wherein the program of the program product defines the functionality of the embodiments (including the methods described herein). In one embodiment, the program(s) may be contained on a variety of non-transitory computer-readable storage media, wherein, as used herein, the expression "non-transitory computer-readable storage media" includes all computer-readable media, with the sole exception of temporarily propagated signals. In another embodiment, the program(s) may be contained on a variety of transient computer-readable storage media. Illustrative computer-readable storage media include, but are not limited to: (i) non-writable storage media on which information is permanently stored (e.g., read-only memory devices within a computer, such as CD-ROM discs readable by a CD-ROM drive, ROM chips, or any type of solid-state non-volatile semiconductor memory); and (ii) writable storage media on which variable information is stored (e.g., flash memory, floppy disk drives, or floppy disks within hard disk drives, or any type of solid-state random access semiconductor memory). The computer program may run on the processor 402 described herein.

[0210] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that when the terms "comprises" and / or "comprising" are used in this specification, they specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0211] All means or steps in the claims, plus corresponding structures, materials, actions, and equivalents of functional elements, are intended to include any structure, material, or action for performing a function in conjunction with other claimed elements as specifically claimed. Descriptions of embodiments of the invention have been presented for illustrative purposes, but such description is not intended to be exhaustive or to limit implementation to the forms disclosed. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the invention. The embodiments were chosen and described to best explain the principles of the invention and some practical applications, and to enable others skilled in the art, when suited to a particular intended use, to understand the invention with respect to various embodiments having various modifications.< / roll> < / pitch> < / yaw> < / z> < / y> < / x> < / port>

Claims

1. An encoder system (21) comprising at least one processor (25), the at least one processor being configured to: - Obtain the input set of volumetric image data (71), - For each of the plurality of views on the volumetric image data (71), data is selected from the volumetric image data (71) such that the plurality of views include only a portion of the volumetric image data (71), the selection being based on the visibility of the data from the respective viewpoint in the respective viewing direction and / or within the respective field of view, wherein the selection of data from the volumetric image data (71) for each of the plurality of views includes an initial data selection for frustum culling of each of the plurality of views. - Encode each of the removed views into a separate volumetric image data output set, and - Generate metadata that indicates multiple viewpoints.

2. The encoder system (21) as described in claim 1, wherein, The at least one processor (25) is configured to: - Select additional data for the plurality of views, the selection being based on the visibility of the additional data from one or more corresponding additional viewpoints associated with the respective viewpoint.

3. The encoder system (21) as described in claim 1 or 2, wherein, The at least one processor (25) is configured to specify in the metadata where the output set of the volumetric image data is obtained or a portion thereof.

4. The encoder system (21) as described in any one of claims 1-2, wherein, The metadata further indicates multiple viewing directions and / or multiple fields of view and / or additional viewpoint configurations.

5. The encoder system (21) as described in any of the preceding claims, wherein, The input set of the volumetric image data (71) includes one or more point clouds.

6. The encoder system (21) as described in any of the preceding claims, wherein, The at least one processor (25) is configured to select the data from the volumetric image data (71) for each of the plurality of views in such a way that, for each corresponding view, all the volumetric image data visible from the corresponding viewpoint in the corresponding viewing direction and / or in the corresponding field of view are selected from the volumetric image data.

7. The encoder system (21) as described in any of the preceding claims, wherein, The multiple views collectively include all of the volumetric image data (71).

8. A decoder system (1) comprising at least one processor (5), the at least one processor being configured to: - Determine the desired user viewpoint (91). - Obtain metadata associated with the encoded volumetric image data, the metadata indicating multiple available viewpoints (73-75), each of which corresponds to a view from which the available view frustum has been culled. - Select one or more of the plurality of available viewpoints (73-75) based on the desired user viewpoint (91). - Based on the selected one or more viewpoints, obtain one or more volumetric image datasets, in which one or more available culling views corresponding to the selected one or more viewpoints have been encoded, and - Decode at least one of the one or more available culling views from the one or more volumetric image datasets.

9. The decoder system (1) as described in claim 8, wherein, The at least one processor (5) is configured to: - Determine alternative desired user viewpoints. - Select another available viewpoint from the available viewpoints (73-75) based on the other desired user viewpoint. - Obtain an additional volumetric image dataset in which additional available views corresponding to the additional available viewpoints have been encoded. - Decode the additional available views from the additional volumetric image dataset, and - Merge the additional decoded available view with at least one decoded available culled view.

10. The decoder system (1) as claimed in claim 8, wherein, The at least one processor (5) is configured to: - Obtain an additional volumetric image dataset in which data from one or more related views have been encoded, said one or more related views being associated with the one or more available views. - Decode at least one of the one or more related views from the additional volumetric image dataset, and - Merge the at least one relevant decoded view with the at least one available culling view.

11. The decoder system (1) as claimed in any one of claims 8 to 10, wherein, The at least one processor (5) is configured to obtain metadata that indicates the available viewpoints (73-75) and specifies where a volumetric image dataset or a portion thereof is obtained, in which available views corresponding to the available viewpoints (73-75) have been encoded.

12. The decoder system (1) as claimed in claim 11, wherein, The metadata further indicates the viewing direction and / or field of view and / or additional view configuration for each of the available viewpoints (73-75).

13. A method for encoding volumetric image data, the method comprising: - Obtain the input set of (101) volumetric image data; - For each of the plurality of views on the volumetric image data, select (103) data from the volumetric image data such that the plurality of views include only a portion of the volumetric image data, the selection being based on the visibility of the data from the respective viewpoint in the respective viewing direction and / or in the respective field of view, wherein the selection of data from the volumetric image data (71) for each of the plurality of views includes an initial data selection for frustum culling of each of the plurality of views; - Encode (105) each of the culled views into a separate volumetric image data output set; and - Generate (107) metadata that indicates multiple viewpoints.

14. A method for decoding encoded volumetric image data, the method comprising: - Determine (121) the desired user viewpoint; - Obtain (123) metadata associated with the encoded volumetric image data, the metadata indicating a plurality of available viewpoints, each of which corresponds to a view from which an available frustum is culled; - Select one or more of the plurality of available viewpoints based on the desired user viewpoint; - Based on the selected one or more viewpoints, obtain (127) one or more volumetric image datasets, in which one or more volumetric image datasets have been encoded one or more available culled views corresponding to the selected one or more available viewpoints; as well as - Decode at least one of the one or more available culling views from the one or more volumetric image datasets (129).

15. A computer program product comprising or storing at least one software code portion, the software code portion being configured to perform the method as claimed in any one of claims 13 to 14 when running on a computer system.

Images