Depth image coding for meshes and point clouds

Abstract

This disclosure provides techniques for processing texture and depth images for volumetric video consumption for holographic communication. A sending device (40) pre-processes depth and texture images by dividing (270) a depth image into a foreground section and a background section and storing this information in an occupancy map. The depth image and occupancy map are then compressed (276) into a bitstream and sent to a receiving device (60, 240). The receiving device decompresses (356) the bitstream into decoded depth images and decoded occupancy maps, and post-processes the decoded information for use in generating volumetric video. The post-processing includes filtering (362) the depth image using the occupancy map to remove artifacts in the depth image introduced by compression at the sending device.

Description

[0001] DEPTH IMAGE CODING FOR MESHES AND POINT CLOUDS

[0002] TECHNICAL FIELD

[0003] The present disclosure relates generally to image processing techniques, and more particularly, to processing texture and depth images for use in generating volumetric video.

[0004] BACKGROUND

[0005] Versatile Video Coding (WC), and its predecessors High Efficiency Video Coding (HEVC) and Advanced Video Coding (AVC), are video codecs standardized and developed jointly by the International Telecommunication Union Telecommunication Standardization Sector (ITU-T) and the Moving Picture Experts Group (MPEG). Like many others, the WC, HEVC, and AVC codecs are block-based. The codecs utilize both temporal and spatial prediction, where spatial prediction is achieved using intra (I) prediction from within the current picture and temporal prediction is achieved using unidirectional (P) or bi-directional inter (B) prediction on the block level from previously decoded reference pictures.

[0006] In the encoder, the difference between the original pixel data and the predicted pixel data, referred to as the “residual,” is transformed into the frequency domain, quantized, and then entropy coded before being transmitted to a decoder together with necessary prediction parameters. Such parameters include, for example, prediction mode and motion vectors, which are also entropy coded before being transmitted to the decoder. Upon receipt, the decoder performs entropy decoding, inverse quantization, and inverse transformation to obtain the residual, and then adds the residual to an intra or inter prediction to reconstruct a picture. Transforming the residual data to the frequency domain before quantization improves the efficiency of the compression and better masks artifacts for the human eye when the quantization is high. However, due to their complexity, conventional encoding / decoding techniques do not always provide an acceptable Quality of Experience (QoE) within typical latency bounds.

[0007] SUMMARY

[0008] The embodiments described herein provide a system, devices, and corresponding methods for processing texture and depth images for volumetric video consumption. Volumetric video is a 3D representation of an object, such as a person (e.g., a person’s face, head, and / or upper torso), and can be described as a point cloud, a mesh, or other, similar format. Volumetric video can be generated for applications without real-time constraints and low-complexity coding demands. However, volumetric video can also be generated for other, more complex applications having more stringent requirements, such as conversational video applications.

[0009] In more detail, the present embodiments provide low-complexity processing techniques for real-time conversational video applications, such as holographic video communications, immersive video applications, 3D video calls, and the like. Example uses of such applications include, but are not limited to, those used to enable communications between a consultant and a therapist (e.g., in the e-health arena), families communicating with each other in an immersive manner, social media influencers communicating with his / her followers, and 3D calls with a remotely located subject matter expert.

[0010] In a first aspect, the present disclosure provides a method, implemented by a sending device, for conveying video data for a volumetric video representation to a receiving device. In this aspect, the method comprises the sending device partitioning a depth image into a foreground section and a background section, generating an occupancy map for the foreground and background sections, wherein the occupancy map is generated to comprise a first sample value corresponding to the foreground section and a second sample value corresponding to the background section, encoding the depth image and the occupancy map into a bitstream, and sending the bitstream over a communications network to the receiving device.

[0011] A second aspect of the present disclosure provides a method, implemented by a receiving device, for receiving video data for a volumetric video representation from a sending device. In this aspect, the method the receiving device receiving, from the sending device, an encoded bitstream comprising a depth image and occupancy map, wherein the occupancy map comprises a first sample value corresponding to a foreground section of the depth image and a second sample value corresponding to a background section of the depth image, decoding the depth image and the occupancy map from the encoded bitstream, and filtering the depth image based on the occupancy map.

[0012] A third aspect of the present disclosure provides a sending device for conveying video data for a volumetric video representation to a receiving device. The sending device is configured to partition a depth image into a foreground section and a background section, generate an occupancy map for the foreground and background sections, wherein the occupancy map is generated to comprise a first sample value corresponding to the foreground section and a second sample value corresponding to the background section, encode the depth image and the occupancy map into a bitstream, and send the bitstream over a communications network to the receiving device.

[0013] A fourth aspect of the present disclosure provides a sending device for conveying video data for a volumetric video representation to a receiving device. The sending device comprises communication circuitry for communicating with the receiving device and processing circuitry. The processing circuitry is configured to partition a depth image into a foreground section and a background section, generate an occupancy map for the foreground and background sections, wherein the occupancy map is generated to comprise a first sample value corresponding to the foreground section and a second sample value corresponding to the background section, encode the depth image and the occupancy map into a bitstream, and send the bitstream over a communications network to the receiving device.

[0014] A fifth aspect of the present disclosure provides a non-transitory computer-readable storage medium containing a computer program stored thereon. The computer program comprises executable instructions that, when executed by a processing circuit in a sending device causes the sending device to partition a depth image into a foreground section and a background section, generate an occupancy map for the foreground and background sections, wherein the occupancy map is generated to comprise a first sample value corresponding to the foreground section and a second sample value corresponding to the background section, encode the depth image and the occupancy map into a bitstream, and send the bitstream over a communications network to the receiving device.

[0015] A sixth aspect of the disclosure comprises a computer program for a sending device. The computer program comprises executable instructions that, when executed by processing circuitry in the sending device, causes the sending device to perform the method according to the first aspect.

[0016] A seventh aspect of the disclosure comprises a carrier containing a computer program according to the sixth aspect. The carrier is one of an electronic signal, optical signal, radio signal, or a non-transitory computer readable storage medium.

[0017] An eighth aspect of the present disclosure provides a receiving device for receiving video data for a volumetric video representation from a sending device. The receiving device is configured to receive, from the sending device, an encoded bitstream comprising a depth image and occupancy map, wherein the occupancy map comprises a first sample value corresponding to a foreground section of the depth image and a second sample value corresponding to a background section of the depth image, decode the depth image and the occupancy map from the encoded bitstream, and filter the depth image based on the occupancy map.

[0018] A ninth aspect of the present disclosure provides a receiving device for receiving video data for a volumetric video representation from a sending device. The receiving device in this aspect comprises communication circuitry for communicating with the sending device and processing circuitry. The processing circuitry is configured receive, from the sending device, an encoded bitstream comprising a depth image and occupancy map, wherein the occupancy map comprises a first sample value corresponding to a foreground section of the depth image and a second sample value corresponding to a background section of the depth image, decode the depth image and the occupancy map from the encoded bitstream, and filter the depth image based on the occupancy map.

[0019] A tenth aspect of the present disclosure provides a non-transitory computer-readable storage medium containing a computer program. The computer program comprises executable instructions that, when executed by a processing circuit in a receiving device causes the receiving device to receive, from the sending device, an encoded bitstream comprising a depth image and occupancy map, wherein the occupancy map comprises a first sample value corresponding to a foreground section of the depth image and a second sample value corresponding to a background section of the depth image, decode the depth image and the occupancy map from the encoded bitstream, and filter the depth image based on the occupancy map.

[0020] An eleventh aspect of the disclosure comprises a computer program for a receiving device. The computer program comprises executable instructions that, when executed by processing circuitry in the sending device, causes the receiving device to perform the method according to the second aspect.

[0021] A twelfth aspect of the disclosure comprises a carrier containing a computer program according to the sixth aspect. The carrier is one of an electronic signal, optical signal, radio signal, or a non-transitory computer readable storage medium.

[0022] A thirteenth aspect of the present disclosure provides a communication system for conveying video data for a volumetric video representation. In this aspect, the communication system comprises a sending device and a receiving device. The sending device is configured to partition a depth image into a foreground section and a background section, generate an occupancy map for the foreground and background sections, wherein the occupancy map is generated to comprise a first sample value corresponding to the foreground section and a second sample value corresponding to the background section, encode the depth image and the occupancy map into a bitstream, and send the bitstream over a communications network to the receiving device. The receiving device is configured to receive, from the sending device, an encoded bitstream comprising a depth image and occupancy map, wherein the occupancy map comprises a first sample value corresponding to a foreground section of the depth image and a second sample value corresponding to a background section of the depth image, decode the depth image and the occupancy map from the encoded bitstream, and filter the depth image based on the occupancy map.

[0023] BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 illustrates a mesh created using polygonal faces.

[0025] Figure 2 illustrates rendered mesh for a depth image compressed at a low or moderate bitrate.

[0026] Figure 3 is a functional block diagram illustrating a communication system configured to convey and receive texture and depth data for a volumetric video representation according to embodiments of the present disclosure.

[0027] Figures 4A-4B are flow charts illustrating a method for conveying and receiving texture and depth data for a volumetric video representation according to embodiments of the present disclosure.

[0028] Figures 5A-5B illustrate an example texture image and an example depth image, respectively, obtained according to embodiments of the present disclosure.

[0029] Figure 6 illustrates an example occupancy map generated according to embodiments of the present disclosure. Figures 7A-7B illustrate an example texture image and an example depth image, respectively, with background sections removed according to embodiments of the present disclosure.

[0030] Figure 8 illustrates a depth image in which the foreground section has been padded according to embodiments of the present disclosure.

[0031] Figures 9A-9C illustrate histograms of the depth values of a depth image according to embodiments of the present disclosure.

[0032] Figure 10A illustrates a mesh generated without using an occupancy map.

[0033] Figure 10B illustrates a mesh generated using an occupancy map and without padding the foreground of the depth image prior to encoding according to embodiments of the present disclosure.

[0034] Figure 10C illustrates a mesh generated using an occupancy map, and also, with padding the foreground of the depth image prior to encoding according to embodiments of the present disclosure.

[0035] Figures 11A-11 B are flow charts illustrating a method for conveying and receiving data for volumetric video according to embodiments of the present disclosure.

[0036] Figure 12 is a functional block diagram illustrating a communication system configured to convey and receive data for volumetric video according to embodiments of the present disclosure.

[0037] Figure 13 is a flow diagram illustrating an example method for conveying data for generating a volumetric video to a receiving device according to embodiments of the present disclosure.

[0038] Figures 14-18 are flow charts illustrating example methods for pre-processing video data at a sending device according to embodiments of the present disclosure.

[0039] Figure 19 is a flow diagram illustrating a method for receiving video data for volumetric video from a sending device according to embodiments of the present disclosure.

[0040] Figures 20-21 are flow diagrams illustrating example methods for post-processing received video data at a receiving device according to embodiments of the present disclosure.

[0041] Figure 22 is a functional block diagram illustrating some components of a sending device configured according to embodiments of the present disclosure.

[0042] Figure 23 is a functional block diagram illustrating some components of a receiving device configured according to embodiments of the present disclosure.

[0043] Figure 24 is a functional block diagram illustrating some components of a network-based receiving device configured according to embodiments of the present disclosure.

[0044] Figure 25 is a functional block diagram illustrating an example of a communication system in accordance with some embodiments.

[0045] Figure 26 is a functional block diagram illustrating an example User Equipment (UE) in accordance with some embodiments. Figure 27 is a functional block diagram illustrating an example network node in accordance with some embodiments.

[0046] Figure 28 is a functional block diagram of a host, which may be an embodiment of the host illustrated in Figure 25, in accordance with some embodiments described herein.

[0047] Figure 29 is a functional block diagram illustrating a virtualization environment in which functions implemented by some embodiments of the present disclosure may be virtualized.

[0048] Figure 30 is a functional block diagram illustrating a communication diagram of a host communicating via a network node with a UE over a partially wireless connection in accordance with some embodiments.

[0049] DETAILED DESCRIPTION

[0050] The embodiments described herein provide a method for processing texture and depth images for volumetric video consumption, for example, in the form of point clouds and / or meshes. The embodiments herein can be applied to real-time conversational video applications, such as for holographic calls, for example, where low-complexity processing of video data is desired.

[0051] As will be described more fully below, a sending device (e.g., an encoder) pre-processes depth and texture images by dividing a depth image into a foreground section and a background section and storing this information in an occupancy map. The texture image, depth image, and occupancy map are then compressed into one or more bitstreams and sent to a receiving device (e.g., a decoder) over a communication network. Upon receipt, the receiving device decompresses the one or more bitstreams into decoded texture images, decoded depth images, and decoded occupancy maps, and then post-processes the decoded information for use in generating volumetric video. The post-processing includes filtering the depth image using the occupancy map to remove artifacts in the depth image coming from compression.

[0052] Components and color formats

[0053] A video sequence consists of a series of pictures with each picture consisting of one or more components. As is known in the art, components are sometimes referred to as “color components,” and other times as “channels.” Additionally, a picture in a video sequence is sometimes denoted “image” or “frame.” Thus, in the context of this disclosure, and unless stated otherwise, the terms picture, frame, and image are used interchangeably. Similarly, the terms “encoding” and “compressing” are used interchangeably, as are the terms “decoding” and “decompressing,” and the terms “pixel” and “sample.”

[0054] Each component in a picture can be described as a two-dimensional rectangular array of sample values. Uncompressed pictures that come from a camera and uncompressed pictures that are rendered in a display are often in red, green, blue (RGB) format. This is because the camera sensors and the sub-pixels of the pixels in displays are historically compatible with RGB format. Video coding, on the other hand, is different. Particularly, the human eye is more sensitive to luminance than to chrominance. Therefore, a picture in a video sequence more commonly consists of three components; one luma component “Y,” where the sample values are luma values, and two chroma components “Cb” and “Cr,” where the sample values are chroma values. This is sometimes referred to as “YCbCr” color format or “YUV” color format where the chroma components are "U" and “V”. To optimize perceived quality, more bits can be spent on the luminance component than on the chrominance components. For the same reason, the dimensions of the chroma components are typically smaller than the luma components by a factor of two in each dimension. This is often referred to as the chroma components having been subsampled compared to the luma component. For example, the size of the luma component Y of an HD picture would be 1920x1080 and the chroma components Cb and Cr would each have the dimension of 960x540. A YUV format in which the chroma components have been subsampled in both the vertical and horizontal directions is often referred to as the “YUV420” format.

[0055] Blocks and Units

[0056] In many video coding standards, such as HEVC and WC, each component is split into blocks, with each block being a two-dimensional array of samples. A coded video bitstream, therefore, consists of a series of coded blocks. It is common in video coding that the picture is split into coding units that cover a specific area of the picture. Each coding unit consists of all blocks from all components that make up that specific area and each block belongs fully to one coding unit. In HEVC and WC the coding units are referred to as Coding Units (CUs). Other video codecs, however, refer to such coding units by another name. For example, the H.264 coding standard refers to such coding units as “macroblocks.” Residuals, Transforms and Quantization

[0057] A “residual block” consists of samples that represent sample value differences between the sample values of the original source blocks and the sample values of the prediction blocks. The residual block is typically processed using a spatial transform. Particularly, the encoder quantizes transform coefficients according to a quantization parameter (QP), which controls the precision of the quantized coefficients. The quantized coefficients can be referred to as “residual coefficients.” High QP values result in low precision of the coefficients, and therefore, a low fidelity / quality of the residual block. Upon receiving the residual coefficients, the decoder applies inverse quantization and inverse transform to derive the residual block.

[0058] Video Codec Profiles

[0059] A profile in HEVC and WC is defined as a specified subset of the syntax of the specification. More specifically, a profile defines the particular codec tools, bit depth, color format, and color subsampling that a decoder conforming to the profile should support, as well as what information is to be included in a bitstream that conforms to the profile. Practically, a codec specification may include a number of different profiles targeting different use cases, devices, and platforms, which can range from mainstream decoding in mobile devices to professional capturing and editing in high-end devices.

[0060] By way of example, HEVC version 1 comprises the Main and Main 10 profiles with support for 8- and 1O-bits bit depth with 4:2:0 chroma subsampling. Later versions of HEVC include additional profiles to support higher bitdepths, 4:4:4 chroma subsampling, and additional coding tools. Similarly, WC version 1 comprises the Main 10 profile, a Main 10 Still Picture profile, a Main 4:4:4 10 profile, a Main 4:4:4 10 Still Picture profile, a Multilayer Main 10 profile, and a Multilayer Main 10 4:4:4 profile, with support for higher bit depths in later versions.

[0061] It should be noted here that the terms “mainstream” video codecs and profiles are used throughout the disclosure. In the context of the present disclosure, these terms simply mean video codecs and profiles that are widely adopted in the market, or will be widely adopted in the market, and are supported (e.g., in hardware) by relevant devices, such as mobile devices, Head Mounted Devices (HMDs), computers, and TV sets, for example. According to this definition, such profiles may include, but are not limited to, the following video codec profiles:

[0062] • AVC constrained baseline profile;

[0063] • AVC baseline profile;

[0064] • AVC Extended profile;

[0065] • AVC Main profile;

[0066] • AVC High profile;

[0067] • HEVC Main profile;

[0068] • HEVC Main 10 profile;

[0069] • HEVC Main 12 profile; and

[0070] • WC Main 10 profile.

[0071] Depth image

[0072] A depth image is an image that contains information relating to the distance of the surfaces of the objects in a scene from a viewpoint of the camera (i.e., “depth”). Depth images are also sometimes referred to as “depth maps,” “depth pictures,” “z-buffer,” and “z-depth.” Each sample value of a depth image corresponds to the depth of that sample. Normally, only one component is used to represent the depth. A higher value (lighter luminance) in the depth image corresponds to a point that is close to the camera and a lower value (dark luminance) corresponds to a point that is distanced further away from the camera. In some applications, however, the opposite notation is used. That is, a low value in a depth image means that a given point in the image is close to the camera, while a high value in the depth image means that the point is distanced further away from the camera.

[0073] Depth images may be used for many different applications including, but not limited to, 3D object reconstruction and video conferencing (holographic communication). Depth images can correspond to certain parts of the head (e.g., a person’s face) and / or body (e.g., a person’s upper body / torso) that can represent a participant in a video conferencing session, and further, may be computer generated or captured by a depth image camera. Such depth image cameras may, for example, be a stand-alone camera or a camera that is configured to produce both texture and depth images as output, such as a Red Green Blue Depth (RGBD) camera, for example. Depth images can also be acquired, for example, by Lidar sensors (e.g., such as those used with Microsoft Kinect® and Intel Real-Sense®), true-depth sensors (e.g., iPhones®), or other devices.

[0074] Point clouds and V-PCC

[0075] A point cloud is a discrete set of data points in space, in which the points may represent a 3D shape or object. A point cloud is represented by vertices and a set of attributes (e.g., color).

[0076] MPEG has developed and standardized video-based point cloud compression (V-PCC) in MPEG-I part 5 (i.e., ISO / IEC 23090-5:2021), which is incorporated herein by reference in its entirety. In V-PCC, 3D patches of point clouds are projected onto 2D patches, which are then packed into frames for three types of images: texture images, geometry maps, and occupancy maps. The geometry map stores the distance between the missing coordinates of points of a 3D position and the projection surface in the 3D bounding box used in the 3D to 2D projection. The occupancy map is used to specify which samples in the geometry map are used for the 3D reconstruction and which are not.

[0077] The texture images, geometry maps, and occupancy maps are coded as sub-bitstreams using HEVC. These sub-bitstreams are then multiplexed into a V-PCC bitstream together with an atlas metadata sub-bitstream that comprises information detailing how to reconstruct the point cloud from the 2D patches.

[0078] Meshes

[0079] A 3D object or person may also be represented using polygon meshes. Such meshes describe a 3D object or person in terms of connected polygons, which are typically triangles. Objects created with polygon meshes store different types of elements, including vertices, edges, faces, and surfaces. A vertex is a position in space and includes other information such as color and normal vector. An edge is a connection between two vertices, and a face is a closed set of vertices (e.g., a triangle face has three vertices). Surfaces are not required for meshes but may be used to group smooth regions of the mesh. An example of a mesh 10 created using triangular faces is seen in Figure 1 .

[0080] Meshes may, for example, be generated from point clouds, depth images, and / or texture images. However, when generating a mesh from camera captured content, knowledge of intrinsic and extrinsic camera parameters used in capturing an image may be necessary to properly place the mesh in 3D space. Intrinsic camera parameters specify the camera image format including focal length, image resolution, and camera principal point, while extrinsic camera parameters define the camera pose with position and orientation.

[0081] Holographic communication

[0082] Work is currently being performed on holographic communications solutions to enable 3D calling between two or more participants. Current solutions define an end-to-end pipeline, which includes capturing RGB and depth images (e.g., using mobile devices or standalone cameras), converting the resulting stream into a point cloud or mesh, and then transmitting the point cloud or mesh to one or more end-user device, such as AR glasses, for example. In some cases, depth compression is realized by the device that captures the images. The compressed RGB and depth streams are then transmitted to a server (e.g., cloud or edge server) where the point cloud or mesh is created (e.g., the server creates a hologram that can be consumed by an end-user device). In some cases, the depth images are compressed at a low bitrate, which can be essential in reducing uplink bitrate requirements.

[0083] However, depth image compression at low or moderate bitrates, and then using that depth image to render a mesh, can be problematic. For example, as seen in mesh 20 of Figure 2, the quality of the rendered mesh 20 is suboptimal, particularly at the internal edges of the depth image. Specifically, undesirable artifacts might be more visible in 3D representations of a sensitive part of a person, such as his / her face. Additionally, pre-processing according to V- PCC is very complex and demanding on system resources. This includes the pre-processing associated with creating and packing patches. Heavy pre-processing usually leads to higher latency, which is undesirable for use cases involving conversational video use. Additionally, V- PCC is used for point clouds, which are typically very large and data heavy.

[0084] Although V-PCC uses HEVC internally, a V-PCC codec is still required at both the sender device and the receiver device, to run V-PCC. However, the current market penetration for the V-PCC codec is questionable, particularly in the realm of hardware support. These, and other issues, make V-PCC and other conventional standards less than optimal for use with at least some services. For example, where real-time conversational services are concerned, a low complexity solution is desired to provide a QoE within typical latency bounds.

[0085] Accordingly, the present embodiments address these and other issues by providing a method for processing texture and depth images for volumetric video consumption, for example, in the form of point clouds and / or meshes. In more detail, the sending device, such as a user’s mobile device, pre-processes the depth and texture images that are captured by one or more cameras. The sending device then divides the depth image into a foreground section and a background section and stores the information in an occupancy map. The sending device then compresses (i.e., encodes) the texture image, the depth image, and the occupancy map into one or more bitstreams and sends one or more bitstreams to a receiving device (e.g., a decoder) over a communication network. Upon receipt, the receiving device decompresses (i.e., decodes) the one or more bitstreams into a decoded texture image, a decoded depth image, and a decoded occupancy map, and then post-processes the decoded information to use in generating volumetric video. In some embodiments, artifacts in the depth image caused by the compression are removed by filtering the depth image using the occupancy map.

[0086] In more detail, a first embodiment of the present disclosure configures a sending device to convey video data (e.g., texture images, depth images, and occupancy maps) in one or more bitstreams to a receiving device.

[0087] A second embodiment of the present disclosure, which is based on the first embodiment, configures a receiving device to generate volumetric video from the received and processed texture images, depth images, and occupancy maps in the one or more received bitstreams, and to render the volumetric video to a display.

[0088] A third embodiment of the present disclosure, which is based on the previously described first and second embodiments, configures a network node to transcode video data received from a sending device before sending the transcoded video data to the receiving device in one or more bitstreams. Upon receipt, the network node decompresses (i.e., decodes) the bitstreams, processes the information in the bitstreams, and generates a volumetric video object. The network node then compresses (i.e., encodes) a representation of the video object and sends it to a second receiving device (e.g., a user’s mobile device) for rendering, for example. In one version of this embodiment, the representation comprises a reduced representation - i.e., a 2D representation of a viewport of the volumetric video object (i.e., a split rendering is performed).

[0089] Embodiments of the present disclosure provide benefits and advantages that conventional methods and techniques cannot or do not provide. For example, in one advantage, the present embodiments significantly improve the quality of a rendered mesh after compression by using an occupancy map to improve the pre- and post-processing of the depth image and texture image used to render the mesh. This allows for a much lower bitrate of the depth image when compared to conventional methods and techniques.

[0090] Another advantage of the present embodiments, especially over V-PCC, is that preprocessing is much simpler. With the present embodiments, there is no need to create and pack patches, which as those of ordinary skill in the art readily understand, is very demanding. Simpler processing means that a lower latency can be achieved, which is essential for real-time conversational video.

[0091] Additionally, embodiments of the present disclosure may be used to generate meshes in addition to other volumetric video such as point clouds. V-PCC, however, is intended for point clouds. The difference here can be significant. Point clouds are typically very large and data heavy, while meshes, in contrast, are typically lighter weight.

[0092] Further, although V-PCC uses HEVC internally, a V-PCC codec is needed to run V-PCC both on the sender side and the receiver side. The market penetration for a V-PCC codec, however, may be questioned, particularly in the area of hardware support. However, with the present embodiments, mainstream codecs can be used for encoding / compression with minimal pre- and post-processing that can be handled with software. In fact, the present embodiments are codec agnostic, meaning that any available video codec could be used with the present disclosure.

[0093] Another advantage is that the present embodiments can be applied for real-time usecases, such as holographic communication, and support bitrates as low as 4-6 Mbps. Thus, the present disclosure provides a scalable deployment over current networks.

[0094] Processing, encoding, and conveying video data for volumetric video to a receiving device Figure 3 is a functional block diagram illustrating a system 30 for implementing the present disclosure according to the first embodiment. As seen in Figure 3, system 30 comprises a communication network 32 communicatively interconnecting a sending device 40 and a receiving device 60. In different embodiments, the communication network 32 may comprise any number of wired or wireless networks, network nodes, base stations, controllers, wireless devices, relay stations, and / or any other components or systems that may facilitate or participate in the communication of video data and / or signals between the sending device 40 and the receiving device 60 whether via wired or wireless connections.

[0095] According to the present disclosure, sending device 40 pre-processes and encodes / compresses input video data for volumetric video into one or more bitstreams 34 and sends the bitstreams 34 to receiving device 60 via communication network 32. Upon receipt, receiving device 60 decompresses the one or more bitstreams 34 into decoded video data, which is then post-processed. According to various embodiments, the decoded video data may be used for a variety of purposes including, but not limited to, generating a volumetric video stream in which the volumetric video may comprise a mesh or a point cloud.

[0096] In one embodiment of the present disclosure, the input video data received / obtained by sending device 40 comprises a video stream of one or more texture images and a video stream of one or more depth images. The decoded video data received by receiving device 60 comprises one or more decoded texture images, one or more decoded depth images, and one or more decoded occupancy maps. Those of ordinary skill in the art should appreciate, however, that the present embodiments are not limited to the receipt of a single video stream. In another embodiment, the input video data comprises multiple video streams of texture and / or multiple video streams of depth images. Similarly, the decoded video data may, according to the present disclosure, comprise multiple video streams of texture, and / or multiple video streams of depth images, and / or multiple video streams of occupancy maps.

[0097] Sending device 40 comprises various components described in more detail below. These components work together in order to pre-process the video data and convey that video data to the receiving device 60. As seen in the embodiment of Figure 3, these components comprise a video source 42, a depth source 44, pre-processing circuitry 46, one or more encoders 48, and a transmitter 50. The sending device 40 may, for instance, be a mobile device, a computer, a head mounted display (HMD), such as Virtual Reality (VR), Mixed Reality (MR), or Augmented Reality (AR) glasses, another type of wearable device that comprises camera sensors, or the like. Additionally, or alternatively, sending device 40 may be implemented in another form including, but not limited to, a native operating system (e.g., iOS or Android device), device middleware, and a web-browser.

[0098] Video source 42 and depth source 44, respectively, provide the stream of texture images and the stream of depth images. Although not limiting, the video source 42, in one embodiment, is a video camera configured to provide a stream of texture images in a particular format (e.g., RGB format). In another embodiment, video source 42 is a file that comprises the stream of texture images and is stored in memory (e.g., memory that is part of, or is at least accessible to) sending device 40. Similarly, in one embodiment, depth source 44 is a depth camera configured to provide a stream of depth images to the sending device 40. In another embodiment, depth source 44 is a file comprising the stream(s) of depth images. In this latter embodiment, the file comprising the stream of texture images is stored in memory that is part of, or accessible to, sending device 40.

[0099] It should be noted that the present embodiments do not require the video source 42 and the depth source 44 to be separate entities. Rather, in at least one embodiment, video source 42 and depth source 44 comprise a single entity, such as a camera that provides both the texture and depth video in RGBD format, for example, or a single file comprising both the texture and depth video data.

[0100] The pre-processing circuitry 46 is configured to perform a variety of pre-processing functions on the video data; however, according to the present disclosure, the pre-processing functions comprise one or more of:

[0101] • determining foreground and background sections of a scene from the depth image;

[0102] • creating an occupancy map based on the depth image;

[0103] • performing hole filling functions, if necessary;

[0104] • removing the background section in the texture image and / or the depth image;

[0105] • padding the foreground of the texture image and / or the depth image;

[0106] • converting the depth image to a lower bit depth;

[0107] • converting the depth image to a lower resolution; and

[0108] • converting the color format for the texture image, the depth image, and / or the occupancy map.

[0109] The one or more encoder(s) 48 are encoder instances configured to compress the texture images, the depth images, and the occupancy maps into one or more bitstreams. According to the present disclosure, encoders 48 advantageously comprise what those of ordinary skill in the art would consider as being “mainstream” video codecs that are widely adopted for use in the market and are supported by relevant devices. Examples of mainstream encoders 48 include, but are not limited to, those that operate according to well-known standards such as WC, AVC, and HEVC.

[0110] Transmitter 50 comprises circuitry for transmitting the one or more bitstreams generated by the sending device 40 to receiving device 60. In at least one embodiment, transmitter 50 is further configured to convey metadata associated with the texture and depth images to receiving device 60. As described in more detail later, receiving device 60 uses the metadata along with the video data received from sending device 40 to generate the volumetric video (e.g., a mesh or point cloud).

[0111] The receiving device 60, in this embodiment, may be a mobile device, a computer, an HMD such as VR, MR or AR glasses, or a network entity (e.g., a cloud server or edge computing server) and comprises a receiver 62, one or more decoders 64, post-processing circuitry 66, and optionally, a display device 68.

[0112] The receiver 62 comprises circuitry configured to receive the one or more bitstreams from sending device 40 via network 32. In some embodiments, receiver 62 is also configured to receive the metadata from sending device 40 in cases where the metadata is not conveyed by other means.

[0113] Decoder(s) 64 comprise one or more decoder instances configured to decode the one or more bitstreams received from sending device 40 into decoded texture images, decoded depth images, and decoded occupancy maps. Similar to encoders 48, decoders 64 comprise mainstream video codecs, such as those that operate according to well-known standards such as WC, AVC, and HEVC.

[0114] The post-processor 66 is configured to perform various post-processing functions on the decoded texture images, decoded depth images, and decoded occupancy maps. As above, such post-processing functions may be any functions needed or desired; however, according to the present disclosure, the post-processing functions in one or more embodiments may comprise one or more of:

[0115] • filtering the depth image based on the occupancy map;

[0116] • converting the depth image to a higher bit depth;

[0117] • converting a color format for the texture image, the depth image, and / or the occupancy map; and

[0118] • generating volumetric video from the post-processed texture images and depth images. Such volumetric video may comprise, for instance, a mesh or point cloud.

[0119] Figure 4A is a flowchart illustrating a method 70, implemented at sending device 40, for obtaining, processing, and conveying video data to a receiving device via network 32 according to one embodiment of the present disclosure. For ease of discussion only, method 70 is separated into three constituent methods 80, 100, and 120 to clarify the processes that are performed on each of the texture image (method 80), the depth image (method 100), and the occupancy map (method 120). Obtaining texture and depth images

[0120] As seen in Figure 4A, sending device 40 first obtains the texture images and the depth images from the video source (box 82) and the depth source (box 102), respectively. Examples of a texture image 170 and a corresponding depth image 172 according to the present embodiments are shown in Figures 5A and 5B, respectively. As previously described, the video source and the depth source may be an RGB camera and a depth camera, respectively, and may be located, for example, on a mobile device, a tablet device, a computer, or some other device, or they may be implemented as one or more standalone cameras.

[0121] Each of the texture images and the depth images have a resolution. According to the present disclosure, the depth images may be the same resolution as that of the texture images, or they may be different resolutions (e.g., a smaller resolution). In situations where the depth images are of a different resolution than the texture images, or are not aligned with each other, then one or both of the depth images and the texture images could be resized and / or aligned before further processing is performed. In one embodiment, for example, the resolution of the depth images is modified in a pre-processing step by subsampling the resolution of the depth image with a predetermined factor (e.g., a factor of 2).

[0122] In some cases, the depth images obtained by sending device 40 may be low quality images. By way of example only, the depth images may be missing depth data for some sample (i.e., pixel) positions, thereby resulting in holes in the depth image. In such cases, the sending device 40 may fix such issues before further processing. For example, an algorithm that configures sending device 40 to correct such issues may fill a detected hole in a depth image with a sample value that is derived (e.g., averaged) from the sample values of neighboring samples.

[0123] Dividing into foreground and background sections

[0124] Once the depth images are received, the sending device 40 partitions the depth image into a foreground section and a background section (box 122). For example, consider an image showing the upper body of a person. In one embodiment, the depth image would be partitioned such that the foreground section includes the head and shoulders / upper torso, while the background section includes everything else in the image.

[0125] There are a variety of ways in which sending device 40 can partition the depth image into foreground and background sections. For instance, one embodiment of the present disclosure utilizes a threshold value. According to such embodiments, samples of the depth image having a sample value that is equal to or exceeds the threshold value are assigned to the foreground section. Samples of the depth image having a value below the threshold value, however, are assigned to the background section. The reverse is also possible in which samples of the depth image having a value that is equal to or exceeds the threshold value are assigned to the background section, while samples having a value that is less than the threshold value are assigned to the foreground section. Regardless of the assignment, however, the threshold value can be predetermined, or alternatively, be set by a control entity such as a user control, for example, and can change depending on use-case. When predetermined, the threshold value may be based on one or more of the intrinsic parameters of the camera, the surrounding environment, such as the size of a room, or a user selected value.

[0126] In another embodiment, the threshold value is adaptively selected based on a distribution or histogram of the sample values of the depth image, or on the coverage of sample values of the depth image. For example, the foreground section is expected to cover at least a certain percentage of the depth image. Therefore, in one embodiment of the present disclosure, the threshold value is selected to ensure that when the depth image is partitioned, the foreground section covers that specified percentage of the depth image.

[0127] In yet another embodiment, the threshold value is based on the sample values at predetermined positions of the depth image. For example, the predetermined position may be a central position of the depth image, such as the sample in the middle of the depth image. In these cases, a sample in a middle lower part of the depth image may be assumed to be within the foreground section, while samples in the top left and the top right corners of the depth image may be assumed to belong to the background section.

[0128] Once the depth image has been partitioned, sending device 40 generates an occupancy map (box 124) from the depth image. An example of an occupancy map 174 is shown in Figure 6. According to one embodiment, samples corresponding to a foreground section 176 are assigned a foreground value (e.g., 1) and the samples corresponding to a background section 178 are assigned a background value (e.g., 0). Thus, as seen in Figure 6, for example, the foreground section 176 of occupancy map 174, which includes a subject’s head, shoulders, and upper torso, appears in a first color (e.g., white), while everything else in the depth image is assigned to the background section 178 and appears is a second color (e.g., black).

[0129] In this embodiment, the occupancy map 174 is a binary map having only two possible value per sample (e.g., 1 or 0). Thus, the occupancy map may be generated to support a single foreground section and a single background section. However, the present disclosure is not so limited. In other embodiments, the occupancy map is stored as a map with a higher bit depth (i.e., having more than two possible values per sample). In such embodiments, the occupancy map may be generated to support multiple foreground sections with each foreground section being assigned its own unique value.

[0130] As previously described, sending device 40 is configured to detect and fix holes in the depth image upon receipt of the depth image from the depth source. However, such “hole-filling” functions need not be performed upon receipt of the depth image. In some cases, hole-filling functions of the present disclosure may be performed after the occupancy map has been constructed. If sending device 40 detects and fills holes in the depth image after the construction of the occupancy map, sending device 40 may also be configured to perform the hole-filling functions in the occupancy map. Removal of the background

[0131] Next, based on the determined foreground section and background section as stored in the occupancy map, sending device 40 is configured to remove the background section from one or both of the texture image (box 84) and the depth image (box 104). For example, as described in the embodiment above, sending device 40 partitions the depth image into the foreground section and the background section by assigning a specific value (i.e., 0 or 1) to the samples based on whether the sample is in the foreground section or background section. For instance, the specific value used to assign a sample to the background section may represent the value of whichever sample is deemed to be the farthest distance away from the camera that captured the image. Similarly, the specific value used to assign a sample to the foreground section may represent the value of whichever sample is deemed to be the closest to the camera that captured the image in terms of distance. Figures 7A-7B illustrate examples of a texture image 180 and a depth image 182, respectively, in which the background sections have been removed.

[0132] Removing the background section from one or both of the texture image and the depth image advantageously reduces the number of bits required for compressing the texture image and the depth image. Compressing a single color, such as black, is almost “free” when compared to compressing a normal video. Thus, in a situation where the foreground section covers about half of the texture image, the remaining part of the texture image would have the same value (e.g., 0). Thus, the bitrate for compression would be reduced by approximately 50%.

[0133] It should be noted, however, it is not essential for sending device 40 to remove the background section from one or both of the texture image and the depth image. In some embodiments, the removal of the background section from the depth image and / or the texture image is omitted. This may occur, for example, in situations where the background section of the texture and / or the depth image is useful at the receiving side. For example, the background section of the texture and / or depth images may not be removed in situations where the foreground section is not to be overlayed on top of a local background. Such is the case, for example, in an HMD, where the HMD may be an AR, MR, or R device.

[0134] Padding of foreground

[0135] Next, as seen in Figure 4A, sending device 40 pads one or both of the foreground section and the background section of one or both of the texture image (box 86) and the depth image (box 106) based on the information stored in the occupancy map. For example, in at least one embodiment, samples in areas of the background section that are close to (e.g., neighboring) the foreground section are assigned sample values that are similar to those of nearby (e.g., neighboring) samples in the foreground section. Figure 8 illustrates an example in which the foreground section of the depth image has been padded by replacing the sample values of the samples in areas of background section that contain both background and foreground with a median value of the foreground.

[0136] In more detail, the depth image, in one embodiment, is divided into a plurality of blocks that correspond to the CUs of the video codec being used (e.g., a block size of 32x32 samples). For each block, a check is made to determine whether the block is in both the foreground section and the background section of the depth image. If the block is determined to be in both in the foreground section and the background section, the samples of the block belonging to the background section are assigned a value that corresponds to the sample values of the neighboring blocks in the foreground section. For instance, in one embodiment, the values to be assigned to the samples of the block belonging to the background section may be extrapolated from the foreground section to the background section. Alternatively, the values to be assigned to the samples of the block belonging to the background section may be an average or median value of the foreground samples of the same block.

[0137] Regardless of the specific value, however, padding the foreground section according to the present embodiments beneficially reduces the number of bits that are required to encode the image. For example, blocks with sharp transitions, such as those on the border between the foreground and background sections, are typically more demanding to encode in terms of bits than “padded” blocks with fewer details.

[0138] Another reason for padding the foreground for depth images relates to mitigating artifacts that may be created during transform coding. Particularly, the transform coding of the encoder at low to medium bitrates can, at times, produce undesirable artifacts around sharp transitions, such as those at the borders between the foreground and background sections. These artifacts may not be easy to directly spot at the depth image but can become an issue at the receiving side when generating a mesh from the depth image. As will be described in more detail later, the artifacts are mainly mitigated when filtering a decoded depth image based on the occupancy map. However, padding the foreground section of the depth image by the sending device 40 also helps to mitigate these artifacts.

[0139] It should be noted that although this embodiment describes the padding function in the context of the depth image, the present disclosure also provides for padding the texture image in a similar manner. Further, it should also be noted that padding the foreground of one or both of the depth and texture images as described herein is not essential. In one embodiment, for example, the padding of the foreground section for the depth image and / or for the texture image is omitted.

[0140] Converting the bit depth of a depth image

[0141] Once padded, sending device 40 advantageously converts the depth image to a lower bit depth (box 108). In more detail, conventional video codecs, such as Advanced Video Coding (AVC), HEVC, and WC, typically operate at a bit depth of 8 or 10 bits for “mainstream” profiles (e.g., AVC 8 bits, HEVC 8 or 10 bits, and WC 10 bits). These profiles are, by far, the most widely adopted in the market. HEVC and WC specifications also include profiles with support for bit depths up to 16 bits. However, these profiles are not widely deployed.

[0142] Depth images are often captured at a precision that is higher than 10 bits. For example, in many cases, depth images are captured with a precision of up to 16 bits. By converting the depth images to a lower bit depth before compression (e.g., to 10 bits), the widely adopted profiles of conventional mainstream video codecs and video profiles can be used for compression. As will be described in more detail later, after the one or more bitstreams have been decompressed by receiving device 60, the bit depth of the depth image may be converted back to a higher bit depth (e.g., back up to 16 bits) if needed before generating the mesh.

[0143] When converting from a higher bit depth (e.g., from 16 bits to 10 bits), it may not be optimal to perform a typical linear scaling of all values. When depth images are captured, each value corresponds to a distance from the camera. The minimum and maximum values are set by the capturing device and may typically range from a few centimeters to tens of meters. In many scenarios, such as when an image contains a head and shoulders / upper torso, only a limited range of distances of the available range of distances is actually used. This is illustrated in histogram 186 seen in Figure 9A, which shows an example of the distribution of depth values of the depth image of Figure 5B. In particular, the first set of values 186a plotted on histogram 186 corresponds to the head and shoulders in the foreground of the image, and the second set of values 186b plotted on histogram 186 corresponds to the background section.

[0144] Note that the original 16-bit depth image has 216(i.e., 65536) possible depth values. However, merely for illustrative purposes, only values up to 4000 are shown in histogram 186. The value range of interest, i.e., the foreground, covers approximately 400 depth values in the 16-bit depth image. If converting to 10 bits using linear scaling of all values, this would mean that only every 26(i.e., 64) values would be sampled resulting in around 400 / 64 = 6 depth levels for the foreground. This is clearly not adequate.

[0145] The present disclosure, however, exploits the fact that the range of the input values is known. Therefore, in one embodiment, sending device 40 selects and clips only the range of interest (e.g., the 400 values + some margin on both sides), and then converts that range to the desired lower bit depth. This is illustrated in the histogram 188 of Figure 9B where the range of interest has been scaled to fit within 256 values to fit within bit depth 8. The scaling in the example is linear scaling, but some other type of scaling (e.g., logarithmical scaling) could be used as well. In such embodiments, values closer to the camera get a higher importance, or a weighted scaling, such that more frequent values get a higher importance (i.e., codewords are denser around frequent values).

[0146] Histogram 190 of Figure 9C shows the values after a receiving device 60 expands the bit depth back to 16 bits, where most of the details of the foreground are kept. If converting the example image to 10 bits instead of 8 bits, this could be done without losing any data since the foreground (400 values + the margin) easily fits within 210(i.e. ,1024) values. Convert color format

[0147] Prior to compression, the texture image, the depth image, and the occupancy map may need to be converted to a color format more suitable for compression. The most widely deployed mainstream profiles of AVC, HEVC and WC use the YUV format with 4:2:0 pixel subsampling, meaning that the U and color components of the YUV format are subsampled by a factor 2 in both the vertical and horizontal directions compared to the luma Y component.

[0148] Therefore, a texture image using the RGB format should be converted to YUV before compression (box 88). Similarly, the mainstream profiles do not support single component coding, so a depth image and / or an occupancy map using a single component should also be converted to the YUV format (boxes 110, 126, respectively). As will be described in more detail later, the texture image, the depth image, and the occupancy image may, after decompression, be converted to a color format more suitable for the post-processing and generating of the mesh (e.g., the texture image may be converted back to RGB and the depth image and occupancy map may be converted to single component images). Converting the color format of one or more of the texture image, the depth image, and the occupancy map is not essential for the embodiments described herein and may be omitted.

[0149] Compress and decompress with mainstream video codec

[0150] The texture images, the depth images, and the occupancy map may all be compressed with a conventional “mainstream” video codec (boxes 90, 112, 128, respectively) after color format conversion, when such conversion is performed. However, the present embodiments are not limited solely to the use of conventional mainstream video codecs. Rather, the present embodiments are codec agnostic, and thus, are configured to support other codecs that are not so widely accepted or implemented in the current market.

[0151] In one embodiment the texture images, the depth images, and the occupancy map are encoded into separate bitstreams. In other embodiments, the texture images, the depth images, and the occupancy map are combined into one or two bitstreams. In these latter embodiments, the texture image, depth image, and / or the occupancy map are either spatially aligned in a coded picture or interleaved in the bitstream.

[0152] In some embodiments, the present disclosure packs the depth image and the occupancy map into the same picture for compression using a color format with more than one component (e.g. YUV). In other embodiments, the depth image is signaled in the luma component, and the occupancy map is signaled in at least one of the U and V color components, which may or may not be subsampled. In another embodiment, the occupancy map is signaled in the Y luma component, and the depth image is signaled in at least one of the U and V color components, which may or may not be subsampled. In yet another embodiment, both the occupancy map and the depth image are signaled in one color component each.

[0153] There are benefits and advantages of compressing and signaling the depth image and the occupancy map in the same image. One benefit, for example, is the simplicity of temporally aligning the depth image and occupancy map. Another benefit is the ability to utilize the redundancy between the depth image and occupancy map in the compression. Many conventional mainstream codecs, such as profiles of HEVC and WC, for example, support prediction between the luma channel and the color channels.

[0154] Additionally, in one embodiment, the occupancy map is encoded using lossless coding or near lossless coding to preserve the edges between the foreground and the background sections. Further, in at least one embodiment, the depth image is encoded with lower quality compared to the texture image.

[0155] The one or more bitstreams are conveyed from the sending device 40 to the receiving device 60. In a preferred embodiment, the one or more bitstreams are conveyed to the receiving device 60 via network 32 (e.g., a 5G or 6G network, an Ethernet network, and / or the Internet). Alternatively, however, the one or more bitstreams may be conveyed to the receiving device 60 using other means (e.g., by using a USB drive or some other portable media format to convey the data from sending device 40 to receiving device 60.

[0156] As seen in Figure 4A, the texture image is obtained and conveyed to the receiving device 60 via network 32. However, this need not always be the case. For example, in some situations, the texture image may be conveyed to the receiving device 60 via a USB drive or other means. In such cases, one or more of the removal of the background section in the texture image, the padding of foreground section in the texture image, the color conversion of the texture image, and the compression and decompression of the texture image, as previously described, need not be performed.

[0157] Figure 4B is a flowchart illustrating a method 130, implemented at receiving device 60, for receiving and post-processing the encoded texture image, depth image, and occupancy map data via network 32 according to one embodiment of the present disclosure. For ease of discussion only, method 130 is separated into three constituent methods 140, 150, and 160 to clarify the processes that are performed on each of the texture image (method 140), the depth image (method 150), and the occupancy map (method 160).

[0158] As seen in Figure 4B, each of the one or more bitstreams comprising the encoded texture image, depth image, and occupancy map are decoded / decompressed (boxes 142, 152, and 162, respectively) using a decoder instance of the same conventional mainstream codec and profile used to encode / compress the one or more bitstreams. Additionally, provided the sending device 40 converted the color format of the texture image, depth image, and occupancy map, the receiving device will convert the color format back to their original color formats (boxes 144, 154, and 164, respectively.

[0159] Filter depth image by occupancy

[0160] After decompression, and possibly color conversion, the decoded depth image is filtered (i.e., modified) based on the decoded occupancy map (box (156). In one embodiment, for example, the depth image is filtered by assigning a specific value (e.g., 0) for each sample of the depth image that corresponds to the background section indicated in the occupancy map. This prevents edge artifacts that are caused by compression artifacts at the edges of the foreground of the depth image. Additionally, as stated previously, padding the foreground of the depth image before compression mitigates these artifacts even more. Further, by coding the occupancy map using lossless or near lossless compression, and by coding the depth image with relatively low quality, the edges of the foreground of the depth image may be preserved even at moderately low overall bitrates.

[0161] Figures 10A-10C illustrate example depth image filtering using the occupancy. In this example, the depth image was lossy compressed at 152 kbps and the occupancy map was losslessly compressed at 208 kbps. The first image 192 (Figure 10A) shows the generated mesh without using an occupancy map to filter a depth image. In the second image 194 (Figure 10B), the depth image has been filtered using an occupancy map but the foreground of the depth image has not been padded by sending device 40 prior to compression. In the third image 196 (Figure 10C), the depth image has been filtered using an occupancy map and the sending device 40 has padded the foreground of the depth image prior compression.

[0162] Accordingly, a sending device 40 may perform all or a subset of the following functions according to one or more embodiments of the present disclosure, as previously described.

[0163] • Obtain a texture image from a video source;

[0164] • Obtain a depth image from a depth source, wherein the depth image corresponds to the same content as the texture image;

[0165] • Divide the depth image into a foreground section and a background section;

[0166] • Create an occupancy map with a first sample value, e.g. 1 , for the samples; corresponding to the foreground section and a second sample value, e.g. 0, for the samples corresponding to the background section;

[0167] • Modify depth image / occupancy map by filling holes;

[0168] • Remove the background section from the texture image based on the foreground and / or background section of the occupancy map;

[0169] • Remove the background section from the depth image based on the foreground and / or background section of the occupancy map;

[0170] • Pad the foreground section of the texture image;

[0171] • Pad the foreground section of the depth image;

[0172] • Convert depth image to a lower bit depth;

[0173] • Convert the color format of the texture image;

[0174] • Convert the color format of the depth image;

[0175] • Convert the color format of the occupancy map;

[0176] • Compress the texture image into a first video bitstream using a mainstream video codec; • Compress the depth image into a second video bitstream using a mainstream video codec;

[0177] • Compress the occupancy map into a third video bitstream using a mainstream video codec;

[0178] • Send the first video bitstream over a network to the receiving entity;

[0179] • Send the second video bitstream over a network to the receiving entity; and

[0180] • Send the third video bitstream over a network to the receiving entity. The first, second and / or third bitstream may be the same bitstream.

[0181] A receiving device 60 may perform all or a subset of the following functions according to one or more embodiments of the present disclosure, as previously described.

[0182] • Receive a first video bitstream from a sending entity;

[0183] • Receive a second video bitstream from the sending entity;

[0184] • Receive a third video bitstream from the sending entity. The first, second and / or third bitstream may be the same bitstream;

[0185] • Decode from first video bitstream a decoded texture image;

[0186] • Decode from second video bitstream a decoded depth image;

[0187] • Decode from third video bitstream a decoded occupancy map;

[0188] • Convert the color format of the decoded texture image;

[0189] • Convert the color format of the decoded depth image;

[0190] • Convert the color format of the decoded occupancy map;

[0191] • Filter the decoded depth image using the decoded occupancy map such that sample values in the depth image corresponding to sample values in the occupancy map having a first sample value, e.g. 1 , are not modified and sample values in the depth image corresponding to sample values in the occupancy map having a second sample value, e.g. 0, are assigned a particular value (e.g. the value corresponding to being furthest away from the camera, e.g. 0, or a value indicating that the sample is not part of a foreground object); and

[0192] • Convert the depth image to a higher bit depth.

[0193] Generate and Render a volumetric video object

[0194] The previous embodiments illustrate functions implemented by a sending device 40 configured to pre-process and encode video data (e.g., texture images, depth images, and occupancy maps), and a receiving device 60 configured to decode and post-process the video data received from sending device 40. However, the present embodiments are not limited solely to implementing these functions. In another embodiment, seen in Figures 11A-11B, the sending device 40 is further configured to obtain and convey metadata to receiving device 60. In one embodiment, the receiving device 60 may be configured to use metadata to generate and / or render volumetric video. The volumetric video may, according to the present disclosure, be a mesh or a point cloud. In other embodiments, however, receiving device may use the metadata, along with the post-processed texture image and / or depth image, for other purposes. By way of example only, such purposes may include, but are not limited to, localization and spatial awareness, such as SLAM.

[0195] As seen in Figures 11 A-11 B, methods 200, 220, respectively, include the same submethods 80, 100, 120, 140, 150, 160 and their constituent functions previously described in Figures 4A-4B. Therefore, the details of these functions need not be described again. However, as seen in Figure 11A, method 200 comprises an additional sub-method 210 performed by sending device 40, and method 220 of Figure 11 B comprises an additional sub-method 230 performed by the receiving device 60.

[0196] In more detail, and as seen in Figure 11 A, this embodiment of the present disclosure configures sending device 40 to obtain metadata (box 212) in addition to the texture image and depth image. Once provided to the receiving device 60, the received metadata (box 232) may be used to generate the volumetric video (box 234). In one embodiment, the metadata may comprise extrinsic and / or intrinsic parameters of the camera(s) used to capture the texture and / or depth images. By way of example only, these parameters may include one or more of a focal length of the camera, an aperture of the camera, a field-of-view of the camera, and the resolution of the captured image. Such parameters may be extracted, for example, from known devices (e.g., phones or standalone cameras) using known techniques. Regardless, sending device 40 sends the metadata it obtains to receiving device 60 via network 32, as is seen in Figures 11A-11 B, or by some other means (e.g., a priori knowledge of the metadata).

[0197] Receiving device 60 may also, according to at least some embodiments of the present disclosure, comprise a display device on which the generated volumetric video is rendered (box 236). The display may, for instance, be a screen such as a mobile device screen, a TV, a computer screen, or an HMD, such as AR, MR or VR glasses, with or without pass-through.

[0198] Thus, in this embodiment of the present disclosure, sending device 40 may perform the following functions in addition to the functions previously described with Figures 4A-4B.

[0199] • Send metadata to the receiving device 60. The metadata may, for example, be used by receiving device 60 to generate the generated volumetric video (e.g., a mesh or point cloud). The metadata may, for example, comprise intrinsic and / or extrinsic camera parameters. The metadata may be embedded into the first, second, or third stream.

[0200] A receiving device configured according to this embodiment may perform all or a subset of the following functions in addition to the previously described functions it performs:

[0201] • Receive metadata, e.g., intrinsic camera parameters, used for generating the mesh or point cloud; • Generate a volumetric video object (e.g. mesh or point cloud) from the decoded texture, the decoded depth image (that may have been filtered) and possibly the metadata; and

[0202] • Render the mesh to a display (e.g., a TV set, computer screen or HMD, such as VR, MR and AR glasses).

[0203] Transcoding in the network

[0204] In another embodiment of the present disclosure, illustrated in Figure 12, the generation of the received video data is transcoded by a first receiving device 240 (e.g., a network node such as a cloud server or edge server) in network 32 as a part of a cloud service, for example. A representation of the volumetric video comprised in the video data is then reencoded / recompressed by the first receiving device 240 and sent to a second receiving device, such as receiving device 60. In one embodiment, the representation comprises the full volumetric video.

[0205] In one aspect, the representation is a reduced representation (e.g., a 2D representation of a viewport of the volumetric video). The reduced representation may be encoded to a 2D video stream and transmitted to an end-user device capable of decoding the 2D video stream. This is referred to as split rendering.

[0206] More particularly, in this embodiment, first receiving device 240 comprises a network node such as a cloud server or edge server located in network 32. As seen in Figure 12, first receiving device 240 comprises a receiver 232, one or more decoders 244, post-processing circuitry 246, one or more encoders 248, and a transmitter 250. Each of these components are similar to those described previously with respect to receiving device 60, and thus, are not described in detail here.

[0207] In operation, first receiving device 240 receives one or more bitstreams 34 from sending device 40, decodes the one or more bitstreams, and generates the volumetric video as previously described. However, in this embodiment, first receiving device 240 does not render the volumetric video to a display. Instead, the volumetric video is re-encoded by the one or more encoders 248 and sent to another receiving entity, such as receiving device 60, in one or more bitstreams 252. In these embodiments, the re-encoded volumetric video may comprise the full volumetric video, or a reduced representation of the volumetric video. In one embodiment, where the reduced representation of the volumetric video is generated and sent to receiving device 60, a viewport is first selected for the reduced representation of the volumetric video (e.g., a 2D video picture). The reduced representation of the volumetric video is then encoded using a conventional mainstream video codec, for example, and sent to receiving device 60. For example, the receiving device 60 may be capable of decoding and displaying a 2D video stream but may not be capable of decoding multiple streams and rendering a volumetric video. Accordingly, a receiving device 240 may perform all or a subset of the following functions according to one or more embodiments of the present disclosure, as previously described.

[0208] • Receive a first video bitstream from a sending device;

[0209] • Receive a second video bitstream from the sending device;

[0210] • Receive a third video bitstream from the sending device. The first, second and / or third bitstream may be the same bitstream;

[0211] • Receive metadata, e.g., intrinsic camera parameters, used for generating the mesh;

[0212] • Decode the first video bitstream into a decoded texture image;

[0213] • Decode the second video bitstream into a decoded depth image;

[0214] • Decode the third video bitstream into a decoded occupancy map;

[0215] • Convert the color format of the decoded texture image;

[0216] • Convert the color format of the decoded depth image;

[0217] • Convert the color format of the decoded occupancy map;

[0218] • Filter the decoded depth image using the decoded occupancy map;

[0219] • Convert the depth image to a higher bit depth;

[0220] • Generate a volumetric video (e.g. a mesh or a point cloud) from the decoded texture, the decoded depth image (that may have been filtered) and the metadata, when sent by the sending device;

[0221] • Select and encode / transcode a representation of the volumetric video to a fourth video bitstream. The representation may comprise the full volumetric video or a reduced representation of the volumetric video (e.g., a viewport of the volumetric video encoded as a 2D video); and

[0222] • Send the fourth bitstream to a second receiving device.

[0223] Figure 13 illustrates an example method 260, implemented by sending device 40, for conveying video data for volumetric video to a receiving device, such as the first and / or second receiving devices 60, 240. As seen in method 260, sending device 40 obtains a depth image from a depth image source (box 262), and if needed, modifies the resolution of the depth image (box 264). In one embodiment, sending device 40 modifies the depth image resolution by subsampling the resolution of the depth image according to a predetermined factor. Sending device 40 then determines whether one or more samples of the depth image is missing data (box 266). If so, sending device 40 fills the one or more samples of the depth image with data derived from one or more neighboring samples of the depth image (box 268).

[0224] Then, regardless of whether sending device 40 fills the one or more samples, sending device partitions the depth image into a foreground section and a background section (box 270) and generates an occupancy map for the foreground and background sections (box 272). In one embodiment, the occupancy map is generated to comprise a first sample value corresponding to the foreground section and a second sample value corresponding to the background section. Provided sending device 40 detected and filled holes in the depth image, sending device 40 also fills one or more samples in the occupancy map with data derived from one or more samples of the occupancy map (box 274). Sending device 40 then encodes the depth image and the occupancy map into a bitstream (box 276) and sends the bitstream over network 32 to a receiving device (box 278).

[0225] Figure 14 illustrates an example method 280, implemented by sending device 40, for partitioning a depth image into a foreground section and a background section according to embodiments of the present disclosure. As seen in Figure 14, sending device 40 samples the depth image to generate a plurality of depth image sample values (box 282). Sending device 40 them compares a depth image sample value to a threshold sample value (box 284) and assigns the depth image sample value to one of the foreground section and the background section based on a result of the comparing (box 286).

[0226] In one embodiment, the depth image sample value is assigned to the foreground section when the depth image sample value is equal to or exceeds the threshold sample value.

[0227] In one embodiment, the depth image sample value is assigned to the background section when the depth image sample value is less than the threshold sample value.

[0228] In one embodiment, the depth image sample value is assigned to the background section when the depth image sample value is equal to or exceeds the threshold sample value. in one embodiment, the depth image sample value is assigned to the foreground section when the depth image sample value is less than the threshold sample value.

[0229] In one embodiment, the threshold sample value is set based on one or more parameters associated with the depth image.

[0230] In one embodiment, the threshold sample value is set based on a position of one or more of the depth image sample values in the depth image.

[0231] In one embodiment, the threshold sample value is adaptively selected based on at least one of a distribution of the depth image sample values of the depth image and a coverage of the depth image sample values of the depth image.

[0232] In one embodiment, the one or more parameters associated with the depth image comprise at least one of a parameter of a camera that captured the depth image and a parameter of an environment associated with the depth image.

[0233] In one embodiment, the depth image is partitioned into a plurality of foreground sections and a background section. In such embodiments, each of the plurality of foreground sections comprises a unique sample value.

[0234] Figure 15 illustrates an example method 290, implemented by sending device 40, for obtaining the texture image and for sending the metadata to a receiving device. As seen in Figure 15, sending device 40 obtains the texture image from a texture image source (box 292). In this embodiment, the texture image corresponds to content in the depth image. So obtained, sending device 40 encodes the texture image into the bitstream (box 294). Additionally, in at least one embodiment, sending device 40 sends metadata to the receiving device (box 296). In one embodiment, the metadata comprises one or more parameters intrinsic to a camera that captured one or both of the depth image and the texture image.

[0235] Figure 16 illustrates an example method 300, implemented by sending device 40, for processing the background section, padding the foreground section, and converting the bit depth and color of the images. As seen in Figure 16, sending device 40 removes the background section from one or both of the depth image and the texture image based on one or both of the foreground section and the background section of the occupancy map prior to the encoding (box 302). In one embodiment, removing the background section from one or both of the depth image and the texture image comprises assigning a predetermined value to one or more samples in the background section of the one or both of the depth image and the texture image (box 304). Sending device 40 then pads the foreground section of one or both of the depth image and the texture image (box 306), converts the depth image from a current bit depth to a lower bit depth prior to the encoding, in which the lower bit depth comprises fewer bits than the current bit depth (box 308), and converts a color format of one or more of the texture image, the depth image, and the occupancy map (box 310).

[0236] Figure 17 illustrates an example method 320, implemented by sending device 40, for padding the foreground section (box 322) in more detail. As seen in Figure 17, one embodiment of the present disclosure configures sending device 40 to pad the foreground section of one or both of the depth image and the texture image by assigning a value of a sample in the foreground section of the one or both of the depth image and the texture image to a sample in the background section of the one or both of the depth image and the texture image (box 324). In this embodiment, the sample in the foreground section of the one or both of the depth image and the texture image is in a position that is proximate to a position of the sample in the background section of the one or both of the depth image and the texture image.

[0237] In another embodiment, ending device pads the foreground section of one or both of the depth image and the texture image by partitioning one or both of the depth image and the texture image into one or more blocks (box 326). In this case, each block comprises two or more samples and corresponds to a coding unit (CU) of a video codec used to encode the depth image and / or the texture image into the bitstream. Then, for each of the two or more samples in the one or more blocks, sending device 40 determines whether a block of the sample is in both the foreground section and the background section of the depth image and / or the texture image (box 328), and also determines whether the sample is in the background section of the one or both of the depth image and the texture image (box 330). Responsive to determining that the block of the sample is in both the foreground section and the background section of the depth image and / or the texture image, and that the sample is in the background section of the depth image and / or the texture image, sending device 40 assigns a padding value to the sample of the block of the background section (box 332). The padding value is derived from one or more sample values of the foreground section that neighbors the sample of the block of the background section. The method 330 continues as long as there are more samples (box 334) and more blocks (box 336) to process.

[0238] In one embodiment, the padding value is extrapolated from the one or more sample values of the foreground section.

[0239] In one embodiment, the padding value is an average value or a median value of a plurality of samples of the foreground section and the block comprises samples from both the foreground section and the background section.

[0240] In one embodiment, the depth image comprises a plurality of depth values with each depth value indicating a distance from a camera that captured the depth image and being bounded by a minimum depth value and a maximum depth value.

[0241] Figure 18 illustrates an example method 340, implemented by sending device 40, for converting the bit depth of the depth image from a current bit depth to a lower bit depth (box 342) according to one embodiment. As seen in Figure 18, sending device 40 selects a range of depth values from the plurality of depth values (box 344), and then converts the range of depth values to lower depth values by scaling the depth values in the range of depth values to fit within the lower bit depth (box 346).

[0242] In one embodiment, the bitstream comprises one of a single bitstream in which the texture image, the depth image, and the occupancy map are encoded into the single bitstream.

[0243] In one embodiment, the bitstream comprises first and second separate bitstreams, in which two of the texture image, the depth image, and the occupancy map are encoded into the first bitstream and the other of the texture image, the depth image, and the occupancy map are encoded into the second bitstream.

[0244] In one embodiment, the bitstream comprises first, second, and third separate bitstreams in which the depth image is encoded into the first bitstream, the occupancy map is encoded into the second bitstream, and the texture image is encoded into the third bitstream.

[0245] In one embodiment, the multi-component color format comprises a YUV format.

[0246] In one embodiment, the depth image is signaled in the luminance component of the YUV format, and the occupancy map is signaled in at least one of the color components of the YUV format.

[0247] In one embodiment, the occupancy map is signaled in the luminance component of the YUV format, and the depth image is signaled in at least one of the color components of the YUV format.

[0248] In one embodiment, the occupancy map is signaled in a first color component of the YUV format, and the depth image is signaled in a second color component of the YUV format.

[0249] In one embodiment, the depth image is encoded with a lower quality than the texture image. Figure 19 illustrates an example method 350, implemented by a receiving device, such as receiving device 60 or receiving device 240, for receiving video data for volumetric video from a sending device. As seen in Figure 19, the receiving device receives, from the sending device, an encoded bitstream comprising a depth image and occupancy map (box 352). The occupancy map in this embodiment comprises a first sample value corresponding to a foreground section of the depth image and a second sample value corresponding to a background section of the depth image. The receiving device may also receive a texture image (box 354). The texture image corresponds to the content in the received depth image and is also encoded in the bitstream. The receiving device then decodes the depth image and the occupancy map from the encoded bitstream (box 356) and, when received, also decodes the texture image from the encoded bitstream (box 358). Receiving device then converts a color format of one or more of the texture image, the depth image, and the occupancy map (box 360) and filters the depth image based on the occupancy map (box 362). Then, based on the texture image and the depth image, the receiving device generates a volumetric video (box 364). the volumetric video may comprise a mesh or a point cloud. In one embodiment, the receiving device then renders the volumetric video to a display device (box 366).

[0250] Figure 20 illustrates an example method 370, implemented by the receiving device, for filtering the depth image based on the occupancy map. As seen in Figure 20, to filter the depth image based on the occupancy map, the receiving device modifies the depth values in the depth image according to the occupancy map (box 372). The depth values correspond to the background section of the depth image according to the occupancy map. To accomplish the modifications, the receiving device can, according to the present disclosure, perform either of the following. In one embodiment, receiving device assigns a sample value to each sample in the depth image that corresponds to the background section of the depth image as indicated in the occupancy map (box 374). In another embodiment, the receiving device assigns a sample value to each sample in the depth image that does not correspond to the foreground section of the depth image as indicated in the occupancy map (box 376). The receiving device then converts the depth image from a current bit depth to a higher bit depth (box 378). The higher bit depth comprises more bits than the current bit depth. In some embodiments, the receiving device also receives metadata from the sending device (box 380). The meta data comprises one or more parameters intrinsic to a camera that captured one or both of the depth image and the texture image.

[0251] In one embodiment, a first receiving device 240 in the communications network implements method 350. In such embodiments, the first receiving device 240 is configured to perform additional functions, which are illustrated in Figure 21 . It should be noted here that the method 350 of Figure 21 is the same method 350 as previously described with respect to Figure 19. Therefore, steps that are the same in Figures 19 and 21 are indicated using the same reference numbers. As seen in Figure 21 , once the first receiving device 240 has filtered the depth image based on the occupancy map (box 362), the first receiving device 240 generates a fourth bitstream comprising a representation of either the mesh or the point cloud (box 390). So generated, the first receiving device 240 sends the generated fourth bitstream to a second receiving device, which in one embodiment is receiving device 60 (box 392).

[0252] In one embodiment, the representation of the one of the mesh and the point cloud is a reduced representation that comprises a viewpoint of the one of the mesh and the point cloud.

[0253] In one embodiment, the occupancy map is a binary map.

[0254] In one embodiment, the occupancy map comprises more than two values per sample.

[0255] In one embodiment, the sending device comprises a mobile device.

[0256] In one embodiment, the sending device comprises a computing device.

[0257] In one embodiment, the sending device comprises a Head Mounted Display (HMD). In such embodiments, the HMD may be, for example, a Virtual Reality (VR) device, an augmented reality (AR) device, a mixed reality device, or a wearable device comprising one or more sensors.

[0258] In one embodiment, the receiving device comprises a mobile device.

[0259] In one embodiment, the receiving device comprises a computing device.

[0260] In one embodiment, the receiving device comprises a Head Mounted Display (HMD). In such embodiments, the HMD may be, for example, a Virtual Reality (VR) device, an augmented reality (AR) device, a mixed reality device, or a wearable device comprising one or more sensors.

[0261] Additionally, in one embodiment, the receiving device comprises a network node.

[0262] An apparatus can perform any of the methods herein described by implementing any functional means, modules, units, or circuitry. In one embodiment, for example, the apparatuses comprise respective circuits or circuitry configured to perform the steps shown in the method figures. The circuits or circuitry in this regard may comprise circuits dedicated to performing certain functional processing and / or one or more microprocessors in conjunction with memory. For instance, the circuitry may include one or more microprocessor or microcontrollers, as well as other digital hardware, which may include Digital Signal Processors (DSPs), special-purpose digital logic, and the like. The processing circuitry may be configured to execute program code stored in memory, which may include one or several types of memory such as read-only memory (ROM), random-access memory, cache memory, flash memory devices, optical storage devices, etc. Program code stored in memory may include program instructions for executing one or more telecommunications and / or data communications protocols as well as instructions for carrying out one or more of the techniques described herein, in several embodiments. In embodiments that employ memory, the memory stores program code that, when executed by the one or more processors, carries out the techniques described herein. Figure 22 illustrates an example sending device 40 configured for conveying video data for volumetric video to a receiving device. As seen in Figure 22, sending device 40 comprises processing circuitry 400, a memory 402, and communication circuitry 406.

[0263] The communication circuitry 406 comprises the hardware required for communicating with a receiving device 60, 240 via network 32. In some embodiments, sending device is capable of wireless communications, and thus, comprises the radio frequency (RF) circuitry needed for transmitting and receiving signals over a wireless communication channel. In these cases, sending device 40 may also be coupled to one or more antenna (not shown). In other embodiments, however, sending device 40 is configured to communicate via a wire interface. In these embodiments, communication circuitry 406 may comprise an ETHERNET or similar interface.

[0264] The processing circuitry 400 controls the overall operation of the sending device 40 and processes the signals and data that is transmitted to or received by the sending device 40. The processing circuitry 400 may comprise one or more microprocessors, hardware, firmware, or a combination thereof. The processing circuitry 400 in one embodiment is configured to implement any of the methods of any of Figures 13-18.

[0265] Memory 402 comprises both volatile and non-volatile memory for storing computer program code and data needed by the processing circuit 400 for operation. Memory 402 may comprise any tangible, non-transitory computer-readable storage medium for storing data including electronic, magnetic, optical, electromagnetic, or semiconductor data storage. Memory 402 stores a computer program 404 comprising executable instructions that configure the processing circuitry 400 to implement any of the methods of any of Figures 13-18. A computer program 404 in this regard may comprise one or more code modules corresponding to the means or units described above. In general, computer program instructions and configuration information are stored in a non-volatile memory, such as a ROM, erasable programmable read only memory (EPROM) or flash memory. Temporary data generated during operation may be stored in a volatile memory, such as a random access memory (RAM). In some embodiments, computer program 404 for configuring the processing circuit 400 as herein described may be stored in a removable memory, such as a portable compact disc, portable digital video disc, or other removable media. The computer program 404 may also be embodied in a carrier such as an electronic signal, optical signal, radio signal, or computer readable storage medium.

[0266] Figure 23 illustrates an example receiving device 60 configured for receiving video data for volumetric video from a sending device. As seen in Figure 23, receiving device 60 comprises processing circuitry 500, a memory 502, and communication circuitry 506.

[0267] The communication circuitry 506 comprises the hardware required for communicating with sending device 40 and / or the first receiving device 240 via network 32. In some embodiments, receiving device 60 is capable of wireless communications, and thus, comprises the radio frequency (RF) circuitry needed for transmitting and receiving signals over a wireless communication channel. In these cases, receiving device 60 may also be coupled to one or more antenna (not shown). In other embodiments, however, receiving device 60 is configured to communicate via a wire interface. In these embodiments, communication circuitry 506 may comprise an ETHERNET or similar interface.

[0268] The processing circuitry 500 controls the overall operation of the receiving device 60 and processes the signals and data that is transmitted to or received by the receiving device 60. The processing circuitry 500 may comprise one or more microprocessors, hardware, firmware, or a combination thereof. The processing circuitry 500 in one embodiment is configured to implement any of the methods of any of Figures 19-20.

[0269] Memory 502 comprises both volatile and non-volatile memory for storing computer program code and data needed by the processing circuit 500 for operation. Memory 502 may comprise any tangible, non-transitory computer-readable storage medium for storing data including electronic, magnetic, optical, electromagnetic, or semiconductor data storage. Memory 502 stores a computer program 504 comprising executable instructions that configure the processing circuitry 400 to implement any of the methods of any of Figures 19-20. A computer program 504 in this regard may comprise one or more code modules corresponding to the means or units described above. In general, computer program instructions and configuration information are stored in a non-volatile memory, such as a ROM, erasable programmable read only memory (EPROM) or flash memory. Temporary data generated during operation may be stored in a volatile memory, such as a random access memory (RAM). In some embodiments, computer program 504 for configuring the processing circuit 500 as herein described may be stored in a removable memory, such as a portable compact disc, portable digital video disc, or other removable media. The computer program 504 may also be embodied in a carrier such as an electronic signal, optical signal, radio signal, or computer readable storage medium.

[0270] Figure 24 illustrates an example first receiving device 24 configured for receiving video data for volumetric video from a sending device, and further, transcoding the data to send to a second receiving device, such as receiving device 60, for example. As seen in Figure 24, receiving device 240 comprises processing circuitry 600, a memory 602, and communication circuitry 606.

[0271] The communication circuitry 606 comprises a network interface having the hardware required for communicating with sending device 40 and / or the second receiving device 40 via network 32. In some embodiments, receiving device 240 is capable of wireless communications, and thus, comprises the radio frequency (RF) circuitry needed for transmitting and receiving signals over a wireless communication channel. In these cases, receiving device 240 may also be coupled to one or more antenna (not shown). In other embodiments, however, receiving device 240 is configured to communicate via a wire interface. In these embodiments, communication circuitry 606 may comprise an ETHERNET or similar interface.

[0272] The processing circuitry 600 controls the overall operation of the receiving device 240 and processes the signals and data that is transmitted to or received by the receiving device 240. The processing circuitry 600 may comprise one or more microprocessors, hardware, firmware, or a combination thereof. The processing circuitry 600 in one embodiment is configured to implement method 350 of Figure 21 .

[0273] Memory 602 comprises both volatile and non-volatile memory for storing computer program code and data needed by the processing circuit 600 for operation. Memory 602 may comprise any tangible, non-transitory computer-readable storage medium for storing data including electronic, magnetic, optical, electromagnetic, or semiconductor data storage. Memory 602 stores a computer program 604 comprising executable instructions that configure the processing circuitry 600 to implement method 350 of Figure 21 . A computer program 604 in this regard may comprise one or more code modules corresponding to the means or units described above. In general, computer program instructions and configuration information are stored in a non-volatile memory, such as a ROM, erasable programmable read only memory (EPROM) or flash memory. Temporary data generated during operation may be stored in a volatile memory, such as a random access memory (RAM). In some embodiments, computer program 604 for configuring the processing circuit 600 as herein described may be stored in a removable memory, such as a portable compact disc, portable digital video disc, or other removable media. The computer program 604 may also be embodied in a carrier such as an electronic signal, optical signal, radio signal, or computer readable storage medium.

[0274] Those skilled in the art will also appreciate that embodiments herein further include corresponding computer programs. A computer program comprises instructions which, when executed on at least one processor of an apparatus, cause the apparatus to carry out any of the respective processing described above. A computer program in this regard may comprise one or more code modules corresponding to the means or units described above.

[0275] Embodiments further include a carrier containing such a computer program. This carrier may comprise one of an electronic signal, optical signal, radio signal, or computer readable storage medium.

[0276] In this regard, embodiments herein also include a computer program product stored on a non-transitory computer readable (storage or recording) medium and comprising instructions that, when executed by a processor of an apparatus, cause the apparatus to perform as described above.

[0277] Embodiments further include a computer program product comprising program code portions for performing the steps of any of the embodiments herein when the computer program product is executed by a computing device. This computer program product may be stored on a computer readable recording medium. Additional embodiments will now be described. At least some of these embodiments may be described as applicable in certain contexts and / or wireless network types for illustrative purposes, but the embodiments are similarly applicable in other contexts and / or wireless network types not explicitly described.

[0278] Figure 25 shows an example of a communication system 1100 in accordance with some embodiments.

[0279] In the example, the communication system 1100 includes a telecommunication network 1102 that includes an access network 1104, such as a radio access network (RAN), and a core network 1106, which includes one or more core network nodes 1108. The access network 1104 includes one or more access network nodes, such as network nodes 1110a and 1110b (one or more of which may be generally referred to as network nodes 1110), or any other similar 3rd Generation Partnership Project (3GPP) access nodes or non-3GPP access points. Moreover, as will be appreciated by those of skill in the art, a network node is not necessarily limited to an implementation in which a radio portion and a baseband portion are supplied and integrated by a single vendor. Thus, it will be understood that network nodes include disaggregated implementations or portions thereof. For example, in some embodiments, the telecommunication network 1102 includes one or more Open-RAN (ORAN) network nodes. An ORAN network node is a node in the telecommunication network 1102 that supports an ORAN specification (e.g., a specification published by the O-RAN Alliance, or any similar organization) and may operate alone or together with other nodes to implement one or more functionalities of any node in the telecommunication network 1102, including one or more network nodes 1110 and / or core network nodes 1108.

[0280] Examples of an ORAN network node include an open radio unit (O-RU), an open distributed unit (O-DU), an open central unit (O-CU), including an O-CU control plane (O-CU- CP) or an O-CU user plane (O-CU-UP), a RAN intelligent controller (near-real time or non-real time) hosting software or software plug-ins, such as a near-real time control application (e.g., xApp) or a non-real time control application (e.g., rApp), or any combination thereof (the adjective “open” designating support of an ORAN specification). The network node may support a specification by, for example, supporting an interface defined by the ORAN specification, such as an A1 , F1 , W1 , E1 , E2, X2, Xn interface, an open fronthaul user plane interface, or an open fronthaul management plane interface. Moreover, an ORAN access node may be a logical node in a physical node. Furthermore, an ORAN network node may be implemented in a virtualization environment (described further below) in which one or more network functions are virtualized. For example, the virtualization environment may include an O-Cloud computing platform orchestrated by a Service Management and Orchestration Framework via an 0-2 interface defined by the O-RAN Alliance or comparable technologies. The network nodes 1110 facilitate direct or indirect connection of user equipment (UE), such as by connecting UEs 1112A, 1112B, 1112C, and 1112D (one or more of which may be generally referred to as UEs 1112) to the core network 1106 over one or more wireless connections.

[0281] Example wireless communications over a wireless connection include transmitting and / or receiving wireless signals using electromagnetic waves, radio waves, infrared waves, and / or other types of signals suitable for conveying information without the use of wires, cables, or other material conductors. Moreover, in different embodiments, the communication system 1100 may include any number of wired or wireless networks, network nodes, UEs, and / or any other components or systems that may facilitate or participate in the communication of data and / or signals whether via wired or wireless connections. The communication system 1100 may include and / or interface with any type of communication, telecommunication, data, cellular, radio network, and / or other similar type of system.

[0282] The UEs 1112 may be any of a wide variety of communication devices, including wireless devices arranged, configured, and / or operable to communicate wirelessly with the network nodes 1110 and other communication devices. Similarly, the network nodes 1110 are arranged, capable, configured, and / or operable to communicate directly or indirectly with the UEs 1112 and / or with other network nodes or equipment in the telecommunication network 1102 to enable and / or provide network access, such as wireless network access, and / or to perform other functions, such as administration in the telecommunication network 1102.

[0283] In the depicted example, the core network 1106 connects the network nodes 1110 to one or more hosts, such as host 1116. These connections may be direct or indirect via one or more intermediary networks or devices. In other examples, network nodes may be directly coupled to hosts. The core network 1106 includes one more core network nodes (e.g., core network node 1108) that are structured with hardware and software components. Features of these components may be substantially similar to those described with respect to the UEs, network nodes, and / or hosts, such that the descriptions thereof are generally applicable to the corresponding components of the core network node 1108. Example core network nodes include functions of one or more of a Mobile Switching Center (MSC), Mobility Management Entity (MME), Home Subscriber Server (HSS), Access and Mobility Management Function (AMF), Session Management Function (SMF), Authentication Server Function (AUSF), Subscription Identifier De-concealing function (SIDF), Unified Data Management (UDM), Security Edge Protection Proxy (SEPP), Network Exposure Function (NEF), and / or a User Plane Function (UPF).

[0284] The host 1116 may be under the ownership or control of a service provider other than an operator or provider of the access network 1104 and / or the telecommunication network 1102, and may be operated by the service provider or on behalf of the service provider. The host 1116 may host a variety of applications to provide one or more service. Examples of such applications include live and pre-recorded audio / video content, data collection services such as retrieving and compiling data on various ambient conditions detected by a plurality of UEs, analytics functionality, social media, functions for controlling or otherwise interacting with remote devices, functions for an alarm and surveillance center, or any other such function performed by a server.

[0285] As a whole, the communication system 1100 of Figure 25 enables connectivity between the UEs, network nodes, and hosts. In that sense, the communication system may be configured to operate according to predefined rules or procedures, such as specific standards that include, but are not limited to: Global System for Mobile Communications (GSM); Universal Mobile Telecommunications System (UMTS); Long Term Evolution (LTE), and / or other suitable 2G, 3G, 4G, 5G standards, or any applicable future generation standard (e.g., 6G); wireless local area network (WLAN) standards, such as the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standards (WiFi); and / or any other appropriate wireless communication standard, such as the Worldwide Interoperability for Microwave Access (WiMax), Bluetooth, Z-Wave, Near Field Communication (NFC) ZigBee, LiFi, and / or any low- power wide-area network (LPWAN) standards such as LoRa and Sigfox.

[0286] In some examples, the telecommunication network 1102 is a cellular network that implements 3GPP standardized features. Accordingly, the telecommunications network 1102 may support network slicing to provide different logical networks to different devices that are connected to the telecommunication network 1102. For example, the telecommunications network 1102 may provide Ultra Reliable Low Latency Communication (URLLC) services to some UEs, while providing Enhanced Mobile Broadband (eMBB) services to other UEs, and / or Massive Machine Type Communication (mMTC)ZMassive loT services to yet further UEs.

[0287] In some examples, the UEs 1112 are configured to transmit and / or receive information without direct human interaction. For instance, a UE may be designed to transmit information to the access network 1104 on a predetermined schedule, when triggered by an internal or external event, or in response to requests from the access network 1104. Additionally, a UE may be configured for operating in single- or multi-RAT or multi-standard mode. For example, a UE may operate with any one or combination of Wi-Fi, NR (New Radio) and LTE, i.e. being configured for multi-radio dual connectivity (MR-DC), such as E-UTRAN (Evolved-UMTS Terrestrial Radio Access Network) New Radio - Dual Connectivity (EN-DC).

[0288] In the example, the hub 1114 communicates with the access network 1104 to facilitate indirect communication between one or more UEs (e.g., UE 1112c and / or 1112D) and network nodes (e.g., network node 1110B). In some examples, the hub 1114 may be a controller, router, content source and analytics, or any of the other communication devices described herein regarding UEs. For example, the hub 1114 may be a broadband router enabling access to the core network 1106 for the UEs. As another example, the hub 1114 may be a controller that sends commands or instructions to one or more actuators in the UEs. Commands or instructions may be received from the UEs, network nodes 1110, or by executable code, script, process, or other instructions in the hub 1114. As another example, the hub 1114 may be a data collector that acts as temporary storage for UE data and, in some embodiments, may perform analysis or other processing of the data. As another example, the hub 1114 may be a content source. For example, for a UE that is a VR headset, display, loudspeaker or other media delivery device, the hub 1114 may retrieve VR assets, video, audio, or other media or data related to sensory information via a network node, which the hub 1114 then provides to the UE either directly, after performing local processing, and / or after adding additional local content. In still another example, the hub 1114 acts as a proxy server or orchestrator for the UEs, in particular if one or more of the UEs are low energy loT devices.

[0289] The hub 1114 may have a constant / persistent or intermittent connection to the network node 1110B. The hub 1114 may also allow for a different communication scheme and / or schedule between the hub 1114 and UEs (e.g., UE 1112C and / or 1112D), and between the hub 1114 and the core network 1106. In other examples, the hub 1114 is connected to the core network 1106 and / or one or more UEs via a wired connection. Moreover, the hub 1114 may be configured to connect to an M2M service provider over the access network 1104 and / or to another UE over a direct connection. In some scenarios, UEs may establish a wireless connection with the network nodes 1110 while still connected via the hub 1114 via a wired or wireless connection. In some embodiments, the hub 1114 may be a dedicated hub - that is, a hub whose primary function is to route communications to / from the UEs from / to the network node 1110B. In other embodiments, the hub 1114 may be a non-dedicated hub - that is, a device which is capable of operating to route communications between the UEs and network node 1110B, but which is additionally capable of operating as a communication start and / or end point for certain data channels.

[0290] Figure 26 shows a UE 1200 in accordance with some embodiments. As used herein, a UE refers to a device capable, configured, arranged and / or operable to communicate wirelessly with network nodes and / or other UEs. Examples of a UE include, but are not limited to, a smart phone, mobile phone, cell phone, voice over IP (VoIP) phone, wireless local loop phone, desktop computer, personal digital assistant (PDA), wireless cameras, gaming console or device, music storage device, playback appliance, wearable terminal device, wireless endpoint, mobile station, tablet, laptop, laptop-embedded equipment (LEE), laptop-mounted equipment (LME), smart device, wireless customer-premise equipment (CPE), vehicle, vehicle-mounted or vehicle embedded / integrated wireless device, etc. Other examples include any UE identified by the 3rd Generation Partnership Project (3GPP), including a narrow band internet of things (NB- loT) UE, a machine type communication (MTC) UE, and / or an enhanced MTC (eMTC) UE.

[0291] A UE may support device-to-device (D2D) communication, for example by implementing a 3GPP standard for sidelink communication, Dedicated Short-Range Communication (DSRC), vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I), or vehicle-to-everything (V2X). In other examples, a UE may not necessarily have a user in the sense of a human user who owns and / or operates the relevant device. Instead, a UE may represent a device that is intended for sale to, or operation by, a human user but which may not, or which may not initially, be associated with a specific human user (e.g., a smart sprinkler controller). Alternatively, a UE may represent a device that is not intended for sale to, or operation by, an end user but which may be associated with or operated for the benefit of a user (e.g., a smart power meter).

[0292] The UE 1200 includes processing circuitry 1202 that is operatively coupled via a bus 1204 to an input / output interface 1206, a power source 1208, a memory 1210, a communication interface 1212, and / or any other component, or any combination thereof. Certain UEs may utilize all or a subset of the components shown in Figure 26. The level of integration between the components may vary from one UE to another UE. Further, certain UEs may contain multiple instances of a component, such as multiple processors, memories, transceivers, transmitters, receivers, etc.

[0293] The processing circuitry 1202 is configured to process instructions and data and may be configured to implement any sequential state machine operative to execute instructions stored as machine-readable computer programs in the memory 1210. The processing circuitry 1202 may be implemented as one or more hardware-implemented state machines (e.g., in discrete logic, field-programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), etc.); programmable logic together with appropriate firmware; one or more stored computer programs, general-purpose processors, such as a microprocessor or digital signal processor (DSP), together with appropriate software; or any combination of the above. For example, the processing circuitry 1202 may include multiple central processing units (CPUs).

[0294] In the example, the input / output interface 1206 may be configured to provide an interface or interfaces to an input device, output device, or one or more input and / or output devices. Examples of an output device include a speaker, a sound card, a video card, a display, a monitor, a printer, an actuator, an emitter, a smartcard, another output device, or any combination thereof. An input device may allow a user to capture information into the UE 1200. Examples of an input device include a touch-sensitive or presence-sensitive display, a camera (e.g., a digital camera, a digital video camera, a web camera, etc.), a microphone, a sensor, a mouse, a trackball, a directional pad, a trackpad, a scroll wheel, a smartcard, and the like. The presence-sensitive display may include a capacitive or resistive touch sensor to sense input from a user. A sensor may be, for instance, an accelerometer, a gyroscope, a tilt sensor, a force sensor, a magnetometer, an optical sensor, a proximity sensor, a biometric sensor, etc., or any combination thereof. An output device may use the same type of interface port as an input device. For example, a Universal Serial Bus (USB) port may be used to provide an input device and an output device.

[0295] In some embodiments, the power source 1208 is structured as a battery or battery pack. Other types of power sources, such as an external power source (e.g., an electricity outlet), photovoltaic device, or power cell, may be used. The power source 1208 may further include power circuitry for delivering power from the power source 1208 itself, and / or an external power source, to the various parts of the UE 1200 via input circuitry or an interface such as an electrical power cable. Delivering power may be, for example, for charging of the power source 1208. Power circuitry may perform any formatting, converting, or other modification to the power from the power source 1208 to make the power suitable for the respective components of the UE 1200 to which power is supplied.

[0296] The memory 1210 may be or be configured to include memory such as random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic disks, optical disks, hard disks, removable cartridges, flash drives, and so forth. In one example, the memory 1210 includes one or more application programs 1214, such as an operating system, web browser application, a widget, gadget engine, or other application, and corresponding data 1216. The memory 1210 may store, for use by the UE 1200, any of a variety of various operating systems or combinations of operating systems.

[0297] The memory 1210 may be configured to include a number of physical drive units, such as redundant array of independent disks (RAID), flash memory, USB flash drive, external hard disk drive, thumb drive, pen drive, key drive, high-density digital versatile disc (HD-DVD) optical disc drive, internal hard disk drive, Blu-Ray optical disc drive, holographic digital data storage (HDDS) optical disc drive, external mini-dual in-line memory module (DIMM), synchronous dynamic random access memory (SDRAM), external micro-DIMM SDRAM, smartcard memory such as tamper resistant module in the form of a universal integrated circuit card (UICC) including one or more subscriber identity modules (SIMs), such as a USIM and / or ISIM, other memory, or any combination thereof. The UICC may for example be an embedded UICC (eUlCC), integrated UICC (iUICC) or a removable UICC commonly known as ‘SIM card.’ The memory 1210 may allow the UE 1200 to access instructions, application programs and the like, stored on transitory or non-transitory memory media, to off-load data, or to upload data. An article of manufacture, such as one utilizing a communication system may be tangibly embodied as or in the memory 1210, which may be or comprise a device-readable storage medium.

[0298] The processing circuitry 1202 may be configured to communicate with an access network or other network using the communication interface 1212. The communication interface 1212 may comprise one or more communication subsystems and may include or be communicatively coupled to an antenna 1222. The communication interface 1212 may include one or more transceivers used to communicate, such as by communicating with one or more remote transceivers of another device capable of wireless communication (e.g., another UE or a network node in an access network). Each transceiver may include a transmitter 1218 and / or a receiver 1220 appropriate to provide network communications (e.g., optical, electrical, frequency allocations, and so forth). Moreover, the transmitter 1218 and receiver 1220 may be coupled to one or more antennas (e.g., antenna 1222) and may share circuit components, software or firmware, or alternatively be implemented separately.

[0299] In the illustrated embodiment, communication functions of the communication interface 1212 may include cellular communication, Wi-Fi communication, LPWAN communication, data communication, voice communication, multimedia communication, short-range communications such as Bluetooth, near-field communication, location-based communication such as the use of the global positioning system (GPS) to determine a location, another like communication function, or any combination thereof. Communications may be implemented in according to one or more communication protocols and / or standards, such as IEEE 802.11 , Code Division Multiplexing Access (CDMA), Wideband Code Division Multiple Access (WCDMA), GSM, LTE, New Radio (NR), UMTS, WiMax, Ethernet, transmission control protocol / internet protocol (TCP / IP), synchronous optical networking (SONET), Asynchronous Transfer Mode (ATM), QUIC, Hypertext Transfer Protocol (HTTP), and so forth.

[0300] Regardless of the type of sensor, a UE may provide an output of data captured by its sensors, through its communication interface 1212, via a wireless connection to a network node. Data captured by sensors of a UE can be communicated through a wireless connection to a network node via another UE. The output may be periodic (e.g., once every 15 minutes if it reports the sensed temperature), random (e.g., to even out the load from reporting from several sensors), in response to a triggering event (e.g., when moisture is detected an alert is sent), in response to a request (e.g., a user initiated request), or a continuous stream (e.g., a live video feed of a patient).

[0301] As another example, a UE comprises a mobile device, such as phone or tablet having a depth sensor, Augmented Reality (AR) glasses, a Mixed Reality (MR) device, a Virtual Reality (VR) device, an actuator, a motor, or a switch, related to a communication interface configured to receive wireless input from a network node via a wireless connection. In response to the received wireless input the states of the actuator, the motor, or the switch may change. For example, the UE may comprise a motor that adjusts the control surfaces or rotors of a drone in flight according to the received input or to a robotic arm performing a medical procedure according to the received input.

[0302] A UE, when in the form of an Internet of Things (loT) device, may be a device for use in one or more application domains, these domains comprising, but not limited to, city wearable technology, extended industrial application and healthcare. Non-limiting examples of such an loT device are a device which is or which is embedded in: a connected refrigerator or freezer, a TV, a connected lighting device, an electricity meter, a robot vacuum cleaner, a voice controlled smart speaker, a home security camera, a motion detector, a thermostat, a smoke detector, a door / window sensor, a flood / moisture sensor, an electrical door lock, a connected doorbell, an air conditioning system like a heat pump, an autonomous vehicle, a surveillance system, a weather monitoring device, a vehicle parking monitoring device, an electric vehicle charging station, a smart watch, a fitness tracker, a head-mounted display for Augmented Reality (AR) or Virtual Reality (VR), a wearable for tactile augmentation or sensory enhancement, a water sprinkler, an animal- or item-tracking device, a sensor for monitoring a plant or animal, an industrial robot, an Unmanned Aerial Vehicle (UAV), and any kind of medical device, like a heart rate monitor or a remote controlled surgical robot. A UE in the form of an loT device comprises circuitry and / or software in dependence of the intended application of the loT device in addition to other components as described in relation to the UE 1200 shown in Figure 26.

[0303] As yet another specific example, in an loT scenario, a UE may represent a machine or other device that performs monitoring and / or measurements, and transmits the results of such monitoring and / or measurements to another UE and / or a network node. The UE may in this case be an M2M device, which may in a 3GPP context be referred to as an MTC device. As one particular example, the UE may implement the 3GPP NB-loT standard. In other scenarios, a UE may represent a vehicle, such as a car, a bus, a truck, a ship and an airplane, or other equipment that is capable of monitoring and / or reporting on its operational status or other functions associated with its operation.

[0304] In practice, any number of UEs may be used together with respect to a single use case. For example, a first UE might be or be integrated in a drone and provide the drone’s speed information (obtained through a speed sensor) to a second UE that is a remote controller operating the drone. When the user makes changes from the remote controller, the first UE may adjust the throttle on the drone (e.g. by controlling an actuator) to increase or decrease the drone’s speed. The first and / or the second UE can also include more than one of the functionalities described above. For example, a UE might comprise the sensor and the actuator, and handle communication of data for both the speed sensor and the actuators.

[0305] Figure 27 shows a network node 1300 in accordance with some embodiments. As used herein, network node refers to equipment capable, configured, arranged and / or operable to communicate directly or indirectly with a UE and / or with other network nodes or equipment, in a telecommunication network. Examples of network nodes include, but are not limited to, access points (APs) (e.g., radio access points), base stations (BSs) (e.g., radio base stations, Node Bs, evolved Node Bs (eNBs) and NR NodeBs (gNBs)), O-RAN nodes or components of an O-RAN node (e.g., O-RU, O-DU, O-CU).

[0306] Base stations may be categorized based on the amount of coverage they provide (or, stated differently, their transmit power level) and so, depending on the provided amount of coverage, may be referred to as femto base stations, pico base stations, micro base stations, or macro base stations. A base station may be a relay node or a relay donor node controlling a relay. A network node may also include one or more (or all) parts of a distributed radio base station such as centralized digital units, distributed units (e.g., in an O-RAN access node) and / or remote radio units (RRUs), sometimes referred to as Remote Radio Heads (RRHs). Such remote radio units may or may not be integrated with an antenna as an antenna integrated radio. Parts of a distributed radio base station may also be referred to as nodes in a distributed antenna system (DAS).

[0307] Other examples of network nodes include multiple transmission point (multi-TRP) 5G access nodes, multi-standard radio (MSR) equipment such as MSR BSs, network controllers such as radio network controllers (RNCs) or base station controllers (BSCs), base transceiver stations (BTSs), transmission points, transmission nodes, multi-cell / multicast coordination entities (MCEs), Operation and Maintenance (O&M) nodes, Operations Support System (OSS) nodes, Self-Organizing Network (SON) nodes, positioning nodes (e.g., Evolved Serving Mobile Location Centers (E-SMLCs)), and / or Minimization of Drive Tests (MDTs).

[0308] The network node 1300 includes a processing circuitry 1302, a memory 1304, a communication interface 1306, and a power source 1308. The network node 1300 may be composed of multiple physically separate components (e.g., a NodeB component and a RNC component, or a BTS component and a BSC component, etc.), which may each have their own respective components. In certain scenarios in which the network node 1300 comprises multiple separate components (e.g., BTS and BSC components), one or more of the separate components may be shared among several network nodes. For example, a single RNC may control multiple NodeBs. In such a scenario, each unique NodeB and RNC pair, may in some instances be considered a single separate network node. In some embodiments, the network node 1300 may be configured to support multiple radio access technologies (RATs). In such embodiments, some components may be duplicated (e.g., separate memory 1304 for different RATs) and some components may be reused (e.g., a same antenna 1310 may be shared by different RATs). The network node 1300 may also include multiple sets of the various illustrated components for different wireless technologies integrated into network node 1300, for example GSM, WCDMA, LTE, NR, WiFi, Zigbee, Z-wave, LoRaWAN, Radio Frequency Identification (RFID) or Bluetooth wireless technologies. These wireless technologies may be integrated into the same or different chip or set of chips and other components within network node 1300.

[0309] The processing circuitry 1302 may comprise a combination of one or more of a microprocessor, controller, microcontroller, central processing unit, digital signal processor, application-specific integrated circuit, field programmable gate array, or any other suitable computing device, resource, or combination of hardware, software and / or encoded logic operable to provide, either alone or in conjunction with other network node 1300 components, such as the memory 1304, to provide network node 1300 functionality.

[0310] In some embodiments, the processing circuitry 1302 includes a system on a chip (SOC). In some embodiments, the processing circuitry 1302 includes one or more of radio frequency (RF) transceiver circuitry 1312 and baseband processing circuitry 1314. In some embodiments, the radio frequency (RF) transceiver circuitry 1312 and the baseband processing circuitry 1314 may be on separate chips (or sets of chips), boards, or units, such as radio units and digital units. In alternative embodiments, part or all of RF transceiver circuitry 1312 and baseband processing circuitry 1314 may be on the same chip or set of chips, boards, or units.

[0311] The memory 1304 may comprise any form of volatile or non-volatile computer-readable memory including, without limitation, persistent storage, solid-state memory, remotely mounted memory, magnetic media, optical media, random access memory (RAM), read-only memory (ROM), mass storage media (for example, a hard disk), removable storage media (for example, a flash drive, a Compact Disk (CD) or a Digital Video Disk (DVD)), and / or any other volatile or non-volatile, non-transitory device-readable and / or computer-executable memory devices that store information, data, and / or instructions that may be used by the processing circuitry 1302. The memory 1304 may store any suitable instructions, data, or information, including a computer program, software, an application including one or more of logic, rules, code, tables, and / or other instructions capable of being executed by the processing circuitry 1302 and utilized by the network node 1300. The memory 1304 may be used to store any calculations made by the processing circuitry 1302 and / or any data received via the communication interface 1306. In some embodiments, the processing circuitry 1302 and memory 1304 is integrated.

[0312] The communication interface 1306 is used in wired or wireless communication of signaling and / or data between a network node, access network, and / or UE. As illustrated, the communication interface 1306 comprises port(s) / terminal(s) 1316 to send and receive data, for example to and from a network over a wired connection. The communication interface 1306 also includes radio front-end circuitry 1318 that may be coupled to, or in certain embodiments a part of, the antenna 1310. Radio front-end circuitry 1318 comprises filters 1320 and amplifiers 1322. The radio front-end circuitry 1318 may be connected to an antenna 1310 and processing circuitry 1302. The radio front-end circuitry may be configured to condition signals communicated between antenna 1310 and processing circuitry 1302. The radio front-end circuitry 1318 may receive digital data that is to be sent out to other network nodes or UEs via a wireless connection. The radio front-end circuitry 1318 may convert the digital data into a radio signal having the appropriate channel and bandwidth parameters using a combination of filters 1320 and / or amplifiers 1322. The radio signal may then be transmitted via the antenna 1310. Similarly, when receiving data, the antenna 1310 may collect radio signals which are then converted into digital data by the radio front-end circuitry 1318. The digital data may be passed to the processing circuitry 1302. In other embodiments, the communication interface may comprise different components and / or different combinations of components.

[0313] In certain alternative embodiments, the network node 1300 does not include separate radio front-end circuitry 1318, instead, the processing circuitry 1302 includes radio front-end circuitry and is connected to the antenna 1310. Similarly, in some embodiments, all or some of the RF transceiver circuitry 1312 is part of the communication interface 1306. In still other embodiments, the communication interface 1306 includes one or more ports or terminals 1316, the radio front-end circuitry 1318, and the RF transceiver circuitry 1312, as part of a radio unit (not shown), and the communication interface 1306 communicates with the baseband processing circuitry 1314, which is part of a digital unit (not shown).

[0314] The antenna 1310 may include one or more antennas, or antenna arrays, configured to send and / or receive wireless signals. The antenna 1310 may be coupled to the radio front-end circuitry 1318 and may be any type of antenna capable of transmitting and receiving data and / or signals wirelessly. In certain embodiments, the antenna 1310 is separate from the network node 1300 and connectable to the network node 1300 through an interface or port.

[0315] The antenna 1310, communication interface 1306, and / or the processing circuitry 1302 may be configured to perform any receiving operations and / or certain obtaining operations described herein as being performed by the network node. Any information, data and / or signals may be received from a UE, another network node and / or any other network equipment. Similarly, the antenna 1310, the communication interface 1306, and / or the processing circuitry 1302 may be configured to perform any transmitting operations described herein as being performed by the network node. Any information, data and / or signals may be transmitted to a UE, another network node and / or any other network equipment.

[0316] The power source 1308 provides power to the various components of network node 1300 in a form suitable for the respective components (e.g., at a voltage and current level needed for each respective component). The power source 1308 may further comprise, or be coupled to, power management circuitry to supply the components of the network node 1300 with power for performing the functionality described herein. For example, the network node 1300 may be connectable to an external power source (e.g., the power grid, an electricity outlet) via an input circuitry or interface such as an electrical cable, whereby the external power source supplies power to power circuitry of the power source 1308. As a further example, the power source 1308 may comprise a source of power in the form of a battery or battery pack which is connected to, or integrated in, power circuitry. The battery may provide backup power should the external power source fail.

[0317] Embodiments of the network node 1300 may include additional components beyond those shown in Figure 27 for providing certain aspects of the network node’s functionality, including any of the functionality described herein and / or any functionality necessary to support the subject matter described herein. For example, the network node 1300 may include user interface equipment to allow input of information into the network node 1300 and to allow output of information from the network node 1300. This may allow a user to perform diagnostic, maintenance, repair, and other administrative functions for the network node 1300.

[0318] Figure 28 is a block diagram of a host 1400, which may be an embodiment of the host 1116 of Figure 25, in accordance with various aspects described herein. As used herein, the host 1400 may be or comprise various combinations hardware and / or software, including a standalone server, a blade server, a cloud-implemented server, a distributed server, a virtual machine, container, or processing resources in a server farm. The host 1400 may provide one or more services to one or more UEs.

[0319] The host 1400 includes processing circuitry 1402 that is operatively coupled via a bus 1404 to an input / output interface 1406, a network interface 1408, a power source 1410, and a memory 1412. Other components may be included in other embodiments. Features of these components may be substantially similar to those described with respect to the devices of previous figures, such as Figures 26 and 27, such that the descriptions thereof are generally applicable to the corresponding components of host 1400.

[0320] The memory 1412 may include one or more computer programs including one or more host application programs 1414 and data 1416, which may include user data, e.g., data generated by a UE for the host 1400 or data generated by the host 1400 for a UE. Embodiments of the host 1400 may utilize only a subset or all of the components shown. The host application programs 1414 may be implemented in a container-based architecture and may provide support for video codecs (e.g., Versatile Video Coding (WC), High Efficiency Video Coding (HEVC), Advanced Video Coding (AVC), MPEG, VP9) and audio codecs (e.g., FLAC, Advanced Audio Coding (AAC), MPEG, G.711), including transcoding for multiple different classes, types, or implementations of UEs (e.g., handsets, desktop computers, wearable display systems, heads-up display systems). The host application programs 1414 may also provide for user authentication and licensing checks and may periodically report health, routes, and content availability to a central node, such as a device in or on the edge of a core network. Accordingly, the host 1400 may select and / or indicate a different host for over-the-top services for a UE. The host application programs 1414 may support various protocols, such as the HTTP Live Streaming (HLS) protocol, Real-Time Messaging Protocol (RTMP), Real-Time Streaming Protocol (RTSP), Dynamic Adaptive Streaming over HTTP (MPEG-DASH), etc.

[0321] Figure 29 is a block diagram illustrating a virtualization environment 1500 in which functions implemented by some embodiments may be virtualized. In the present context, virtualizing means creating virtual versions of apparatuses or devices which may include virtualizing hardware platforms, storage devices and networking resources. As used herein, virtualization can be applied to any device described herein, or components thereof, and relates to an implementation in which at least a portion of the functionality is implemented as one or more virtual components. Some or all of the functions described herein may be implemented as virtual components executed by one or more virtual machines (VMs) implemented in one or more virtual environments 1500 hosted by one or more of hardware nodes, such as a hardware computing device that operates as a network node, UE, core network node, or host. Further, in embodiments in which the virtual node does not require radio connectivity (e.g., a core network node or host), then the node may be entirely virtualized. In some embodiments, the virtualization environment 1500 includes components defined by the O-RAN Alliance, such as an O-Cloud environment orchestrated by a Service Management and Orchestration Framework via an 0-2 interface.

[0322] Applications 1502 (which may alternatively be called software instances, virtual appliances, network functions, virtual nodes, virtual network functions, etc.) are run in the virtualization environment Q400 to implement some of the features, functions, and / or benefits of some of the embodiments disclosed herein.

[0323] Hardware 1504 includes processing circuitry, memory that stores software and / or instructions executable by hardware processing circuitry, and / or other hardware devices as described herein, such as a network interface, input / output interface, and so forth. Software may be executed by the processing circuitry to instantiate one or more virtualization layers 1506 (also referred to as hypervisors or virtual machine monitors (VMMs)), provide VMs 1508A and 1508B (one or more of which may be generally referred to as VMs 1508), and / or perform any of the functions, features and / or benefits described in relation with some embodiments described herein. The virtualization layer 1506 may present a virtual operating platform that appears like networking hardware to the VMs 1508.

[0324] The VMs 1508 comprise virtual processing, virtual memory, virtual networking or interface and virtual storage, and may be run by a corresponding virtualization layer 1506. Different embodiments of the instance of a virtual appliance 1502 may be implemented on one or more of VMs 1508, and the implementations may be made in different ways. Virtualization of the hardware is in some contexts referred to as network function virtualization (NFV). NFV may be used to consolidate many network equipment types onto industry standard high volume server hardware, physical switches, and physical storage, which can be located in data centers, and customer premise equipment.

[0325] In the context of NFV, a VM 1508 may be a software implementation of a physical machine that runs programs as if they were executing on a physical, non-virtualized machine. Each of the VMs 1508, and that part of hardware 1504 that executes that VM, be it hardware dedicated to that VM and / or hardware shared by that VM with others of the VMs, forms separate virtual network elements. Still in the context of NFV, a virtual network function is responsible for handling specific network functions that run in one or more VMs 1508 on top of the hardware 1504 and corresponds to the application 1502.

[0326] Hardware 1504 may be implemented in a standalone network node with generic or specific components. Hardware 1504 may implement some functions via virtualization. Alternatively, hardware 1504 may be part of a larger cluster of hardware (e.g. such as in a data center or CPE) where many hardware nodes work together and are managed via management and orchestration 1510, which, among others, oversees lifecycle management of applications 1502. In some embodiments, hardware 1504 is coupled to one or more radio units that each include one or more transmitters and one or more receivers that may be coupled to one or more antennas. Radio units may communicate directly with other hardware nodes via one or more appropriate network interfaces and may be used in combination with the virtual components to provide a virtual node with radio capabilities, such as a radio access node or a base station. In some embodiments, some signaling can be provided with the use of a control system 1512 which may alternatively be used for communication between hardware nodes and radio units.

[0327] Figure 30 shows a communication diagram of a host 1602 communicating via a network node 1604 with a UE 1606 over a partially wireless connection in accordance with some embodiments. Example implementations, in accordance with various embodiments, of the UE (such as a UE 1112A of Figure 25 and / or UE 1200 of Figure 26), network node (such as network node 1110A of Figure 25 and / or network node 1300 of Figure 27), and host (such as host 1116 of Figure 25 and / or host 1400 of Figure 28) discussed in the preceding paragraphs will now be described with reference to Figure 30.

[0328] Like host 1400, embodiments of host 1602 include hardware, such as a communication interface, processing circuitry, and memory. The host 1602 also includes software, which is stored in or accessible by the host 1602 and executable by the processing circuitry. The software includes a host application that may be operable to provide a service to a remote user, such as the UE 1606 connecting via an over-the-top (OTT) connection 1650 extending between the UE 1606 and host 1602. In providing the service to the remote user, a host application may provide user data which is transmitted using the OTT connection 1650.

[0329] The network node 1604 includes hardware enabling it to communicate with the host 1602 and UE 1606. The connection 1660 may be direct or pass through a core network (like core network 1106 of Figure 25) and / or one or more other intermediate networks, such as one or more public, private, or hosted networks. For example, an intermediate network may be a backbone network or the Internet.

[0330] The UE 1606 includes hardware and software, which is stored in or accessible by UE 1606 and executable by the UE’s processing circuitry. The software includes a client application, such as a web browser or operator-specific “app” that may be operable to provide a service to a human or non-human user via UE 1606 with the support of the host 1602. In the host 1602, an executing host application may communicate with the executing client application via the OTT connection 1650 terminating at the UE 1606 and host 1602. In providing the service to the user, the UE's client application may receive request data from the host's host application and provide user data in response to the request data. The OTT connection 1650 may transfer both the request data and the user data. The UE's client application may interact with the user to generate the user data that it provides to the host application through the OTT connection 1650.

[0331] The OTT connection 1650 may extend via a connection 1660 between the host 1602 and the network node 1604 and via a wireless connection 1670 between the network node 1604 and the UE 1606 to provide the connection between the host 1602 and the UE 1606. The connection 1660 and wireless connection 1670, over which the OTT connection 1650 may be provided, have been drawn abstractly to illustrate the communication between the host 1602 and the UE 1606 via the network node 1604, without explicit reference to any intermediary devices and the precise routing of messages via these devices.

[0332] As an example of transmitting data via the OTT connection 1650, in step 1608, the host 1602 provides user data, which may be performed by executing a host application. In some embodiments, the user data is associated with a particular human user interacting with the UE 1606. In other embodiments, the user data is associated with a UE 1606 that shares data with the host 1602 without explicit human interaction. In step 1610, the host 1602 initiates a transmission carrying the user data towards the UE 1606. The host 1602 may initiate the transmission responsive to a request transmitted by the UE 1606. The request may be caused by human interaction with the UE 1606 or by operation of the client application executing on the UE 1606. The transmission may pass via the network node 1604, in accordance with the teachings of the embodiments described throughout this disclosure. Accordingly, in step 1612, the network node 1604 transmits to the UE 1606 the user data that was carried in the transmission that the host 1602 initiated, in accordance with the teachings of the embodiments described throughout this disclosure. In step 1614, the UE 1606 receives the user data carried in the transmission, which may be performed by a client application executed on the UE 1606 associated with the host application executed by the host 1602.

[0333] In some examples, the UE 1606 executes a client application which provides user data to the host 1602. The user data may be provided in reaction or response to the data received from the host 1602. Accordingly, in step 1616, the UE 1606 may provide user data, which may be performed by executing the client application. In providing the user data, the client application may further consider user input received from the user via an input / output interface of the UE 1606. Regardless of the specific manner in which the user data was provided, the UE 1606 initiates, in step 1618, transmission of the user data towards the host 1602 via the network node 1604. In step 1620, in accordance with the teachings of the embodiments described throughout this disclosure, the network node 1604 receives user data from the UE 1606 and initiates transmission of the received user data towards the host 1602. In step 1622, the host 1602 receives the user data carried in the transmission initiated by the UE 1606.

[0334] One or more of the various embodiments improve the performance of OTT services provided to the UE 1606 using the OTT connection 1650, in which the wireless connection 1670 forms the last segment. More precisely, the teachings of these embodiments may improve the low latency processing on sender and receiver devices, and thereby provide benefits such as low bitrate uplink transmission of depth and texture bitstreams.

[0335] In an example scenario, factory status information may be collected and analyzed by the host 1602. As another example, the host 1602 may process audio and video data which may have been retrieved from a UE for use in creating maps. As another example, the host 1602 may collect and analyze real-time data to assist in controlling vehicle congestion (e.g., controlling traffic lights). As another example, the host 1602 may store surveillance video uploaded by a UE. As another example, the host 1602 may store or control access to media content such as video, audio, VR or AR which it can broadcast, multicast or unicast to UEs. As other examples, the host 1602 may be used for energy pricing, remote control of non-time critical electrical load to balance power generation needs, location services, presentation services (such as compiling diagrams etc. from data collected from remote devices), or any other function of collecting, retrieving, storing, analyzing and / or transmitting data.

[0336] In some examples, a measurement procedure may be provided for the purpose of monitoring data rate, latency and other factors on which the one or more embodiments improve. There may further be an optional network functionality for reconfiguring the OTT connection 1650 between the host 1602 and UE 1606, in response to variations in the measurement results. The measurement procedure and / or the network functionality for reconfiguring the OTT connection may be implemented in software and hardware of the host 1602 and / or UE 1606. In some embodiments, sensors (not shown) may be deployed in or in association with other devices through which the OTT connection 1650 passes; the sensors may participate in the measurement procedure by supplying values of the monitored quantities exemplified above, or supplying values of other physical quantities from which software may compute or estimate the monitored quantities. The reconfiguring of the OTT connection 1650 may include message format, retransmission settings, preferred routing etc.; the reconfiguring need not directly alter the operation of the network node 1604. Such procedures and functionalities may be known and practiced in the art. In certain embodiments, measurements may involve proprietary UE signaling that facilitates measurements of throughput, propagation times, latency and the like, by the host 1602. The measurements may be implemented in that software causes messages to be transmitted, in particular empty or ‘dummy’ messages, using the OTT connection 1650 while monitoring propagation times, errors, etc.

[0337] Although the computing devices described herein (e.g., UEs, network nodes, hosts) may include the illustrated combination of hardware components, other embodiments may comprise computing devices with different combinations of components. It is to be understood that these computing devices may comprise any suitable combination of hardware and / or software needed to perform the tasks, features, functions and methods disclosed herein. Determining, calculating, obtaining or similar operations described herein may be performed by processing circuitry, which may process information by, for example, converting the obtained information into other information, comparing the obtained information or converted information to information stored in the network node, and / or performing one or more operations based on the obtained information or converted information, and as a result of said processing making a determination. Moreover, while components are depicted as single boxes located within a larger box, or nested within multiple boxes, in practice, computing devices may comprise multiple different physical components that make up a single illustrated component, and functionality may be partitioned between separate components. For example, a communication interface may be configured to include any of the components described herein, and / or the functionality of the components may be partitioned between the processing circuitry and the communication interface. In another example, non-computationally intensive functions of any of such components may be implemented in software or firmware and computationally intensive functions may be implemented in hardware.

[0338] In certain embodiments, some or all of the functionality described herein may be provided by processing circuitry executing instructions stored on in memory, which in certain embodiments may be a computer program product in the form of a non-transitory computer- readable storage medium. In alternative embodiments, some or all of the functionality may be provided by the processing circuitry without executing instructions stored on a separate or discrete device-readable storage medium, such as in a hard-wired manner. In any of those particular embodiments, whether executing instructions stored on a non-transitory computer- readable storage medium or not, the processing circuitry can be configured to perform the described functionality. The benefits provided by such functionality are not limited to the processing circuitry alone or to other components of the computing device, but are enjoyed by the computing device as a whole, and / or by end users and a wireless network generally.

[0339] It should be noted that Figures 2 and 10A-10C show slightly different content that that of Figures 5A and 7A. Particularly, the subject in images 2 and 10A-10C is wearing a gray t-shirt while the subject in Figures 5A and 7A is wearing a black sweater. However, the particular clothing worn by the subjects in the figures is not germane to the present embodiments. Rather, all figures are for illustrative purposes only. The present embodiments may, of course, be carried out in other ways than those specifically set forth herein without departing from characteristics described herein. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, and all changes coming within the meaning and equivalency range of the appended claims are intended to be embraced therein.

Claims

CLAIMSWhat is claimed is:1 . A method (260), implemented by a sending device (40), for conveying video data for a volumetric video representation to a receiving device (60, 240), the method comprising: obtaining a depth image from a depth image source; partitioning (270) the depth image into a foreground section and a background section; generating (272) an occupancy map for the foreground and background sections, wherein the occupancy map is generated to comprise a first sample value corresponding to the foreground section and a second sample value corresponding to the background section; encoding (276) the depth image and the occupancy map into a bitstream; and sending (278) the bitstream over a communications network to the receiving device.

2. The method of claim 1 , wherein partitioning a depth image into a foreground section and a background section comprises: sampling (282) the depth image to generate a plurality of depth image sample values; comparing (284) a depth image sample value to a threshold sample value; and assigning (286) the depth image sample value to one of the foreground section and the background section based on a result of the comparing.

3. The method of any of claims 1-2, wherein: the depth image sample value is assigned to: the foreground section when the depth image sample value is equal to or exceeds the threshold sample value; and the background section when the depth image sample value is less than the threshold sample value; or the depth image sample value is assigned to: the background section when the depth image sample value is equal to or exceeds the threshold sample value; and the foreground section when the depth image sample value is less than the threshold sample value.

4. The method of any of claims 2-3, wherein the threshold sample value is: set based on one or more parameters associated with the depth image; or set based on a position of one or more of the depth image sample values in the depth image; or adaptively selected based on at least one of: a distribution of the depth image sample values of the depth image; and a coverage of the depth image sample values of the depth image; and wherein the one or more parameters associated with the depth image comprise at least one of: a parameter of a camera that captured the depth image; and a parameter of an environment associated with the depth image.

5. The method of any of claims 1-4, wherein the depth image is partitioned into a plurality of foreground sections and a background section, and wherein each of the plurality of foreground sections comprises a unique sample value.

6. The method of any of claims 1-5, further comprising: obtaining (292) a texture image from a texture image source, wherein the texture image corresponds to content in the depth image; and encoding (294) the texture image into the bitstream.

7. The method of any of claims 1-6, further comprising removing (302) the background section from one or both of the depth image and the texture image based on one or both of the foreground section and the background section of the occupancy map prior to the encoding.

8. The method of any of claims 1-7, wherein removing the background section from one or both of the depth image and the texture image comprises assigning (304) a predetermined value to one or more samples in the background section of the one or both of the depth image and the texture image.

9. The method of any of claims 1-8, further comprising padding (322) the foreground section of one or both of the depth image and the texture image, wherein the padding comprises: assigning (324) a value of a sample in the foreground section of the one or both of the depth image and the texture image to a sample in the background section of the one or both of the depth image and the texture image, wherein the sample in the foreground section of the one or both of the depth image and the texture image is in a position that is proximate to a position of the sample in the background section of the one or both of the depth image and the texture image; orpartitioning (326) the one or both of the depth image and the texture image into one or more blocks, wherein each block comprises two or more samples and corresponds to a coding unit (CU) of a video codec used to encode the one or both of the depth image and the texture image into the bitstream; and for each of the two or more samples in the one or more blocks: determining (328) whether a block of the sample is in both the foreground section and the background section of the one or both of the depth image and the texture image; determining (330) whether the sample is in the background section of the one or both of the depth image and the texture image; and responsive to determining that the block of the sample is in both the foreground section and the background section of the one or both of the depth image and the texture image and that the sample is in the background section of the one or both of the depth image and the texture image, assigning (332) a padding value to the sample of the block of the background section, wherein the padding value is derived from one or more sample values of the foreground section that neighbors the sample of the block of the background section.

10. The method of claim 9, wherein: the padding value is extrapolated from the one or more sample values of the foreground section; or the padding value is an average value or a median value of a plurality of samples of the foreground section, and wherein the block comprises samples from both the foreground section and the background section.11 . The method of any of claims 1-10, further comprising converting (342) the depth image from a current bit depth to a lower bit depth prior to the encoding, wherein the lower bit depth comprises fewer bits than the current bit depth.

12. The method of any of claims 1-11 , wherein the depth image comprises a plurality of depth values, each depth value indicating a distance from a camera that captured the depth image and being bounded by a minimum depth value and a maximum depth value, and wherein converting the depth image from a current bit depth to a lower bit depth comprises: selecting (344) a range of depth values from the plurality of depth values; and converting (346) the range of depth values to lower depth values by scaling the depth values in the range of depth values to fit within the lower bit depth.

13. The method of any of claims 1-12, wherein the bitstream comprises one of: a single bitstream, wherein the texture image, the depth image, and the occupancy map are encoded into the single bitstream; first and second separate bitstreams, wherein two of the texture image, the depth image, and the occupancy map are encoded into the first bitstream and the other of the texture image, the depth image, and the occupancy map are encoded into the second bitstream; and first, second, and third separate bitstreams, wherein the depth image is encoded into the first bitstream, the occupancy map is encoded into the second bitstream, and the texture image is encoded into the third bitstream.

14. The method of any of claims 1-13, wherein the depth image and the occupancy map are packed into a same picture for encoding using a multi-component color format, wherein the depth image is signaled in a first component of the multi-component color format, and wherein the occupancy map is signaled in a second component of the multi-component color format.

15. The method of any of claims 1-14, wherein the multi-component color format comprises a YUV format, and wherein: the depth image is signaled in the luminance component of the YUV format, and the occupancy map is signaled in at least one of the color components of the YUV format; or the occupancy map is signaled in the luminance component of the YUV format, and the depth image is signaled in at least one of the color components of the YUV format; or the occupancy map is signaled in a first color component of the YUV format, and the depth image is signaled in a second color component of the YUV format.

16. The method of any of claims 1-15, wherein the depth image is encoded with a lower quality than the occupancy map.

17. The method of any of claims 1-16, further comprising sending (296) metadata to the receiving device, wherein the metadata comprises one or more parameters intrinsic to a camera that captured one or both of the depth image and the texture image.

18. A method (350), implemented by a receiving device (60, 240), for receiving video data for a volumetric video representation from a sending device (40), the method comprising: receiving (352), from the sending device, an encoded bitstream comprising a depth image and occupancy map, wherein the occupancy map comprises a first sample value corresponding to a foreground section of the depth image and a second sample value corresponding to a background section of the depth image;decoding (356) the depth image and the occupancy map from the encoded bitstream; and filtering (362) the depth image based on the occupancy map.

19. The method of claim 18, further comprising receiving (354), from the sending device, a texture image, wherein the texture image corresponds to content in the depth image and is encoded in the bitstream.

20. The method of any of claims 18-19, wherein filtering the depth image based on the occupancy map comprises modifying (372) depth values in the depth image according to the occupancy map, wherein the depth values correspond to the background section of the depth image according to the occupancy map.

21. The method of claim 20, wherein modifying depth values in the depth image according to the occupancy map comprises one of: assigning (374) a sample value to each sample in the depth image that corresponds to the background section of the depth image as indicated in the occupancy map; and assigning (376) a sample value to each sample in the depth image that does not correspond to the foreground section of the depth image as indicated in the occupancy map.

22. The method of any of claims 18-21 , further comprising converting (378) the depth image from a current bit depth to a higher bit depth, wherein the higher bit depth comprises more bits than the current bit depth.

23. The method of any of claims 18-22, further comprising receiving (380), from the sending device, metadata comprising one or more parameters intrinsic to a camera that captured one or both of the depth image and the texture image.

24. The method of any of claims 18-23, further comprising: generating (364) a volumetric video based on the texture image and the depth image, wherein the volumetric video comprises one of a mesh and a point cloud; and rendering (366) the volumetric video to a display device.

25. The method of any of claims 18-24, wherein the receiving device is a first receiving device in a communications network, and wherein the method further comprises: generating (390) a fourth bitstream comprising a representation of the one of the mesh and the point cloud; and sending (392) the fourth bitstream to a second receiving device.

26. A sending device (40) for conveying video data for a volumetric video representation to a receiving device (60, 240), the sending device being configured to: obtain a depth image from a depth image source; partition (270) the depth image into a foreground section and a background section; generate (372) an occupancy map for the foreground and background sections, wherein the occupancy map is generated to comprise a first sample value corresponding to the foreground section and a second sample value corresponding to the background section; encode (276) the depth image and the occupancy map into a bitstream; and send (278) the bitstream over a communications network to the receiving device.

27. The sending device of claim 26, further configured to perform the method of any one of claims 2-17.

28. A sending device (40) for conveying video data for a volumetric video representation to a receiving device (60, 240), the sending device comprising: communication circuitry (406) for communicating with the receiving device; and processing circuitry (400) configured to: obtain a depth image from a depth image source; partition (270) the depth image into a foreground section and a background section; generate (372) an occupancy map for the foreground and background sections, wherein the occupancy map is generated to comprise a first sample value corresponding to the foreground section and a second sample value corresponding to the background section; encode (276) the depth image and the occupancy map into a bitstream; and send (278) the bitstream over a communications network (32) to the receiving device.

29. The sending device of claim 28, wherein the processing circuitry is further configured to perform the method of any one of claims 2-17.

30. A computer program (504) comprising executable instructions that, when executed by a processing circuit (400) in a sending device (40), causes the sending device to perform any one of the methods of claims 1-17.31 . A carrier containing a computer program of claim 30, wherein the carrier is one of an electronic signal, optical signal, radio signal, or computer readable storage medium.

32. A non-transitory computer-readable storage medium (402) containing a computer program (404) comprising executable instructions that, when executed by a processing circuit (400) in a sending device (40) causes the sending device to perform any one of the methods of claims 1- 17.

33. A receiving device (60, 240) for receiving video data for a volumetric video representation from a sending device (40), the receiving device being configured to: receive (352), from the sending device, an encoded bitstream comprising a depth image and occupancy map, wherein the occupancy map comprises a first sample value corresponding to a foreground section of the depth image and a second sample value corresponding to a background section of the depth image; decode (356) the depth image and the occupancy map from the encoded bitstream; and filter (362) the depth image based on the occupancy map.

34. The receiving device of claim 33, further configured to perform the method of any one of claims 18-25.

35. A receiving device (60, 240) for receiving video data for a volumetric video representation from a sending device (40), the receiving device comprising: communication circuitry (506, 606) for communicating with the sending device; and processing circuitry (500, 600) configured to: receive (352), from the sending device, an encoded bitstream comprising a depth image and occupancy map, wherein the occupancy map comprises a first sample value corresponding to a foreground section of the depth image and a second sample value corresponding to a background section of the depth image; decode (356) the depth image and the occupancy map from the encoded bitstream; and filter (362) the depth image based on the occupancy map.

36. The receiving device of claim 35, wherein the processing circuitry is further configured to perform the method of any one of claims 18-25.

37. A computer program (504, 604) comprising executable instructions that, when executed by a processing circuit (500, 600) in a receiving device (60, 240), causes the receiving device to perform any one of the methods of claims 18-25.

38. A carrier containing a computer program of claim 37, wherein the carrier is one of an electronic signal, optical signal, radio signal, or computer readable storage medium.

39. A non-transitory computer-readable storage medium (502, 602) containing a computer program (504, 604) comprising executable instructions that, when executed by a processing circuit (500, 600) in a receiving device (60, 240) causes the receiving device to perform any one of the methods of claims 18-25.

40. A communication system (30) for conveying video data for a volumetric video representation, the communication system comprising: a sending device (40) and a receiving device (60, 240); wherein the sending device is configured to: obtain a depth image from a depth image source; partition (270) the depth image into a foreground section and a background section; generate (272) an occupancy map for the foreground and background sections, wherein the occupancy map is generated to comprise a first sample value corresponding to the foreground section and a second sample value corresponding to the background section; encode (276) the depth image and the occupancy map into a bitstream; and send (278) the bitstream over a communications network to the receiving device; and wherein the receiving device (60, 240) is configured to: receive (352), from the sending device, an encoded bitstream comprising a depth image and occupancy map, wherein the occupancy map comprises a first sample value corresponding to a foreground section of the depth image and a second sample value corresponding to a background section of the depth image; decode (356) the depth image and the occupancy map from the encoded bitstream; and filter (362) the depth image based on the occupancy map.

Images