Point Cloud Optimization Using Instance Segmentation

Abstract

Aspects of the present disclosure provide methods and apparatus for point cloud processing. In some examples, the apparatus for point cloud processing includes a processing circuit. For example, the processing circuit acquires point cloud data corresponding to a point cloud in a three-dimensional (3D) space and projects the point cloud in the 3D space onto one or more two-dimensional (2D) planes to generate one or more images. The processing circuit generates a pixel-wise mask of an object instance in the point cloud according to the one or more images. The pixel-wise mask includes a first pixel associated with a first object instance in the point cloud. The processing circuit processes the point cloud based on the pixel-wise mask, and a portion of the point cloud corresponding to the first pixel in the pixel-wise mask is processed based on one or more processing parameters determined for the first object instance.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to U.S. Patent Application No. 18 / 207,614, entitled "POINT CLOUD OPTIMIZATION USING INSTANCE SEGMENTATION," filed June 8, 2023, and claims the benefit of priority to U.S. Provisional Application No. 63 / 398,669, filed August 17, 2022. The entire disclosures of the prior applications are incorporated herein by reference in their entireties.

[0002] This disclosure generally describes embodiments relating to point cloud processing. [Background technology]

[0003] The background discussion provided herein is intended to generally present the context of the present disclosure. To the extent that it is described in this Background section, the inventors' work, as well as aspects of the description that may not otherwise be admitted as prior art at the time of filing, are not admitted, expressly or impliedly, as prior art to the present disclosure.

[0004] Various techniques have been developed to capture and represent the world, such as world objects, the world's environments, and humans and animals in three-dimensional (3D) space. The 3D representation of the world can enable more immersive interaction and communication. In some examples, point clouds and meshes can be used as the 3D representation of the world. Summary of the Invention [Means for solving the problem]

[0005] Aspects of the present disclosure provide a method and apparatus for point cloud processing. In some examples, the apparatus for point cloud processing includes a processing circuit. For example, the processing circuit acquires point cloud data corresponding to a point cloud in a three-dimensional (3D) space and projects the point cloud in the 3D space onto one or more two-dimensional (2D) planes to generate one or more images. The processing circuit generates a pixel-wise mask of an object instance in the point cloud according to the one or more images. The pixel-wise mask includes a first pixel associated with a first object instance in the point cloud. The processing circuit processes the point cloud based on the pixel-wise mask, and a portion of the point cloud corresponding to the first pixel in the pixel-wise mask is processed based on one or more processing parameters determined for the first object instance.

[0006] In some examples, the one or more images are input to a convolutional neural network model trained to generate a pixel-wise mask of the object instance. In some examples, the one or more images are input to a non-neural network based logical model configured to generate a pixel-wise mask of the object instance.

[0007] In some examples, the point cloud includes points representing a person. The processing circuitry generates a pixel-wise mask including a plurality of sub-masks associated with facial and physical features of the person, respectively.

[0008] In some examples, the processing circuitry determines, according to the pre-training data, parameters of each of the one or more virtual cameras associated with the one or more 2D planes for projection of the point cloud.

[0009] In some examples, the processing circuitry determines first parameters for voxelizing the first object instance and voxelizes a portion of the point cloud corresponding to a first pixel in the pixel-wise mask according to the first parameters.

[0010] In some examples, the processing circuit generates a scene graph associated with the point cloud based on the pixel-wise mask, the scene graph including at least a first scene element of the first object instance.

[0011] In some examples, the processing circuit processes the point cloud using a pixel-wise mask by a video-based point cloud compression (V-PCC) scheme. In one example, the processing circuit divides the point cloud into a plurality of segments according to the pixel-wise mask having a plurality of sub-masks corresponding to the plurality of segments. Furthermore, the processing circuit packs each of the plurality of segments into a geometry map and encodes the geometry map into a respective sub-stream. In another example, the processing circuit generates a 2D patch for each of the plurality of segments based on the pixel-wise mask, and the 2D patch of the segment includes geometric information and semantic information of the 2D patch. In another example, the processing circuit determines a quantization parameter for encoding the portion of the point cloud based on the pixel-wise mask.

[0012] In some examples, the processing circuit processes the point cloud using a pixel-wise mask by a geometry-based point cloud compression (G-PCC) scheme. In one example, the processing circuit divides the point cloud into a plurality of slices based on the pixel-wise mask, determines encoder parameters for the plurality of slices based on respective characteristics, and encodes the plurality of slices into sub-streams based on the encoder parameters. In another example, the processing circuit determines a geometry quantization parameter for octree partitioning based on the pixel-wise mask, and performs the octree partitioning based on the geometry quantization parameter.

[0013] Aspects of the present disclosure also provide a non-transitory computer readable medium storing instructions that, when executed by a computer, cause the computer to perform any one or combination of methods for point cloud processing.

[0014] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Brief description of the drawings]

[0015] [Figure 1] 1 is a block diagram of a communication system in some examples. [Diagram 2] FIG. 1 is a block diagram of a streaming system in some examples. [Diagram 3] 1 is a block diagram of an encoder for encoding a point cloud frame in some examples. [Figure 4] FIG. 2 is a block diagram of a decoder for decoding a compressed bitstream corresponding to a point cloud frame in some examples. [Diagram 5] 1 is a block diagram of a video decoder in some examples. [Figure 6] 1 is a block diagram of a video encoder in some examples. [Figure 7] 1 is a block diagram of an encoder for encoding a point cloud frame in some examples. [Figure 8] FIG. 1 is a block diagram of a decoder for decoding a compressed bitstream carrying a point cloud frame in some examples. [Figure 9] FIG. 1 is a diagram of a computer system according to one embodiment of the present disclosure. [Figure 10] 1 is a diagram illustrating a method for generating multiple virtual camera views in some examples. [Figure 11] FIG. 2 illustrates an example of a pixel-wise mask in some examples. [Figure 12] FIG. 2 is a diagram of voxelization in some examples. [Figure 13] 1 is a flowchart outlining an example process in some examples. [Figure 14] FIG. 1 is a schematic diagram of a computer system according to some examples. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Aspects of the present disclosure provide techniques in the field of three-dimensional (3D) media processing.

[0017] Technological developments in 3D media processing, such as advances in three-dimensional (3D) capture, 3D modeling, and 3D rendering, have facilitated the ubiquitous presence of 3D media content across several platforms and devices. In one example, a baby's first steps can be captured on one continent, and media technology can enable grandparents to see (and potentially interact with) the baby on another continent and enjoy an immersive experience with the baby. According to one aspect of the present disclosure, to improve the immersive experience, 3D models are becoming increasingly sophisticated, and the creation and consumption of 3D models occupies a significant amount of data resources, such as data storage, data transmission resources, etc.

[0018] According to some aspects of the present disclosure, point clouds and meshes can be used as 3D models to represent immersive content.

[0019] A point cloud may generally refer to a collection of points in 3D space, where each point includes associated attributes such as color, material properties, texture information, intensity attributes, reflectivity attributes, motion-related attributes, modality attributes, and various other attributes. A point cloud can be used to reconstruct an object or scene as a composition of such points.

[0020] A mesh of an object (also called a mesh model) may include polygons that describe the surface of the object. Each polygon may be defined by the vertices of the polygon in 3D space and information about how the vertices are connected to the polygon. The information about how the vertices are connected is called connectivity information. In some examples, the mesh may also include attributes, such as color, normals, etc., associated with the vertices. According to some aspects of the present disclosure, some coding tools for point cloud compression (PCC) may be used for mesh compression.

[0021] Using the point cloud, an object or scene can be reconstructed as a composite of points. The points can be captured using multiple cameras, depth sensors or lidar in various settings, and may consist of thousands to billions of points to realistically represent the reconstructed scene or object. A patch may generally refer to a contiguous subset of the surface described by the point cloud. In one example, a patch includes points that include surface normal vectors that are offset from each other by less than a threshold amount.

[0022] PCC can be performed according to various schemes, such as a geometry-based scheme called G-PCC, a video coding-based scheme called V-PCC, etc. According to some aspects of the present disclosure, G-PCC is a purely geometry-based approach that directly encodes 3D geometry and does not share much with video coding, while V-PCC is heavily based on video coding. For example, V-PCC can map points of a 3D cloud to pixels of a 2D grid (image). The V-PCC scheme can utilize a general-purpose video codec for point cloud compression. The PCC codec (encoder / decoder) in the present disclosure can be a G-PCC codec (encoder / decoder) or a V-PCC codec.

[0023] According to one aspect of the present disclosure, the V-PCC scheme can compress the geometry, occupancy and texture of the point cloud as three separate video sequences using an existing video codec. Additional metadata required to interpret the three video sequences is compressed separately. A small portion of the overall bitstream is the metadata, which in one example can be efficiently encoded / decoded using a software implementation. The majority of the information is handled by the video codec.

[0024] FIG. 1 shows a block diagram of a communication system (100) in some examples. The communication system (100) includes a plurality of terminal devices capable of communicating with each other, for example, via a network (150). For example, the communication system (100) includes a pair of terminal devices (110) and (120) interconnected via the network (150). In the example of FIG. 1, a first pair of terminal devices (110) and (120) may perform unidirectional transmission of point cloud data. For example, the terminal device (110) may compress a point cloud (e.g., points representing structures) captured by a sensor (105) connected to the terminal device (110). The compressed point cloud may be transmitted to another terminal device (120) via the network (150), for example, in the form of a bit stream. The terminal device (120) may receive the compressed point cloud from the network (150), decompress the bit stream to reconstruct the point cloud, and appropriately display the reconstructed point cloud. One-way data transmission may be common, such as in media serving applications.

[0025] In the example of FIG. 1, terminal devices (110) and (120) may be shown as a server and a personal computer, although the principles of the present disclosure may not be so limited. Embodiments of the present disclosure find application in laptop computers, tablet computers, smart phones, gaming consoles, media players, and / or dedicated three-dimensional (3D) equipment. Network (150) represents any number of networks that transmit compressed point clouds between terminal devices (110) and (120). Network (150) may include, for example, wireline and / or wireless communication networks. Network (150) may exchange data over circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks, the Internet, and the like.

[0026] 2 shows a block diagram of a streaming system (200) in some examples. The streaming system (200) is a point cloud usage application. The disclosed subject matter may be equally applicable to other point cloud-enabled applications such as 3D telepresence applications, virtual reality applications, etc.

[0027] The streaming system (200) may include a capture subsystem (213). The capture subsystem (213) may include a point cloud source (201), such as a light detection and ranging (LIDAR) system, a 3D camera, a 3D scanner, a graphics generation component that generates an uncompressed point cloud in software, and the like, such as to generate an uncompressed point cloud (202). In one example, the point cloud (202) includes points captured by a 3D camera. The point cloud (202) is shown in bold to emphasize its larger amount of data compared to the compressed point cloud (204) (compressed point cloud bitstream). The compressed point cloud (204) may be generated by an electronic device (220) that includes an encoder (203) coupled to the point cloud source (201). The encoder (203) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The compressed point cloud (204) (or a bitstream of the compressed point cloud (204)), shown with thin lines to emphasize its small amount of data compared to the stream of point cloud (202), can be stored on the streaming server (205) for future use. One or more streaming client subsystems, such as the client subsystems (206) and (208) of FIG. 2, can access the streaming server (205) to retrieve copies (207) and (209) of the compressed point cloud (204). The client subsystem (206) can include a decoder (210), for example in the electronic device (230). The decoder (210) decodes the input copy of the compressed point cloud (207) and creates an output stream of a reconstructed point cloud (211) that can be rendered on a rendering device (212).

[0028] It should be noted that the electronic devices (220) and (230) may include other components (not shown). For example, the electronic device (220) may include a decoder (not shown), and the electronic device (230) may include an encoder (not shown).

[0029] In some streaming systems, the compressed point clouds (204), (207) and (209) (e.g., compressed point cloud bitstreams) may be compressed according to a particular standard. In some examples, a video coding standard is used in compressing the point clouds. Examples of such standards include High Efficiency Video Coding (HEVC), or Universal Video Coding (VVC), etc.

[0030] 3 shows a block diagram of a V-PCC encoder (300) for encoding a point cloud frame according to some embodiments. In some embodiments, the V-PCC encoder (300) can be used in the communication system (100) and the streaming system (200). For example, the encoder (203) can be configured and operated similarly to the V-PCC encoder (300).

[0031] The V-PCC encoder (300) receives a point cloud frame as an uncompressed input and generates a bitstream corresponding to the compressed point cloud frame. In some embodiments, the V-PCC encoder (300) may receive a point cloud frame from a point cloud source, such as the point cloud source (201).

[0032] In the example of FIG. 3, the V-PCC encoder (300) includes a patch generation module (306), a patch packing module (308), a geometry image generation module (310), a texture image generation module (312), a patch information module (304), an occupancy map module (314), a smoothing module (336), image padding modules (316) and (318), a group expansion module (320), video compression modules (322), (323), and (332), an auxiliary patch information compression module (338), an entropy compression module (334), and a multiplexer (324).

[0033] According to one aspect of the disclosure, the V-PCC encoder (300) converts the 3D point cloud frame into an image-based representation along with some metadata (e.g., occupancy map and patch information) used to convert the compressed point cloud into a decompressed point cloud. In some examples, the V-PCC encoder (300) converts the 3D point cloud frame into a geometry image, a texture image, and an occupancy map, and then can encode the geometry image, texture image, and occupancy map into a bitstream using video coding techniques. In general, a geometry image is a 2D image that includes pixels filled with geometry values ​​associated with points projected onto the pixels, and the pixels filled with geometry values ​​can be referred to as geometry samples. A texture image is a 2D image that includes pixels filled with texture values ​​associated with points projected onto the pixels, and the pixels filled with texture values ​​can be referred to as texture samples. An occupancy map is a 2D image that includes pixels filled with values ​​that indicate whether they are occupied or unoccupied by a patch.

[0034] The patch generation module (306) segments the point cloud into a set of patches (e.g., a patch is defined as a contiguous subset of the surface described by the point cloud) that may or may not overlap, such that each patch may be described by a depth field for a plane in 2D space. In some embodiments, the patch generation module (306) aims to decompose the point cloud into a minimum number of patches that contain smooth boundaries while minimizing the reconstruction error.

[0035] In some examples, the patch information module (304) may collect patch information indicating the size and shape of the patch, and in some examples, the patch information may be packed into an image frame and then encoded by the auxiliary patch information compression module (338) to generate compressed auxiliary patch information.

[0036] In some examples, the patch packing module (308) is configured to map the extracted patches onto a two-dimensional (2D) grid while minimizing unused space and ensuring that every M×M (e.g., 16×16) block of the grid is associated with a unique patch. Efficient patch packing can have a direct impact on compression efficiency, either by minimizing unused space or by ensuring temporal consistency.

[0037] The geometry image generation module (310) can generate a 2D geometry image associated with the geometry of the point cloud at a given patch location. The texture image generation module (312) can generate a 2D texture image associated with the texture of the point cloud at a given patch location. The geometry image generation module (310) and the texture image generation module (312) use the 3D-2D mapping calculated during the packing process to store the geometry and texture of the point cloud as images. To better handle the case where multiple points are projected onto the same sample, each patch is projected onto two images, called layers. In one example, the geometry image is represented by a W×H monochromatic frame in YUV420-8 bit format. To generate the texture image, the texture generation procedure uses the reconstructed / smoothed geometry to calculate the colors associated with the resampled points.

[0038] The occupancy map module (314) can generate an occupancy map that describes the padding information at each unit. For example, the occupancy image includes a binary map that indicates for each cell of the grid whether the cell belongs to free space or to the point cloud. In one example, the occupancy map uses binary information that describes for each pixel whether the pixel is padded or not. In another example, the occupancy map uses binary information that describes for each block of pixels whether the block of pixels is padded or not.

[0039] The occupancy map generated by the occupancy map module (314) can be compressed using lossless or lossy coding. If lossless coding is used, the entropy compression module (334) is used to compress the occupancy map. If lossy coding is used, the video compression module (332) is used to compress the occupancy map.

[0040] It should be noted that the patch packing module (308) may leave some empty space between the 2D patches packed into the image frame. The image padding modules (316) and (318) may fill the empty space (called padding) to generate an image frame that may be suitable for 2D video and image codecs. Image padding, also called background filling, may fill the unused space with redundant information. In some examples, good background filling increases the bit rate minimally while not introducing significant coding distortion around the patch boundaries.

[0041] The video compression modules (322), (323), and (332) can encode 2D images, such as padded geometry images, padded texture images, and occupancy maps, based on an appropriate video coding standard, such as HEVC or VVC. In one example, the video compression modules (322), (323), and (332) are individual components operating separately. It should be noted that in other examples, the video compression modules (322), (323), and (332) can be implemented as a single component.

[0042] In some examples, the smoothing module (336) is configured to generate a smoothed image of the reconstructed geometry image. The smoothed image can be provided to the texture image generation (312). The texture image generation (312) may then adjust the generation of the texture image based on the reconstructed geometry image. For example, if there is some distortion in the patch shape (e.g., geometry) during encoding and decoding, the distortion may be taken into account when generating the texture image to correct the distortion of the patch shape.

[0043] In some embodiments, the group expansion (320) is configured to pad pixels around object boundaries with redundant low-frequency content to improve the coding gain as well as the visual quality of the reconstructed point cloud.

[0044] A multiplexer (324) can multiplex the compressed geometry images, the compressed texture images, the compressed occupancy maps, and the compressed auxiliary patch information into a compressed bitstream.

[0045] 4 shows a block diagram of a V-PCC decoder (400) for decoding a compressed bitstream corresponding to a point cloud frame in some examples. In some examples, the V-PCC decoder (400) can be used in the communication system (100) and the streaming system (200). For example, the decoder (210) can be configured to operate similarly to the V-PCC decoder (400). The V-PCC decoder (400) receives the compressed bitstream and generates a reconstructed point cloud based on the compressed bitstream.

[0046] In the example of FIG. 4, the V-PCC decoder (400) includes a demultiplexer (432), video decompression modules (434) and (436), an occupancy map decompression module (438), an auxiliary patch information decompression module (442), a geometry reconstruction module (444), a smoothing module (446), a texture reconstruction module (448), and a color smoothing module (452).

[0047] A demultiplexer (432) can receive the compressed bitstream and separate it into compressed texture images, compressed geometry images, compressed occupancy maps, and compressed auxiliary patch information.

[0048] The video decompression modules (434) and (436) can decode compressed images according to an appropriate standard (e.g., HEVC, VVC, etc.) and output decompressed images. For example, the video decompression module (434) decodes compressed texture images and outputs decompressed texture images, and the video decompression module (436) decodes compressed geometry images and outputs decompressed geometry images.

[0049] The occupancy map decompression module (438) can decode the compressed occupancy map according to an appropriate standard (e.g., HEVC, VVC, etc.) and output a decompressed occupancy map.

[0050] The auxiliary patch information decompression module (442) can decode the compressed auxiliary patch information according to an appropriate standard (e.g., HEVC, VVC, etc.) and output the decompressed auxiliary patch information.

[0051] A geometry reconstruction module (444) can receive the decompressed geometry image and generate a reconstructed point cloud geometry based on the decompressed occupancy map and the decompressed auxiliary patch information.

[0052] A smoothing module (446) can smooth discrepancies at the edges of the patches. The smoothing procedure aims to reduce potential discontinuities that may occur at patch boundaries due to compression artifacts. In some embodiments, a smoothing filter may be applied to pixels located on patch boundaries to mitigate distortions that may be caused by compression / decompression.

[0053] The texture reconstruction module (448) can determine texture information for points in the point cloud based on the decompressed texture image and the smoothed geometry.

[0054] The color smoothing module (452) can smooth color discrepancies. Non-neighboring patches in 3D space are often packed adjacent to each other in 2D video. In some instances, pixel values ​​from non-neighboring patches may be mixed by block-based video codecs. The purpose of color smoothing is to reduce visible artifacts that appear at patch boundaries.

[0055] 5 shows a block diagram of a video decoder (510) in some examples. The video decoder (510) can be used in the V-PCC decoder (400). For example, the video decompression modules (434) and (436) and the occupancy map decompression module (438) can be configured similarly to the video decoder (510).

[0056] The video decoder (510) may include a parser (520) for reconstructing symbols (521) from a compressed image, such as an encoded video sequence. These symbol categories include information used to manage the operation of the video decoder (510). The parser (520) may parse / entropy decode a received coded video sequence. The coding of the coded video sequence may follow a video coding technique or standard and may follow various principles including variable length coding, Huffman coding, arithmetic coding with or without context dependency, etc. The parser (520) may extract a set of at least one subgroup parameter of a subgroup of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to the group. The subgroup may include a Group of Pictures (GOP), a picture, a tile, a slice, a macroblock, a coding unit (CU), a block, a transform unit (TU), a prediction unit (PU), etc. Additionally, the parser (520) may extract information from the coded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, etc.

[0057] The parser (520) may perform an entropy decoding / parsing operation on the video sequence received from the buffer memory to produce symbols (521).

[0058] The reconstruction of the symbols (521) can involve several different units, depending on the format of the coded video picture or part thereof (inter-picture and intra-picture, inter-block and intra-block, etc.), and other factors. Which units are involved and how can be controlled by subgroup control information parsed from the coded video sequence by the parser (520). The flow of such subgroup control information between the parser (520) and the following units is not depicted for clarity.

[0059] In addition to the functional blocks already mentioned, the video decoder (510) may be conceptually further divided into several functional units, as described below. In an actual implementation operating under commercial constraints, many of these units may interact closely with each other and may be at least partially integrated with each other. However, for purposes of describing the disclosed subject matter, the following conceptual subdivision into functional units is appropriate:

[0060] The first unit is a scalar / inverse transform unit (551), which receives quantized transform coefficients as well as control information from the parser (520) including which transform to use, block size, quantization coefficients, quantization scaling matrices, etc. as symbol(s) (521). The scalar / inverse transform unit (551) can output blocks containing sample values ​​that can be input to an aggregator (555).

[0061] In some cases, the output samples of the scaler / inverse transform (551) may relate to intra-coded blocks, i.e., blocks that do not use prediction information from a previously reconstructed picture, but can use prediction information from a previously reconstructed portion of the current picture. Such prediction information may be provided by an intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) generates a block of the same size and shape as the block being reconstructed using surrounding already reconstructed information fetched from a current picture buffer (558). The current picture buffer (558) buffers, for example, a partially reconstructed current picture and / or a fully reconstructed current picture. The aggregator (555) may add the prediction information generated by the intra-prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551) on a sample-by-sample basis.

[0062] In other cases, the output samples of the scalar / inverse transform unit (551) may relate to an intra-coded and potentially motion-compensated block. In such cases, the motion compensated prediction unit (553) may access the reference picture memory (557) to fetch samples used for prediction. After motion compensating the fetched samples according to the symbols (521) related to the block, these samples may be added by the aggregator (555) to the output of the scalar / inverse transform unit (551) (in this case referred to as residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory (557) from which the motion compensated prediction unit (553) fetches the prediction samples may be controlled by motion vectors available to the motion compensated prediction unit (553) in the form of symbols (521), which may have, for example, X, Y, and reference picture components. Motion compensation may also include interpolation of sample values ​​fetched from the reference picture memory (557) when sub-sample accurate motion vectors are used, motion vector prediction mechanisms, etc.

[0063] The output samples of the aggregator (555) can be subject to various loop filtering techniques in a loop filter unit (556). Video compression techniques are controlled by parameters contained in the coded video sequence (also called a coded video bitstream) and can include in-loop filter techniques available to the loop filter unit (556) as symbols (521) from the parser (520), but can also be responsive to meta-information obtained during decoding of a coded picture or previous portion of the coded video sequence (in decoding order), or to previously reconstructed and loop filtered sample values.

[0064] The output of the loop filter unit (556) can be a sample stream that can be output to a rendering device, but also stored in a reference picture memory (557) for use in future inter-picture prediction.

[0065] Once a particular coded picture has been fully reconstructed, it can be used as a reference picture for future prediction. For example, once a coded picture corresponding to a current picture has been fully reconstructed and the coded picture has been identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) can become part of the reference picture memory (557), and a new current picture buffer can be reallocated before beginning reconstruction of the next coded picture.

[0066] 6 shows a block diagram of a video encoder (603) according to one embodiment of the present disclosure. The video encoder (603) can be used in the V-PCC encoder (300) to compress point clouds. In one example, the video compression modules (322) and (323) and the video compression module (332) are configured similarly to the encoder (603).

[0067] A video encoder (603) may receive images, such as padded geometry images, padded texture images, etc., and generate compressed images.

[0068] According to one embodiment, the video encoder (603) can code and compress pictures (images) of a source video sequence into a coded video sequence (compressed images) in real time or under any other time constraint required by the application. Enforcing the appropriate coding rate is one function of the controller (650). In some embodiments, the controller (650) controls and is functionally coupled to other functional units described below. This coupling is not depicted for clarity. Parameters set by the controller (650) can include rate control related parameters (picture skip, quantizer, lambda value for rate distortion optimization techniques, ...), picture size, Group of Pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) can be configured to have other appropriate functions for the video encoder (603) optimized for a particular system design.

[0069] In some embodiments, the video encoder (603) is configured to operate in a coding loop. As an oversimplified explanation, in one example, the coding loop can include a source coder (630) (e.g., responsible for creating symbols, such as a symbol stream, based on an input picture to be coded and reference picture(s)) and a (local) decoder (633) built into the video encoder (603). The decoder (633) reconstructs the symbols to create sample data in a similar manner as the (remote) decoder does (since in the video compression techniques considered in the disclosed subject matter, any compression between the symbols and the coded video bitstream is lossless). The reconstructed sample stream (sample data) is input to a reference picture memory (634). Since the decoding of the symbol stream produces bit-exact results regardless of the location of the decoder (local or remote), the contents of the reference picture memory (634) are also bit-exact between the local and remote encoders. In other words, the prediction part of the encoder "sees" exactly the same sample values ​​as the decoder will "see" when using prediction during decoding. This basic principle of reference picture synchrony (if synchrony cannot be maintained due to, for example, channel error, the resulting drift) is also used in some related technologies.

[0070] The operation of the "local" decoder (633) may be the same as that of a "remote" decoder, such as the video decoder (510) already detailed above in connection with Figure 5. However, with brief reference also to Figure 5, because symbols are available and the encoding / decoding of symbols into a coded video sequence by the entropy coder (645) and parser (520) may be lossless, the entropy decoding portion of the video decoder (510), including the parser (520), may not be fully implemented in the local decoder (633).

[0071] In operation, in some examples, the source coder (630) may perform motion-compensated predictive coding, which predictively codes an input picture with reference to one or more previously coded pictures from a video sequence designated as “reference pictures.” In this manner, the coding engine (632) codes differences between pixel blocks of the input picture and pixel blocks of the reference picture(s) that may be selected as predictive reference(s) for the input picture.

[0072] The local video decoder (633) may decode the coded video data of pictures that may be designated as reference pictures based on the symbols created by the source coder (630). The operation of the coding engine (632) may advantageously be a lossy process. If the coded video data may be decoded in a video decoder (not shown in FIG. 6), the reconstructed video sequence may be a replica of the source video sequence, typically with some errors. The local video decoder (633) may replicate the decoding process that may be performed by the video decoder on the reference pictures and store the reconstructed reference pictures in a reference picture cache (634). In this way, the video encoder (603) may locally store copies of reconstructed reference pictures that have common content as reconstructed reference pictures obtained by the far-end video decoder (without transmission errors).

[0073] The predictor (635) may perform a prediction search for the coding engine (632). That is, for a new picture to be coded, the predictor (635) may search the reference picture memory (634) for sample data (as candidate reference pixel blocks) that may serve as suitable prediction references for the new picture, or for certain metadata such as reference picture motion vectors, block shapes, etc. The predictor (635) may operate on sample blocks, pixel blocks at a time, to find a suitable prediction reference. In some cases, as determined by the search results obtained by the predictor (635), the input picture may have prediction references drawn from multiple reference pictures stored in the reference picture memory (634).

[0074] The controller (650) may manage the coding operations of the source coder (630), including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0075] The output of all the aforementioned functional units may be subjected to entropy coding in an entropy coder (645), which converts the symbols produced by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques such as Huffman coding, variable length coding, or arithmetic coding.

[0076] The controller (650) may manage the operation of the video encoder (603). During coding, the controller (650) may assign a particular coded picture format to each coded picture, which may affect the coding technique that may be applied to the respective picture. For example, pictures may often be assigned as one of the following picture formats:

[0077] The video encoder (603) may perform coding operations according to a given video coding technique or standard, such as ITU-T Rec. H.265, in which the video encoder (603) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the coded video data may conform to a syntax specified by the video coding technique or standard being used.

[0078] 7 shows a block diagram of a G-PCC encoder (700) in some examples. The G-PCC encoder (700) can be configured to receive point cloud data and compress the point cloud data to generate a bitstream carrying the compressed point cloud data. In one embodiment, the G-PCC encoder (700) can include a position quantization module (710), a duplicate point removal module (712), an octree encoding module (730), an attribute transfer module (720), a level of detail (LOD) generation module (740), an attribute prediction module (750), a residual quantization module (760), an arithmetic coding module (770), an inverse residual quantization module (780), an addition module (781), and a memory (790) for storing reconstructed attribute values.

[0079] As shown, an input point cloud (701) may be received at a G-PCC encoder (700). The positions (e.g., 3D coordinates) of the point cloud (701) are provided to a quantization module (710). The quantization module (710) is configured to quantize the coordinates to generate quantized positions. The duplicate point removal module (712) is configured to receive the quantized positions and perform a filtering process to identify and remove duplicate points. The octree encoding module (730) is configured to receive the filtered positions from the duplicate point removal module (712) and perform an octree-based encoding process to generate a set of occupancy codes that describe a 3D grid of voxels. The occupancy codes are provided to an arithmetic coding module (770).

[0080] The attribute transfer module (720) is configured to receive attributes of the input point cloud and perform an attribute transfer operation to determine an attribute value for each voxel when multiple attribute values ​​are associated with each voxel. The attribute transfer operation may be performed on the reordered points output from the octree encoding module (730). The attributes after the transfer operation are provided to the attribute prediction module (750). The LOD generation module (740) is configured to operate on the reordered points output from the octree encoding module (730) and reorganize the points into different LODs. The LOD information is provided to the attribute prediction module (750).

[0081] The attribute prediction module (750) processes the points according to the LOD-based order indicated by the LOD information from the LOD generation module (740). The attribute prediction module (750) generates an attribute prediction for the current point based on the reconstructed attributes of a set of points in the vicinity of the current point stored in the memory (790). A prediction residual can then be obtained based on the original attribute values ​​received from the attribute transfer module (720) and the locally generated attribute prediction. When a candidate index is used in each attribute prediction process, an index corresponding to the selected prediction candidate may be provided to the arithmetic coding module (770).

[0082] The residual quantization module (760) is configured to receive and quantize the prediction residual from the attribute prediction module (750) to generate a quantized residual. The quantized residual is provided to the arithmetic coding module (770).

[0083] The inverse residual quantization module (780) is configured to receive the quantized residual from the residual quantization module (760) and generate a reconstructed prediction residual by performing an inverse of the quantization operation performed in the residual quantization module (760). The summation module (781) is configured to receive the reconstructed prediction residual from the inverse residual quantization module (780) and receive the respective attribute predictions from the attribute prediction module (750). Reconstructed attribute values ​​are generated by combining the reconstructed prediction residual and the attribute predictions and stored in the memory (790).

[0084] The arithmetic coding module (770) is configured to receive the occupation codes, the candidate indexes (if used), the quantized residuals (if generated), and other information, and to perform entropy encoding to further compress the received values ​​or information. As a result, a compressed bitstream (702) conveying the compressed information may be generated. The bitstream (702) may be transmitted or otherwise provided to a decoder that decodes the compressed bitstream, or may be stored in a storage device.

[0085] 8 shows a block diagram of a G-PCC decoder (800) according to one embodiment. The G-PCC decoder (800) may be configured to receive a compressed bitstream and perform decompression of point cloud data to decompress the bitstream and generate decoded point cloud data. In one embodiment, the G-PCC decoder (800) may include an arithmetic decoding module (810), an inverse residual quantization module (820), an octree decoding module (830), an LOD generation module (840), an attribute prediction module (850), and a memory (860) for storing reconstructed attribute values.

[0086] As shown, a compressed bitstream (801) may be received at an arithmetic decoding module (810). The arithmetic decoding module (810) is configured to decode the compressed bitstream (801) to obtain quantized residuals of a point cloud (if generated) and occupation codes. The octree decoding module (830) is configured to determine reconstructed positions of points in the point cloud according to the occupation codes. The LOD generation module (840) is configured to reorganize the points into different LODs based on the reconstructed positions and determine an LOD-based order. The inverse residual quantization module (820) is configured to generate reconstructed residuals based on the quantized residuals received from the arithmetic decoding module (810).

[0087] The attribute prediction module (850) is configured to perform an attribute prediction process to determine attribute predictions for the points according to an order based on the LOD. For example, the attribute predictions for the current point can be determined based on reconstructed attribute values ​​of points in the vicinity of the current point stored in the memory (860). In some examples, the attribute predictions can be combined with respective reconstructed residuals to generate reconstructed attributes for the current point.

[0088] The sequence of reconstructed attributes generated from the attribute prediction module (850), along with the reconstructed positions generated from the octree decoding module (830), correspond to the decoded point cloud (802) output from the G-PCC decoder (800) in one example. In addition, the reconstructed attributes are also stored in memory (860) and can then be used to derive attribute predictions for subsequent points.

[0089] In various embodiments, the encoder (300), the decoder (400), the encoder (700), and / or the decoder (800) may be implemented using hardware, software, or a combination thereof. For example, the encoder (300), the decoder (400), the encoder (700), and / or the decoder (800) may be implemented using processing circuitry, such as one or more integrated circuits (ICs) that operate with or without software, such as application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), and the like. In other examples, the encoder (300), the decoder (400), the encoder (700), and / or the decoder (800) may be implemented as software or firmware that includes instructions stored on a non-volatile (or non-transitory) computer-readable storage medium. The instructions, when executed by a processing circuitry, such as one or more processors, cause the processing circuitry to perform the functions of the encoder (300), the decoder (400), the encoder (700), and / or the decoder (800).

[0090] It should be noted that the attribute prediction modules (750) and (850) configured to implement the attribute prediction techniques disclosed herein may be included in other decoders or encoders that may have similar or different structures to those shown in Figures 7 and 8. Additionally, the encoder (700) and decoder (800) may be included in the same device or separate devices in various examples.

[0091] According to some aspects of the present disclosure, mesh compression may use a coding tool different from a PCC coding tool, or may use a PCC coding tool such as a PCC (e.g., G-PCC, V-PCC) encoder described above, a PCC (e.g., G-PCC, V-PCC) decoder described above, etc.

[0092] An object's mesh (also called a mesh model, mesh frame) may include polygons that describe the object's surface. Each polygon may be defined by the polygon's vertices in 3D space and the edges that connect the vertices to the polygon. The information of how the vertices are connected (e.g., the edge information) is called connectivity information. In some examples, an object's mesh is formed by connected triangles that describe the object's surface. Two triangles that share an edge are called two connected triangles. In some other examples, an object's mesh is formed by connected quadrilaterals. Two quadrilaterals that share an edge may be called two connected quadrilaterals. It should be noted that a mesh may be formed by other suitable polygons.

[0093] Certain aspects of the present disclosure provide techniques for generating volumetric segmentation masks for instance segmentation of point clouds and for generating associated metadata about the point clouds for use in point cloud optimization.

[0094] According to one aspect of the disclosure, uncompressed or raw point cloud data of dynamic human subjects and objects can have large file sizes. Point cloud optimization can reduce file sizes. Point cloud optimization can include voxelization, which refers to the process of converting a point cloud into a voxel grid and estimating shapes and attributes associated with voxels in the voxel grid. In some relevant examples, point cloud optimization can rely on manual processing and is not suitable for real-time capture to a transmission pipeline. In some relevant examples, voxelization does not consider visual features that have higher visual sensitivity and require higher detail, such as human face regions and other regions of high visual saliency.

[0095] Some aspects of the present disclosure provide a technique for instance segmentation that does not rely on manual processing, and the technique for instance segmentation can be integrated into a pipeline for point cloud processing in an electronic device. The technique for instance segmentation can generate one or more segmentation masks for the point cloud, and the one or more segmentation masks can assist in further point cloud processing. For example, the point cloud can be divided into segmentation instances according to the one or more segmentation masks, and different processing parameters can be applied to the different segmentation instances. For example, the segmentation masks can be used to retain details of some segmentation instances, such as facial features, and can be used to reduce redundancy of some segmentation instances, such as torso parts.

[0096] According to one aspect of the present disclosure, a point cloud, such as a point cloud of uncompressed point cloud data or raw point cloud data having densely packed points, can be rendered according to one or more virtual cameras to generate a virtual camera view (e.g., an image in a 2D plane). The virtual camera view can be viewed as a projection of the point cloud data in 3D space onto a 2D plane. In some examples, the virtual camera view is used to perform instance segmentation. In some examples, the virtual camera view is fed into a fully convolutional model or other suitable system for real-time instance segmentation. In some examples, the result of the instance segmentation includes a 2D pixel-wise mask associated with the segmented element (e.g., instance). In some examples, the result is used as additional input for enhanced downstream attribute and geometry coding, and point cloud optimization.

[0097] FIG. 9 shows a diagram of a computer system (900) according to one embodiment of the disclosure. The computer system (900) includes a virtual camera module (910), an instance segmentation module (920), and an additional point cloud processing module (930). Various modules in the computer system (900) can be implemented by various technologies. In some examples, the various modules are implemented by hardware circuits to perform various functions. In some examples, the various modules are implemented by software instructions, which can be executed by one or more processors to perform various functions.

[0098] In some examples, the computer system (900) is configured to process a point cloud, such as a point cloud of uncompressed point cloud data or raw point cloud data having dense points. The virtual camera module (910) can render a virtual camera view of the point cloud according to virtual camera parameters such as a virtual camera position, a virtual camera angle, a virtual camera focus, and a virtual camera field of view. The virtual camera view can be viewed as a projection of the point cloud data in 3D space onto a 2D plane. In some examples, the virtual camera parameters are set based on knowledge of pre-trained data, such as a human 3D portrait photography dataset.

[0099] FIG. 10 shows a diagram for generating multiple virtual camera views in some examples. In the example of FIG. 10, four virtual cameras (1021)-(1024) are configured to project the point cloud (1010) onto four 2D planes. The four virtual cameras (1021)-(1024) can have respective virtual camera parameters such as respective virtual camera positions, virtual camera angles, virtual camera focal points, and virtual camera fields of view. In some examples, the virtual camera parameters are set based on knowledge of pre-trained data such as a human 3D portrait photography dataset.

[0100] 9, in some examples, the instance segmentation module (920) can generate one or more pixel-wise masks that correspond to instances in the point cloud according to the virtual camera view. In one example, the instance segmentation module (920) includes an artificial neural network, such as a fully convolutional neural network model, trained to generate pixel-wise masks from the virtual camera view in real time.

[0101] A pixel-wise mask is a semantic segmented pixel cloud that assigns pixel-wise correlations to objects in a 2D frame (e.g., a 2D image, a virtual camera view, etc.). In one example, a point cloud corresponds to a scene with four penguins. The instance segmentation module (920) can generate four pixel-wise masks for the virtual camera view, where each penguin is assigned a pixel-wise mask out of the four pixel-wise masks, and a pixel-wise mask of a penguin can describe all pixels of the penguin. The pixel-wise masks can be any suitable mask, such as a binary mask, an RGB mask, etc.

[0102] In some examples, the pixel-wise mask may include metadata correlating the virtual camera view to the pixel-wise mask, such as metadata indicating the virtual camera pose, the field of view of the virtual camera, the size of the virtual sensor in the virtual camera, the resolution of the virtual camera, the aspect ratio of the virtual camera, etc.

[0103] In some examples, the point cloud corresponds to a scene having a person. The virtual camera module (910) can generate one or more virtual camera views and input the virtual camera views to the instance segmentation module (920). The instance segmentation module (920) includes a fully convolutional neural network model trained to generate pixel-wise masks of segments of the person, recognize features of the segments, and generate metadata for the pixel-wise masks. For example, the instance segmentation module (920) can generate a pixel-wise mask of the person, where the pixel-wise mask can include sub-masks corresponding to the head, arms, legs, body, eyes, mouth, etc. The sub-masks can be assigned metadata, such as a segment name.

[0104] FIG. 11 illustrates an example of a pixel-wise mask (1100) in some examples. For example, the pixel-wise mask (1100) is generated according to a virtual camera view of the point cloud (1010). The pixel-wise mask (1100) can include sub-masks (1101)-(1104). In some examples, the pixel-wise mask can be configured according to a UV atlas (or UV map). In some examples, the UV mapping technique generates a 2D UV atlas (also called a UV map). The UV atlas includes an assignment of 3D points to 2D points in the 2D domain. The UV atlas is a mapping between 3D coordinates and 2D domain coordinates. In one example, a point in the UV atlas at 2D coordinates (u,v) has a value formed by the coordinates (x,y,z) of the point in the 3D domain. In some examples, a pixel in the pixel-wise mask (1100) has a corresponding pixel in the UV atlas. The corresponding pixel in the UV atlas includes the 3D coordinates of the corresponding point in the point cloud (1010). In one example, the pixels in the pixel-wise mask (1100) can have binary values ​​or RGB vector values. In some examples, the submasks (1101)-(1104) may have metadata indicative of different segments. For example, submask (1101) has metadata indicative of a person's head, submask (1102) has metadata indicative of a person's arms, submask (1103) has metadata indicative of a person's legs, and submask (1104) has metadata indicative of a person's torso.

[0105] 9, in some examples, the further point cloud processing module (930) can process the point cloud based on the pixel-wise mask. The further point cloud processing module (930) can perform any suitable processing. In one example, the further point cloud processing module (930) can perform encoding, such as attribute and geometry coding, of the point cloud based on the pixel-wise mask. In other examples, the further point cloud processing module (930) can perform optimization of the point cloud, such as voxelization, based on the pixel-wise mask.

[0106] In some examples, the pixel-wise mask and metadata associated with the pixel-wise mask are used to generate a voxel grid from the point cloud, such as an Eulerian voxel grid, a non-Eulerian voxel grid, a cubic voxel grid, a non-cubic voxel grid, etc. The voxel grid can be improved according to the pixel-wise mask. In one example, an Eulerian voxel grid can be generated according to the pixel-wise mask to preserve the level of detail of facial features and frequently targeted objects, as well as features of known human visual saliency. For example, the pixel-wise mask can include a sub-mask of a human face, and an Eulerian voxel grid can be generated to include a finer grid of the human face to preserve the highest level of detail and quality of the human face, such that the encoded data of the point cloud provides a higher resolution for the human face. In another example, an Eulerian voxel grid can be generated according to the pixel-wise mask to include a larger (e.g., wider) grid for a human torso portion, such that redundant voxels / points in the torso portion can be removed to improve coding efficiency, and thus the encoded data of the point cloud may provide a relatively lower resolution for the torso portion.

[0107] It should be noted that the voxel grid can be a cube-based voxel grid or a non-cube-based voxel grid. In one example, the Eulerian voxel grid is a cube-based voxel grid formed by cube voxels of different sizes. In another example, the Eulerian voxel grid is a non-cube-based voxel grid, such as a sphere-based voxel grid formed by spherical voxels of different sizes. It should also be noted that in some examples, a pixel-wise mask is used as an input to help arrange the set of voxels stored in the non-Eulerian array.

[0108] In some examples, the origin of the object's local coordinate system and the object's pose can be set based on pre-trained parameters. In one example, metadata describing the object's kinematics, joints, links, base links, center of gravity, etc. can be determined in a pixel-wise mask based on the pre-trained parameters. The pixel-wise mask and associated metadata can be used in downstream renderers, 3D engines, and simulation environments. In one example, some pixel-wise masks are described as joints and bones based on the metadata associated with the pixel-wise masks. The pixel-wise masks and metadata can be used to generate new poses for, for example, a human.

[0109] FIG. 12 shows an illustration of voxelization in some examples. Voxelization can convert a point cloud to a voxel grid. In the example of FIG. 12, voxelization can convert a point cloud, such as point cloud (1010), to a voxel grid according to a pixel-wise mask (1100). In one example, a first point (1201) in the point cloud is identified by the pixel-wise mask as a point in an eye segment. The first point (1201) is converted to a voxel in a finer grid (e.g., a minimum grid in one example) to preserve the detail of the eye segment. In another example, a second point (1202) in the point cloud is identified by the pixel-wise mask as a point in a torso segment. The second point (1202) is converted to a voxel in a larger grid (e.g., a maximum grid in one example). In some examples, some neighboring points of the second point (1202) are transformed to the same voxels as the second point (1202) and are considered redundant voxels / points. The redundant voxels / points can be removed to reduce the file size of the encoded point cloud.

[0110] In some examples, the pixel-wise masks and their associated metadata are used to generate an enhanced watertight geometry from the point cloud. In some examples, a mesh corresponding to the point cloud is generated based on the pixel-wise mask and the associated metadata. The mesh can be formed by polygons such as triangles, rectangles, etc. Based on the pixel-wise masks and their associated metadata, different sizes of polygons can be used for different parts of, for example, a human. For example, a relatively small size of polygons can be used for a face part of a mesh representing a human to preserve the detail of facial features, and a relatively large polygon can be used for a torso part of a mesh representing a human to improve coding efficiency. Also, features of different parts can be determined according to the metadata, and conversion errors can be avoided. For example, in response to a pixel-wise mask including metadata identifying a hand, a portion of the point cloud corresponding to the pixel-wise mask can be converted into a mesh representing a hand including general hand features such as five fingers.

[0111] 13 shows a flow chart outlining a process (1300) according to one embodiment of the present disclosure. The process (1300) may be used during point cloud processing. In various embodiments, the process (1300) is performed by a processing circuit. In some embodiments, the process (1300) is implemented with software instructions, and thus, when the processing circuit executes the software instructions, the processing circuit performs the process (1300). The process starts at (S1301) and proceeds to (S1310).

[0112] In (S1310), the cloud of points in three-dimensional (3D) space is projected onto one or more two-dimensional (2D) planes to generate one or more images in 2D.

[0113] At (S1320), a pixel-wise mask of the object instance in the point cloud is generated according to the one or more images, the pixel-wise mask including a first pixel associated with a first object instance in the point cloud.

[0114] At (S1330), the point cloud is processed based on a pixel-wise mask, and a portion of the point cloud corresponding to a first pixel in the pixel-wise mask is processed based on one or more processing parameters determined for the first object instance.

[0115] In some examples, the one or more images are input to a convolutional neural network model trained to generate a pixel-wise mask of the object instance. In one example, a pixel-wise mask including mask information and associated metadata is created from one or more virtual camera views using a trained convolutional neural network.

[0116] In some examples, the one or more images are input to a non-neural network based logic model configured to generate a pixel-wise mask of the object instance. In one example, a pixel-wise mask including mask information and associated metadata is created from one or more virtual camera views using non-artificial intelligence based logic.

[0117] In some examples, the point cloud includes points representing a person. The pixel-wise mask can include multiple sub-masks associated with facial and body elements of the person, respectively. In one example, separate masks (e.g., sub-masks) are generated separately for facial elements (e.g., eyes, nose, mouth, etc.), arms, legs, torso, etc.

[0118] In some examples, parameters of each of one or more virtual cameras associated with one or more 2D planes for projection of the point cloud are determined according to the pre-training data. In one example, the pose and orientation of the virtual cameras are set based on knowledge of pre-training data, such as a human 3D portrait photography dataset.

[0119] In some examples, different parameters for voxelizing different object instances are determined, and voxelization is performed accordingly. For example, a first parameter for voxelizing a first object instance is determined, and a portion of the point cloud corresponding to a first pixel in the pixel-wise mask is voxelized according to the first parameter. For example, the pixel-wise mask is used to calculate a voxel unit size and a bounding box of one or more voxel grids. In one example, multiple voxel grids, or a container grid including multiple temporally or spatially displaced child grids, can be input to a voxelization optimizer, which can output one or more modified, optimized, combined, or otherwise improved voxel grids.

[0120] In some examples, to process the point cloud, a scene graph associated with the point cloud is generated based on the pixel-wise mask, the scene graph including at least a first scene graph element of the first object instance. In one example, the pixel-wise mask is input to a scene graph processor for streaming of the scene graph elements. To assist in processing, the decoder, intermediate software, and renderer can be provided with the scene graph elements.

[0121] In some examples, the pixel-wise mask is used to generate a grid of cubic voxels. In some examples, the pixel-wise mask is used to generate a grid of non-cubic voxels.

[0122] In some examples, the pixel-wise mask is used to generate voxels stored in a Euler array. In some examples, the pixel-wise mask is used to generate voxels stored in a non-Euler array.

[0123] In some examples, the point cloud including the pixel-wise mask is processed by a video-based point cloud compression (V-PCC) system, e.g., the pixel-wise mask is used as input to a video-based point cloud compression (such as MPEG V-PCC) system.

[0124] In some examples, the point cloud is divided into multiple segments according to a pixel-wise mask with multiple sub-masks corresponding to the multiple segments. The multiple segments are packed into a geometry map, and the geometry map is encoded into a respective sub-stream. In one example, the point cloud can be further divided into multiple segments using a pixel-wise mask, and each segment can be packed into a geometry map, and the geometry map is encoded into a different sub-stream by MPEG V-PCC.

[0125] In some examples, 2D patches are generated for each of the multiple segments based on the pixel-wise mask, and the 2D patches of the segments include the geometric and semantic information of the 2D patches. In one example, the pixel-wise mask is used to facilitate the 2D patch generation process in MPEG V-PCC, so that the 2D patches not only carry the geometric / texture information but also carry the semantic information.

[0126] In some examples, coding parameters, such as quantization parameters, for encoding portions of the point cloud can be determined based on the pixel-wise mask. In one example, the pixel-wise mask can be used to determine coding parameters for each segment, such as human face segments that use a smaller quantization parameter while being coded by a video codec.

[0127] In some examples, the point cloud including the pixel-wise mask is processed by a geometry-based point cloud compression (G-PCC) system. In one example, the pixel-wise mask is used as input to a geometry-based point cloud compression (such as MPEG G-PCC) system.

[0128] In some examples, the point cloud is divided into multiple slices based on a pixel-wise mask. Further, encoder parameters for each of the multiple slices are determined based on characteristics of each of the multiple slices, and the multiple slices are each encoded into a respective sub-stream based on the encoder parameters. In one example, the point cloud can be further divided into multiple slices using a pixel-wise mask, and each slice can be coded into a sub-stream by MPEG G-PCC. Each slice can be assigned a different set of encoder parameters that match the characteristics of the slice.

[0129] In some examples, the geometry quantization parameters for the octree partitioning are determined based on a pixel-wise mask, and the octree partitioning is performed based on the geometry quantization parameters. In one example, the pixel-wise mask is used to determine different internal geometry quantization parameters used during the octree partitioning in MPEG G-PCC. For example, for an octree node that contains a large portion of a human face region, a smaller geometry quantization parameter is applied to the octree node.

[0130] Next, the process proceeds to (S1399) and ends.

[0131] The process (1300) may be adapted as appropriate. Steps of the process (1300) may be modified and / or omitted. Additional steps may be added. Any suitable order of implementation may be used.

[0132] The techniques disclosed in this disclosure may be used separately or combined in any order. Furthermore, each of the techniques (e.g., methods, embodiments), the encoder and the decoder may be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In some examples, the one or more processors execute a program stored in a non-transitory computer-readable medium.

[0133] The techniques described above can be implemented as computer software using computer readable instructions and physically stored on one or more computer readable media. For example, Figure 14 illustrates a computer system (1400) suitable for implementing certain embodiments of the disclosed subject matter.

[0134] Computer software can be coded using any suitable machine code or computer language, which may rely on mechanisms such as assembly, compilation, linking, etc. to create code including instructions that can be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., or via interpretation, microcode execution, etc.

[0135] For example, the instructions may be executed on various types of computers or components thereof including personal computers, tablet computers, servers, smart phones, gaming devices, Internet of Things devices, and the like.

[0136] 14 for computer system (1400) are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing the embodiments of the present disclosure, nor should the arrangement of components be interpreted as having any dependency or requirement regarding any one or combination of components illustrated in the exemplary embodiment of computer system (1400).

[0137] The computer system (1400) may include certain human interface input devices. Such human interface input devices may respond to input by one or more human users, for example, via tactile input (keystrokes, swipes, data glove movements, etc.), audio input (voice, clapping, etc.), visual input (gestures, etc.), olfactory input (not shown). The human interface devices may be used to capture certain media not necessarily directly associated with conscious human input, such as audio (speech, music, ambient sounds, etc.), images (scanned images, photographic images obtained from still image cameras, etc.), and video (two-dimensional video, three-dimensional video including stereoscopic video, etc.).

[0138] The input human interface devices may include one or more (only one of each is shown) of a keyboard (1401), a mouse (1402), a trackpad (1403), a touch screen (1410), a data glove (not shown), a joystick (1405), a microphone (1406), a scanner (1407), and a camera (1408).

[0139] The computer system (1400) may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the senses of a human user, for example, via haptic output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen (1410), data gloves (not shown), or joystick (1405), although there may also be haptic feedback devices that do not function as input devices), audio output devices (e.g., speakers (1409), headphones (not shown)), visual output devices (e.g., screens (1410) including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capability, each with or without haptic feedback capability, some of which are capable of outputting two-dimensional visual output or three-dimensional hypervisible output via means such as stereo output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0140] The computer system (1400) may also include human-accessible storage devices and their associated media, such as optical media, including CD / DVD ROM / RW (1420), including CD / DVD or similar media (1421), thumb drives (1422), removable hard drives or solid state drives (1423), legacy magnetic media such as tapes and floppy disks (not shown), and dedicated ROM / ASIC / PLD based devices (not shown), such as security dongles.

[0141] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the subject matter of this disclosure does not encompass transmission media, carrier waves or other transitory signals.

[0142] The computer system (1400) may also include an interface (1454) to one or more communication networks (1455). The networks may be, for example, wireless, wired, optical. The networks may further be local, wide area, metropolitan, vehicular and industrial, real-time, delay tolerant, and the like. Examples of networks include local area networks such as Ethernet, wireless LAN, cellular networks including GSM, 3G, 4G, 5G, LTE, and the like, TV wired or wireless wide area digital networks including cable TV, satellite TV and terrestrial broadcast TV, vehicular and industrial including CANBus, and the like. Certain networks typically require an external network interface adapter attached to a particular general-purpose data port or peripheral bus (1449) (e.g., a USB port of the computer system (1400) or the like), while others are typically integrated into the core of the computer system (1400) by attachment to a system bus as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (1400) may communicate with other entities. Such communications may be, for example, one-way receive only (e.g., a TV broadcast), one-way transmit only (e.g., a CANbus to a particular CANbus device), or two-way, to other computer systems using local or wide area digital networks. Specific protocols and protocol stacks may be used in each of those networks and network interfaces, as described above.

[0143] The aforementioned human interface devices, human accessible storage devices, and network interfaces may be attached to the core (1440) of the computer system (1400).

[0144] The cores (1440) may include one or more central processing units (CPUs) (1441), graphics processing units (GPUs) (1442), dedicated programmable processing units in the form of field programmable gate areas (FPGAs) (1443), hardware accelerators for specific tasks (1444), graphics adapters (1450), etc. These devices may be connected via a system bus (1448), along with read only memory (ROM) (1445), random access memory (1446), internal mass storage (1447) such as an internal hard drive or SSD that is not user accessible. In some computer systems, the system bus (1448) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached either directly to the core's system bus (1448) or via a peripheral bus (1449). In one example, a screen (1410) may be connected to the graphics adapter (1450). Architectures for peripheral buses include PCI, USB, etc.

[0145] The CPU (1441), GPU (1442), FPGA (1443), and accelerator (1444) can execute certain instructions that, in combination, can constitute the aforementioned computer code. The computer code can be stored in a ROM (1445) or a RAM (1446). Transient data can also be stored in the RAM (1446), while persistent data can be stored, for example, in internal mass storage (1447). The use of cache memory, which can be closely associated with one or more of the CPU (1441), GPU (1442), mass storage (1447), ROM (1445), RAM (1446), etc., can enable fast storage and retrieval from any of the memory devices.

[0146] The computer-readable medium can bear computer code for performing various computer-implemented operations. The medium and computer code can be those specially designed and constructed for the purposes of the present disclosure, or they can be of the kind well known and available to those skilled in the computer software arts.

[0147] By way of example and not limitation, a computer system (1400) having an architecture, specifically a core (1440), can provide functionality as a result of a processor(s) (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be user-accessible mass storage as described above, as well as media associated with specific storage of the core (1440) of a non-transitory nature, such as the core internal mass storage (1447) or ROM (1445). Software implementing various embodiments of the present disclosure can be stored in such devices and executed by the core (1440). The computer-readable media can include one or more memory devices or chips, depending on the particular needs. The software can cause the core (1440), and specifically the processors therein (including CPUs, GPUs, FPGAs, etc.) to perform certain operations or certain portions of certain operations described herein, including defining data structures stored in RAM (1446) and modifying such data structures according to operations defined by the software. Additionally or alternatively, the computer system may provide functionality as a result of logic (e.g., accelerator (1444)) that is hardwired or otherwise embodied in circuitry that can operate in place of or together with software to perform certain processes, or certain portions of certain processes, as described herein. Where appropriate, references to software may encompass logic and vice versa. Where appropriate, references to computer-readable media may encompass circuitry (such as integrated circuits (ICs)) that stores software for execution, circuitry that embodies logic for execution, or both. The present disclosure encompasses any suitable combination of hardware and software.

[0148] While this disclosure has described several exemplary embodiments, there are alternatives, modifications, and various substitute equivalents that fall within the scope of this disclosure. It will thus be appreciated that those skilled in the art will be able to devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within the spirit and scope of the present disclosure. [Explanation of symbols]

[0149] 100 Communication Systems 105 Sensors 110 Terminal Devices 120 Terminal Devices 150 Network 200 Streaming System 201 Point Cloud Source 202 point cloud 203 Encoder 204 Compressed Point Cloud 205 Streaming Server 206 Client Subsystem 207 Copying compressed point clouds 208 Client Subsystem 209 Copying compressed point clouds 210 Decoder 211 point cloud 212 Rendering Devices 213 Capture Subsystem 220 Electronic Devices 230 Electronic Devices 300V-PCC Encoder 304 Patch Information Module 306 Patch Generation Module 308 Patch Packing Module 310 Geometry Image Generation Module 312 Texture Image Generation Module 314 Occupancy Map Module 316 Image Padding Module 318 Image Padding Module 320 Group Expansion Module 322 Video Compression Module 323 Video Compression Module 324 Multiplexer 332 Video Compression Module 334 Entropy Compression Module 336 Smoothing Module 338 Auxiliary patch information compression module 400 Decoder 432 Demultiplexer 434 Video Decompression Module 436 Video Decompression Module 438 Occupancy Map Decompression Module 442 Auxiliary patch information extraction module 444 Geometry Reconstruction Module 446 Smoothing Module 448 Texture Reconstruction Module 452 Color Smoothing Module 510 Video Decoder 520 Parser 521 Symbols 551 Scaler / Descaler Unit 552 Intra Prediction Units 553 Motion Compensation Prediction Unit 555 Aggregator 556 Loop Filter Unit 557 Reference Picture Memory 558 Current Picture Buffer 603 Video Encoder 630 Source Coder 632 Coding Engine 633 Local Video Decoder 634 Reference Picture Memory 635 Predictor 645 Entropy Coder 650 Controller 700G-PCC Encoder 701 point cloud 702 compressed bitstream 710 Position Quantization Module 712 Duplicate point removal module 720 Attribute Transfer Module 730 Octree Encoding Module 740 Level of Detail (LOD) Generation Module 750 Attribute Prediction Module 760 Residual Quantization Module 770 Arithmetic Coding Module 780 Inverse Residual Quantization Module 781 Addition Module 790 Memory 800 G-PCC Decoder 801 Compressed Bitstream 802 point cloud 810 Arithmetic Decoding Module 820 Inverse Residual Quantization Module 830 Octree Decoding Module 840 LOD Generation Module 850 Attribute Prediction Module 860 Memory 900 Computer Systems 910 Virtual Camera Module 920 Instance Segmentation Module 930 Point Cloud Processing Module 1010 point cloud 1021 Virtual Camera 1022 Virtual Camera 1023 Virtual Camera 1024 Virtual Cameras 1100 Pixel-by-pixel mask 1101 Submask 1102 Submask 1103 Submask 1104 Submask 1201 First point 1202 Second point 1300 Processing 1400 Computer Systems 1401 Keyboard 1402 Mouse 1403 Trackpad 1405 Joystick 1406 Mike 1407 Scanner 1408 Camera 1409 Audio Output Device 1410 Touch Screen 1420 CD / DVD ROM / RW 1421 CDs / DVDs 1422 Thumb Drive 1423 Removable Hard Drive or Solid State Drive 1440 cores 1441 Central Processing Unit (CPU) 1442 Graphics Processing Unit (GPU) 1443 Field Programmable Gate Area (FPGA) 1444 Hardware Accelerator 1445 Read-Only Memory (ROM) 1446 Random Access Memory 1447 Internal Mass Storage 1448 System Bus 1449 Surrounding Bus 1450 Graphics Adapter 1454 Interface 1455 Communication Network

Claims

1. 1. A method for point cloud processing, comprising: acquiring point cloud data corresponding to a cloud of points in three-dimensional (3D) space; projecting the cloud of points in the 3D space onto one or more two-dimensional (2D) planes to generate one or more images; generating a pixel-wise mask of an object instance in the point cloud according to the one or more images, the pixel-wise mask including a first pixel associated with a first object instance in the point cloud; processing the point cloud based on the pixel-wise mask, where a portion of the point cloud corresponding to the first pixel in the pixel-wise mask is processed based on one or more processing parameters determined for the first object instance; A method comprising:

2. The step of generating the pixel-by-pixel mask comprises: inputting the one or more images into a convolutional neural network model trained to generate pixel-wise masks of object instances; and / or inputting the one or more images into a non-neural network based logical model configured to generate a pixel-wise mask of the object instance; The method of claim 1 , comprising at least one of:

3. The point cloud includes points representing a person, and the step of generating the pixel-wise mask comprises: The method of claim 1 , further comprising generating the pixel-wise mask including a plurality of sub-masks associated with facial and body features of the person, respectively.

4. The step of projecting the point cloud further comprises: The method of claim 1 , further comprising determining parameters of each of one or more virtual cameras associated with the one or more 2D planes for projection of the point cloud according to pre-training data.

5. The step of processing the point cloud further comprises: determining first parameters for voxelizing the first object instance; voxelizing the portion of the point cloud corresponding to the first pixel in the pixel-wise mask according to the first parameter; The method of claim 1, further comprising:

6. The step of processing the point cloud further comprises:

2. The method of claim 1, further comprising: generating a scene graph associated with the point cloud based on the pixel-wise mask, the scene graph including at least a first scene element that identifies the first object instance.

7. The step of processing the point cloud further comprises: The method of claim 1 , further comprising processing the point cloud with the pixel-wise mask by a video-based point cloud compression (V-PCC) system.

8. dividing the point cloud into a plurality of segments according to the pixel-wise mask having a plurality of sub-masks corresponding to a plurality of segments; packing each of the plurality of segments into a geometry map; encoding said geometry map into respective sub-streams; 8. The method of claim 7, further comprising:

9. generating a 2D patch for each of the plurality of segments based on the pixel-wise mask, the 2D patch for a segment including geometric and semantic information of the 2D patch; 9. The method of claim 8, further comprising:

10. determining a quantization parameter for encoding the portion of the point cloud based on the pixel-wise mask; 8. The method of claim 7, further comprising:

11. The step of processing the point cloud further comprises: The method of claim 1 , further comprising processing the point cloud with the pixel-wise mask by a geometry-based point cloud compression (G-PCC) system.

12. Dividing the point cloud into a number of slices based on the pixel-wise mask; determining encoder parameters for each of the slices based on the respective characteristics; encoding each of the slices into a respective sub-stream based on the encoder parameters; 12. The method of claim 11, further comprising:

13. determining a geometry quantization parameter for an octree decomposition based on the pixel-wise mask; performing the octree partitioning based on the geometry quantization parameter; 12. The method of claim 11, further comprising:

14. 1. An apparatus for point cloud processing, comprising: Obtain point cloud data corresponding to a set of points in three-dimensional (3D) space; projecting the cloud of points in the 3D space onto one or more two-dimensional (2D) planes to generate one or more images; generating a pixel-wise mask of an object instance in the point cloud according to the one or more images, the pixel-wise mask including a first pixel associated with a first object instance in the point cloud; processing the point cloud based on the pixel-wise mask, where a portion of the point cloud corresponding to the first pixel in the pixel-wise mask is processed based on one or more processing parameters determined for the first object instance. The apparatus includes a processing circuit configured to:

15. 15. The apparatus of claim 14, wherein the processing circuitry is configured to generate a pixel-wise mask of the object instance based on at least one of a convolutional neural network model and / or non-neural network based logic.

16. The point cloud includes points representing a person, and the processing circuitry The apparatus of claim 14 , configured to generate the pixel-wise mask comprising a plurality of sub-masks associated with facial and body features of the person, respectively.

17. The processing circuitry includes:

15. The apparatus of claim 14, configured to determine, according to pre-training data, parameters of each of one or more virtual cameras associated with the one or more 2D planes for projection of the point cloud.

18. The processing circuitry includes: determining first parameters for voxelizing the first object instance; voxelizing the portion of the point cloud corresponding to the first pixel in the pixel-wise mask according to the first parameter; The apparatus of claim 14 configured to:

19. The processing circuitry includes:

15. The apparatus of claim 14, configured to generate a scene graph associated with the point cloud based on the pixel-wise mask, the scene graph including at least a first scene element that identifies the first object instance.

20. The processing circuitry includes:

15. The apparatus of claim 14, configured to process the point cloud using the pixel-wise mask by at least one of a video-based point cloud compression (V-PCC) method and / or a geometry-based point cloud compression (G-PCC) method.

Images