Inter-component residual prediction for color attributes in geometric point cloud compression coding

Abstract

An apparatus for decoding encoded point cloud data can be configured to determine, for a point in a point cloud, a first color value of a first color component based on a first prediction value and a first residual value; apply a scaling factor to the first residual value to determine a predicted second residual value, wherein the scaling factor has one or both of an absolute value greater than one or a non-integer value; receive, for the point in the point cloud, a second residual value in the encoded point cloud data; determine a final second residual value based on the predicted second residual value and the received second residual value; and determine, for the point in the point cloud, a second color value of a second color component based on a second prediction value and the final second residual value.

Description

[0001] This application claims priority to U.S. Patent Application No. 17 / 484,525, filed September 24, 2021, and U.S. Provisional Patent Application No. 63 / 088,345, filed October 6, 2020, the entire contents of each of which are incorporated herein by reference. U.S. Patent Application No. 17 / 484,525, filed September 24, 2021, claims the benefit of U.S. Provisional Patent Application No. 63 / 088,345, filed October 6, 2020. Technical Field

[0002] This disclosure relates to point cloud encoding and decoding. Background Technology

[0003] A point cloud is a collection of points in three-dimensional space. These points can correspond to points on objects within that space. Therefore, point clouds can be used to represent the physical content of three-dimensional space. Point clouds have practical applications in a wide variety of situations. For example, point clouds can be used in the context of autonomous vehicles to represent the location of objects on a road. In another example, point clouds can be used in the context of representing the physical content of an environment for the purpose of locating virtual objects in augmented reality (AR) or mixed reality (MR) applications. Point cloud compression is the process of encoding and decoding point clouds. Encoding point clouds reduces the amount of data required for their storage and transmission. Summary of the Invention

[0004] According to some techniques, to improve decoding efficiency, G-PCC encoders and decoders can be configured to predict the color value of a point (e.g., RGB value) based on the color values ​​of its decoded neighboring points, and to signal the difference between the predicted and actual color values ​​from the encoder to the decoder. For some color components (e.g., G), the encoder and decoder can be configured to determine an initial prediction value based on the decoded neighboring points, and additionally predict a residual value based on the decoded residual value of another component (e.g., R or B). Therefore, the G-PCC encoder only needs to signal the residual difference between the actual and initial prediction values ​​plus the predicted residual value to the G-PCC decoder, instead of signaling the full difference between the initial prediction and the actual color value. Since this residual value can typically be a number smaller than the full residual value, fewer bits are needed to signal it. This use of prediction and residual values ​​reduces the number of bits required to signal the color value compared to directly signaling it.

[0005] This disclosure describes techniques for determining predicted residual values. G-PCC encoders and decoders implementing the techniques of this disclosure are able to generate more accurate predicted residual values, resulting in smaller residual values ​​compared to those determined by existing residual prediction techniques. Since smaller values ​​can typically be decoded using fewer bits, the techniques of this disclosure reduce the total number of bits required to signal attribute values ​​by making the residual residual values ​​smaller.

[0006] According to one example, an apparatus for decoding encoded point cloud data includes a memory configured to store the encoded point cloud data; and one or more processors implemented in a circuit and configured to: determine a first color value for a first color component based on a first predicted value and a first residual value for points in the point cloud; apply a scaling factor to the first residual value to determine a predicted second residual value, wherein the scaling factor has one or both of an absolute value greater than one or a non-integer value; receive a second residual value in the encoded point cloud data for points in the point cloud; determine a final second residual value based on the predicted second residual value and the received second residual value; and determine a second color value for a second color component based on the second predicted value and the final second residual value for points in the point cloud.

[0007] According to another example, a method for decoding encoded point cloud data includes: for points in the point cloud, determining a first color value for a first color component based on a first predicted value and a first residual value; applying a scaling factor to the first residual value to determine a predicted second residual value, wherein the scaling factor has one or both of an absolute value greater than one or a non-integer value; for points in the point cloud, receiving a second residual value in the encoded point cloud data; determining a final second residual value based on the predicted second residual value and the received second residual value; and for points in the point cloud, determining a second color value for a second color component based on the second predicted value and the final second residual value.

[0008] According to another example, a computer-readable storage medium stores instructions that, when executed by one or more processors, cause the one or more processors to perform the following operations: for points in a point cloud, determining a first color value for a first color component based on a first predicted value and a first residual value; applying a scaling factor to the first residual value to determine a predicted second residual value, wherein the scaling factor has one or both of an absolute value greater than one or a non-integer value; for points in the point cloud, receiving a second residual value in encoded point cloud data; determining a final second residual value based on the predicted second residual value and the received second residual value; and for points in the point cloud, determining a second color value for a second color component based on a second predicted value and a final second residual value.

[0009] According to another example, an apparatus for encoding point cloud data includes a memory configured to store the encoded point cloud data; and one or more processors implemented in circuitry and configured to: determine a first color value for a first color component for points in the point cloud; encode the first color value of the first color component into a first predicted value and a first residual value; determine a second color value for a second color component for points in the point cloud; determine a second predicted value for the second color value of the second color component; determine a second residual value for the second color value of the second color component; apply a scaling factor to the first residual value to determine a predicted second residual value, wherein the scaling factor has one or both of an absolute value greater than one or a non-integer value; and generate a syntax element representing the difference between the second residual value and the predicted second residual value to be included in the point cloud data.

[0010] Details of one or more examples are set forth in the accompanying drawings and the following description. Other features, objects, and advantages will be apparent from the description, drawings, and claims. Attached Figure Description

[0011] Figure 1 This is a block diagram illustrating an example encoding and decoding system that can perform the techniques of this disclosure.

[0012] Figure 2 This is a block diagram illustrating an example geometric point cloud compression (G-PCC) encoder.

[0013] Figure 3 This is a block diagram illustrating an example G-PCC decoder.

[0014] Figure 4 This is a flowchart illustrating an example operation of a G-PCC encoder according to one or more techniques of this disclosure.

[0015] Figure 5 This is a flowchart illustrating an example operation of a G-PCC decoder according to one or more techniques of this disclosure.

[0016] Figure 6 This is a conceptual diagram illustrating an example ranging system that can be used with one or more techniques disclosed herein.

[0017] Figure 7 This is a conceptual diagram illustrating an example vehicle-based scenario in which one or more technologies of this disclosure may be used.

[0018] Figure 8 This is a conceptual diagram illustrating an example extended reality system in which one or more techniques of this disclosure may be used.

[0019] Figure 9This is a conceptual diagram illustrating an example mobile device system in which one or more technologies of this disclosure may be used. Detailed Implementation

[0020] Geometry-based point cloud compression (G-PCC) directly compresses 3D geometry, which is the location of a set of points in 3D space. G-PCC also compresses associated attribute values, such as color values, for each point associated with the 3D geometry.

[0021] A point cloud contains a collection of points in 3D space and can have attributes associated with those points. Attributes can be, for example, color information such as R / G / B, Y / Cb / Cr, reflectance information, or other attributes such as temperature values, humidity values, latitude coordinates, longitude coordinates, etc. Point clouds can be captured by various cameras or sensors such as light detection and ranging (LIDAR) scanners or 3D scanners, and can also be computer-generated. Point cloud data can be used in a variety of applications, including but not limited to architecture (e.g., modeling), graphics (e.g., 3D models for visualization and animation), and the automotive industry (e.g., LIDAR sensors for navigation aids).

[0022] The 3D space occupied by point cloud data can be surrounded by virtual bounding boxes. The positions of points within the bounding boxes can be represented with a specific precision. Therefore, the positions of one or more points can be quantized based on precision. At the smallest level, the bounding box is divided into voxels, which are the smallest units of space represented by unit cubes. A voxel in the bounding box can be associated with zero, one, or more points. The bounding box can be segmented into multiple cubic / cuboid regions, which can be called tiles, and each tile can be decoded into one or more slices. The segmentation of the bounding box into slices and tiles can be based on the number of points in each segment, or on other considerations (e.g., a specific region can be decoded into a tile). Slice regions can be further segmented using segmentation decisions similar to those in video codecs.

[0023] According to some techniques, to improve decoding efficiency, G-PCC encoders and decoders can be configured to predict the color value of a point (e.g., RGB value) based on the color values ​​of its decoded neighboring points, and to signal the difference between the predicted and actual color values ​​from the encoder to the decoder. For some color components (e.g., G), the encoder and decoder can be configured to determine an initial prediction value based on the decoded neighboring points, and additionally predict a residual value based on the decoded residual value of another component (e.g., R or B). Therefore, the G-PCC encoder only needs to signal the residual difference between the actual and initial prediction values ​​plus the predicted residual value to the G-PCC decoder, instead of signaling the full difference between the initial prediction and the actual color value. Since this residual value can typically be a number smaller than the full residual value, fewer bits are needed to signal the residual value. This use of prediction and residual values ​​reduces the number of bits required to signal the color value compared to directly signaling it.

[0024] An exemplary technique for transmitting residual values ​​as signals will now be described. In this example, Res_R, Res_G, and Res_B represent the residual values ​​of the R, G, and B color components, respectively. It can be observed that the R, G, and B residual values ​​can be approximated by the following relationships:

[0025]

[0026]

[0027] For decoding Res_B and Res_R, the encoder and decoder can perform further predictions based on Res_G, and therefore only the residual values ​​need to be included in the point cloud data. From the decoder's perspective, if ResR, ResG, and ResB are residual samples reconstructed from the bitstreams of R, G, and B, respectively, then the final R residual can be determined as Res'R = ResR + ResG, and the final B residual can be determined as Res'B = ResB + ResG, where Res'R and Res'B represent the final R residual value and the final B residual value, respectively.

[0028] Although the above technique utilizes the correlation (positive or negative) between the two color components (i.e., the two channels), it does not take advantage of the fact that the residual energy of the two color components may differ in some decoding scenarios.

[0029] This disclosure describes a technique for utilizing scaling values ​​Sb and Sr, such that the G-PCC decoder 300 can be configured to determine the final B residual and the final R residual as follows:

[0030] Res'B=ResB+Sb*ResG

[0031] Res'R=ResR+Sr*ResG

[0032] This disclosure also describes techniques for signaling these values ​​of Sb and Sr. For example, for p = 1 / 4, possible values ​​of Sb and Sr could be k*p = k / 4, where k is an integer. The value of p can be fixed, predetermined, or signaled. In some examples, the values ​​of Sb and Sr can have a range of [-a, +a] with precision p; for example, for a = 2 and p = 1 / 4, the scaling factor value is s: {-2, -7 / 4, -6 / 4, -5 / 4, -1, -3 / 4, -2 / 4, -1 / 4, 0, 1 / 4, 2 / 4, 3 / 4, 1, 5 / 4, 6 / 4, 7 / 4, 2}. The values ​​of a and p can be fixed, predetermined, signaled, or any combination thereof. As will be explained in more detail below, the scaling factor values ​​of Sb and Sr can also be implemented as a combination of multiplication and bit shift operations to potentially avoid floating-point operations and thus reduce computational complexity.

[0033] As an example, suppose a point in a point cloud has color values ​​(R, G, B) = (81, 100, 82). The G-PCC decoder can predict a predicted color value of (predR, predG, predB) = (76, 96, 75) based on previously decoded points. Therefore, the residual between the actual color value of the point and its predicted color value is (resR, resG, resB) = (5, 4, 7). Thus, the G-PCC encoder can include the value 4 for resG in the point cloud. However, for resB, the G-PCC encoder can include a value equal to 7 - Sb * resG in the point cloud. If Sb can only be equal to 1, the G-PCC encoder includes a value of 3 in the point cloud data (3 = 7 - 4). However, using the techniques disclosed herein, the G-PCC encoder can, for example, include a value of 0 instead of 3 (0 = 7 - Sb * 4, where Sb equals 7 / 4) in the point cloud data. Since 0 is less than 3, the G-PCC encoder can decode the value 0 using fewer bits than the value 3. The G-PCC encoder can encode resR using a scaling value of Sr in a similar manner.

[0034] Therefore, by determining a scaling factor having one or both of an absolute value or a non-integer value greater than one, the G-PCC encoder and G-PCC decoder can generate more accurate predicted residual values, making the residual residual values ​​smaller when compared to those determined by existing residual prediction techniques. Since smaller values ​​can typically be decoded using fewer bits, the techniques of this disclosure reduce the total number of bits required to signal attribute values ​​by making the residual residual values ​​smaller.

[0035] Figure 1This is a block diagram illustrating an example encoding and decoding system 100 capable of implementing the techniques of this disclosure. The techniques of this disclosure are generally aimed at decoding (encoding and / or decoding) point cloud data, i.e., supporting point cloud compression. Typically, point cloud data includes any data used for processing point clouds. Decoding can be efficient when compressing and / or decompressing point cloud data.

[0036] like Figure 1 As shown, system 100 includes a source device 102 and a destination device 116. The source device 102 provides encoded point cloud data to be decoded by the destination device 116. Specifically, in Figure 1 In this example, source device 102 provides point cloud data to destination device 116 via computer-readable medium 110. Source device 102 and destination device 116 can include any of a wide range of devices, including desktop computers, laptop computers, tablet computers, set-top boxes, handsets such as smartphones, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, land or sea vehicles, spacecraft, aircraft, robots, LiDAR devices, satellites, etc. In some cases, source device 102 and destination device 116 may be equipped for wireless communication.

[0037] exist Figure 1 In the example, source device 102 includes a data source 104, a memory 106, a G-PCC encoder 200, and an output interface 108. Destination device 116 includes an input interface 122, a G-PCC decoder 300, a memory 120, and a data consumer 118. According to this disclosure, the G-PCC encoder 200 of source device 102 and the G-PCC decoder 300 of destination device 116 can be configured to apply the techniques disclosed herein related to the decoding of components of color attributes in G-PCC. Therefore, source device 102 represents an example of an encoding device, and destination device 116 represents an example of a decoding device. In other examples, source device 102 and destination device 116 may include other components or arrangements. For example, source device 102 may receive data (e.g., point cloud data) from an internal or external source. Similarly, destination device 116 may interface with an external data consumer, rather than including the data consumer in the same device.

[0038] like Figure 1The system 100 shown is merely an example. Generally, other digital encoding and / or decoding devices can perform the techniques disclosed herein related to the decoding of the final components of color attributes in G-PCC. Source device 102 and destination device 116 are merely examples of such devices, where source device 102 generates decoded data for transmission to destination device 116. This disclosure refers to a “decoding” device as a device that performs the decoding (encoding and / or decoding) of data. Therefore, G-PCC encoder 200 and G-PCC decoder 300 represent examples of decoding devices, particularly encoders and decoders. In some examples, source device 102 and destination device 116 can operate in a substantially symmetrical manner, such that each of source device 102 and destination device 116 includes encoding and decoding components. Therefore, system 100 can support unidirectional or bidirectional transmission between source device 102 and destination device 116, for example, for streaming, playback, broadcasting, telephone, navigation, and other applications.

[0039] Typically, data source 104 represents the source of data (i.e., raw, unencoded point cloud data) and provides a continuous sequence of "frames" of data to G-PCC encoder 200, which encodes the data for the frames. Data source 104 of source device 102 may include point cloud capture devices, such as any of a variety of cameras or sensors, such as a 3D scanner or LiDAR device, one or more video cameras, an archive containing previously captured data, and / or a data feed interface for receiving data from a data content provider. Alternatively or additionally, the point cloud data may be computer-generated from a scanner, camera, sensor, or other data source. For example, data source 104 may generate computer-graphics-based data as source data, or produce a combination of live data, archived data, and computer-generated data. In each case, G-PCC encoder 200 encodes the captured, pre-captured, or computer-generated data. G-PCC encoder 200 may rearrange frames from the received order (sometimes referred to as "display order") into a decoding order for decoding. G-PCC encoder 200 may generate one or more bitstreams comprising the encoded data. The source device 102 can then output encoded data to the computer-readable medium 110 via the output interface 108 for reception and / or retrieval by, for example, the input interface 122 of the destination device 116.

[0040] The memory 106 of source device 102 and the memory 120 of destination device 116 can represent general-purpose memory. In some examples, memory 106 and memory 120 may store raw data, such as raw data from data source 104 and raw decoded data from G-PCC decoder 300. Alternatively, memory 106 and memory 120 may store software instructions executable by, for example, G-PCC encoder 200 and G-PCC decoder 300. Although in this example, memory 106 and memory 120 are shown separately from G-PCC encoder 200 and G-PCC decoder 300, it should be understood that G-PCC encoder 200 and G-PCC decoder 300 may also include internal memory for functionally similar or equivalent purposes. Furthermore, memory 106 and memory 120 may store encoded data, such as encoded data output from G-PCC encoder 200 and input to G-PCC decoder 300. In some examples, portions of memory 106 and memory 120 may be allocated as one or more buffers, for example, to store raw, decoded, and / or encoded data. For instance, memory 106 and memory 120 may store data representing point clouds.

[0041] Computer-readable medium 110 can represent any type of medium or device capable of transmitting encoded data from source device 102 to destination device 116. In one example, computer-readable medium 110 represents a communication medium that enables source device 102 to transmit encoded data to destination device 116 in real time, for example, via a radio frequency network or a computer-based network. Depending on the communication standard, such as a wireless communication protocol, output interface 108 can modulate the transmitted signal including the encoded data, and input interface 122 can demodulate the received transmitted signal. The communication medium can include any wireless or wired communication medium, such as radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium can form part of a packet-based network, such as a local area network, a wide area network, or a global network, such as the Internet. The communication medium can include a router, a switch, a base station, or any other equipment that facilitates communication from source device 102 to destination device 116.

[0042] In some examples, source device 102 can output encoded data to storage device 112 from output interface 108. Similarly, destination device 116 can access encoded data from storage device 112 via input interface 122. Storage device 112 may include any of a variety of distributed or locally accessed data storage media, such as hard disk drives, Blu-ray discs, DVDs, CD-ROMs, flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded data.

[0043] In some examples, source device 102 may output encoded data to file server 114 or another intermediate storage device that may store the encoded data generated by source device 102. Destination device 116 may access the stored data from file server 114 via streaming or downloading. File server 114 may be any type of server device capable of storing encoded data and sending it to destination device 116. File server 114 may represent a web server (e.g., for a website), a file transfer protocol (FTP) server, a content delivery network device, or a network attached storage (NAS) device. Destination device 116 may access the encoded data from file server 114 via any standard data connection, including an internet connection. This may include wireless channels (e.g., Wi-Fi connections), wired connections (e.g., digital subscriber line (DSL), cable modems, etc.), or combinations thereof, adapted to access the encoded data stored on file server 114. File server 114 and input interface 122 may be configured to operate according to streaming protocols, download protocols, or combinations thereof.

[0044] Output interface 108 and input interface 122 may represent a wireless transmitter / receiver, a modem, a wired networking component (e.g., an Ethernet card), a wireless communication component operating according to any of the various IEEE 802.11 standards, or other physical components. In examples where output interface 108 and input interface 122 include wireless components, output interface 108 and input interface 122 may be configured to transmit data such as encoded data according to cellular communication standards such as 4G, 4G-LTE (Long Term Evolution), LTE Advanced, 5G, or similar. In some examples where output interface 108 includes a wireless transmitter, output interface 108 and input interface 122 may be configured according to other wireless standards, such as the IEEE 802.11 specification, the IEEE 802.15 specification (e.g., ZigBee), etc. TM Bluetooth TM A standard or similar standard is used to transmit data, such as encoded data. In some examples, source device 102 and / or destination device 116 may include their respective system-on-chip (SoC) devices. For example, source device 102 may include an SoC device for performing functions belonging to G-PCC encoder 200 and / or output interface 108, and destination device 116 may include an SoC device for performing functions belonging to G-PCC decoder 300 and / or input interface 122.

[0045] The technology disclosed herein can be applied to encoding and decoding of any application supporting a variety of applications, such as communication between autonomous vehicles, scanners, cameras, sensors and processing devices such as local or remote servers, geomapping or other applications.

[0046] The input interface 122 of the destination device 116 receives an encoded bitstream from a computer-readable medium 110 (e.g., a communication medium, storage device 112, file server 114, etc.). The encoded bitstream may include signaling information defined by the G-PCC encoder 200, which is also used by the G-PCC decoder 300, such as syntax elements having values ​​describing the characteristics and / or processing of the decoded units (e.g., slices, frames, frame groups, sequences, etc.). The data consumer 118 uses the decoded data. For example, the data consumer 118 may use the decoded data to determine the location of physical objects. In some examples, the data consumer 118 may include a display for rendering images based on point clouds.

[0047] The G-PCC encoder 200 and G-PCC decoder 300 can each be implemented as any of a variety of suitable encoder and / or decoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. When the technology is partially implemented in software, the device may store instructions for software in a suitable non-transitory computer-readable medium and execute those instructions in hardware using one or more processors to perform the technology of this disclosure. Each of the G-PCC encoder 200 and G-PCC decoder 300 may be included in one or more encoders or decoders, wherein either may be integrated as part of a combined encoder / decoder (CODEC) in the respective device. Devices including the G-PCC encoder 200 and / or G-PCC decoder 300 may include one or more integrated circuits, microprocessors, and / or other types of devices.

[0048] The G-PCC encoder 200 and G-PCC decoder 300 may operate according to a decoding standard, such as the Video Point Cloud Compression (V-PCC) standard or the Geometric Point Cloud Compression (G-PCC) standard. This disclosure generally relates to the decoding (e.g., encoding and decoding) of point clouds to include processes of encoding or decoding data. Encoded bitstreams typically include a series of values ​​for syntax elements representing decoding decisions (e.g., decoding modes).

[0049] This disclosure may generally relate to "signaling" specific information, such as syntax elements. The term "signaling" can generally refer to the transmission of values ​​for syntax elements and / or other data used to decode encoded data. That is, the G-PCC encoder 200 may signal the values ​​of syntax elements in the bitstream. Typically, signaling notification refers to the generation of values ​​in the bitstream. As described above, source device 102 may transmit the bitstream to destination device 116 substantially in real time or non-real time (such as when syntax elements are stored in storage device 112 for later retrieval by destination device 116).

[0050] ISO / IEC MPEG (JTC 1 / SC 29 / WG 11) is investigating the potential need for standardization of point cloud decoding techniques with compression capabilities significantly exceeding current methods, and aims to create such a standard. The group is working collaboratively in this exploratory activity through a team known as the 3D Graphics Team (3DG) to evaluate compression technology designs proposed by their experts in the field.

[0051] Point cloud compression activities are categorized into two distinct approaches. The first is “Video Point Cloud Compression” (V-PCC), which segments a 3D object and projects these segments onto multiple 2D planes (represented as “patches” in 2D frames). These segments are then decoded by a traditional 2D video codec, such as the High Efficiency Video Decoding (HEVC) (ITU-TH.265) codec. The second approach is “Geometry-Based Point Cloud Compression” (G-PCC), which directly compresses 3D geometry—the location of a set of points in 3D space—along with associated attribute values ​​(for each point associated with the 3D geometry). G-PCC applies compression to point clouds in both Category 1 (static point clouds) and Category 3 (dynamically acquired point clouds). The latest draft of the G-PCC standard is available in G-PCC DIS, ISO / IEC JTC1 / SC29 / WG11 w19088, Brussels, Belgium, January 2020, and the codec description is available in G-PCC Codec Description v6, ISO / IEC JTC1 / SC29 / WG11w19091, Brussels, Belgium, January 2020.

[0052] A point cloud is a collection of points in 3D space and can have attributes associated with those points. Attributes can be color information, such as R, G, B or Y, Cb, Cr, or reflectivity information, or other attributes. Point clouds can be captured by various cameras or sensors, such as LiDAR sensors and 3D scanners, and can also be computer-generated. Point cloud data is used in a variety of applications, including but not limited to architecture (modeling), graphics (3D models for visualization and animation), and the automotive industry (LiDAR sensors for navigation aids).

[0053] The 3D space occupied by point cloud data can be surrounded by virtual bounding boxes. The positions of points within the bounding box can be represented with a specific precision; therefore, the positions of one or more points can be quantized based on this precision. At the smallest level, the bounding box is divided into voxels, which are the smallest units of space represented by unit cubes. A voxel within the bounding box can be associated with zero, one, or more points. The bounding box can be segmented into multiple cubic / cuboid regions, which can be called tiles. Each tile can be decoded into one or more slices. Segmenting the bounding box into slices and tiles can be based on the number of points in each segment, or on other considerations (e.g., a specific region can be decoded into a tile). Slice regions can be further segmented using segmentation decisions similar to those in video codecs.

[0054] Figure 2 An overview of the G-PCC encoder 200 is provided. Figure 3 An overview of the G-PCC decoder 300 is provided. The modules shown are logical and do not necessarily correspond one-to-one with the code implemented in the reference implementation of the G-PCC codec, namely the TMC13 test model software studied by ISO / IEC MPEG (JTC 1 / SC 29 / WG 11).

[0055] In both the G-PCC encoder 200 and the G-PCC decoder 300, the point cloud locations are decoded first. Attribute decoding depends on the decoded geometry. Figure 2 and Figure 3 In this configuration, surface approximation analysis units 212 and 310, and RAHT units 218 and 314 represent options typically used for Category 1 data, while LOD generation units 220 and 316, lifting unit 222, and inverse lifting unit 318 represent options typically used for Category 3 data. All other units can be common between Category 1 and Category 3.

[0056] For Category 3 data, the compressed geometry is typically represented as an octree from the root down to the leaf level of each voxel. For Category 1 data, the compressed geometry is typically represented by a pruned octree (i.e., an octree from the root down to the leaf level of blocks larger than voxels) plus a model of the surface within each leaf of the pruned octree. In this way, Category 1 and Category 3 data share the octree decoding mechanism, while Category 1 data can additionally approximate the voxels in each leaf with a surface model (called Trisoup decoding). The surface model used is a triangulation consisting of 1-10 triangles per block, thus generating a triangle soup. Therefore, the Category 1 geometry codec is called a Trisoup geometry codec, while the Category 3 geometry codec is called an octree geometry codec.

[0057] At each node of the octree, the occupancy of one or more of its child nodes (up to eight nodes) is signaled (when not inferred). Multiple neighborhoods are specified, including (a) nodes sharing a face with the current octree node, (b) nodes sharing a face, edge, or vertex with the current octree node, and so on. Within each neighborhood, the occupancy of a node and / or its child nodes can be used to predict the occupancy of the current node or its child nodes. For points sparsely populated in some nodes of the octree, the codec also supports a direct decoding mode, where the 3D position of the point is directly encoded. A signaling flag can be used to indicate the direct signaling mode. At the lowest level, the number of points associated with an octree node / leaf node can also be decoded.

[0058] Once the geometry is decoded, the attributes corresponding to the geometric points are also decoded. When there are multiple attribute points corresponding to a reconstructed / decoded geometric point, the attribute values ​​representing the reconstructed point can be derived.

[0059] G-PCC employs three attribute decoding procedures: Region Adaptive Hierarchical Transform (RAHT) decoding, interpolation-based hierarchical nearest neighbor prediction (prediction transform), and interpolation-based hierarchical nearest neighbor prediction (lifting transform) with update / lifting steps. RAHT and lifting are typically used for Class 1 data, while prediction is typically used for Class 3 data. However, any procedure can be used for any data, and, like the geometry codec in G-PCC, the attribute decoding procedure used to decode the point cloud is specified in the bitstream.

[0060] Attribute decoding can be performed at the level-of-detail (LOD) level, where a finer representation of the point cloud attributes can be obtained using each level of detail. Each level of detail can be specified based on a distance metric from neighboring nodes or based on the sampling distance. Level-of-detail generation divides the set of input points into a series of levels of detail and associated refinement layers. A level of detail is a set of points representing a downsampled version of the slice geometry. The zeroth level of detail is the full-resolution slice geometry.

[0061] At the G-PCC encoder 200, the residuals obtained from the output of the decoding process, which are considered as attributes, are quantized. Context-adaptive arithmetic decoding can be used to decode the quantized residuals.

[0062] exist Figure 2 In the example, the G-PCC encoder 200 may include a coordinate transformation unit 202, a color transformation unit 204, a voxelization unit 206, an attribute transfer unit 208, an octree analysis unit 210, a surface approximation analysis unit 212, an arithmetic coding unit 214, a geometric reconstruction unit 216, a RAHT unit 218, a LOD generation unit 220, a lifting unit 222, a coefficient quantization unit 224, and an arithmetic coding unit 226.

[0063] like Figure 2 As shown in the example, the G-PCC encoder 200 can receive a set of locations and a set of attributes. Locations can include the coordinates of points in the point cloud. Attributes can include information about the points in the point cloud, such as the color associated with a point in the point cloud.

[0064] The coordinate transformation unit 202 can apply a transformation to the coordinates of a point to transform the coordinates from the initial domain to the transformation domain. The transformed coordinates may be referred to as transformed coordinates in this disclosure. The color transformation unit 204 can apply a transformation to transform the color information of an attribute to a different domain. For example, the color transformation unit 204 can transform color information from the RGB color space to the YCbCr color space. The inter-component residual prediction technique described herein can be performed in the RGB color space.

[0065] In addition, Figure 2 In the example, voxelization unit 206 can voxelize the transformed coordinates. Voxelization of the transformed coordinates may include quantization and removal of some points in the point cloud. In other words, multiple points in the point cloud may be included within a single "voxel," which can then be treated as a single point in some respects. Furthermore, octree analysis unit 210 can generate an octree based on the voxelized transformed coordinates. Additionally, in Figure 2In the example, surface approximation analysis unit 212 can analyze these points to potentially determine a surface representation of the set of points. Arithmetic coding unit 214 can entropy encode the syntax elements representing information about the octree and / or surface determined by surface approximation analysis unit 212. G-PCC encoder 200 can output these syntax elements in a geometric bitstream.

[0066] The geometric reconstruction unit 216 can reconstruct the transformed coordinates of points in the point cloud based on an octree, data indicating the surface determined by the surface approximation analysis unit 212, and / or other information. Due to voxelization and surface approximation, the number of transformed coordinates reconstructed by the geometric reconstruction unit 216 may differ from the original number of points in the point cloud. The resulting points can be referred to as reconstructed points. The attribute transfer unit 208 can transfer attributes of the original points in the point cloud to the reconstructed points.

[0067] Furthermore, RAHT unit 218 can apply RAHT decoding to the attributes of the reconstructed points. Alternatively or additionally, LOD generation unit 220 and lifting unit 222 can respectively apply LOD processing and lifting to the attributes of the reconstructed points. RAHT unit 218 and lifting unit 222 can generate coefficients based on the attributes. Coefficient quantization unit 224 can quantize the coefficients generated by RAHT unit 218 or lifting unit 222. Arithmetic encoding unit 226 can apply arithmetic decoding to the syntax elements representing the quantized coefficients. G-PCC encoder 200 can output these syntax elements in the attribute bitstream.

[0068] exist Figure 3 In the example, the G-PCC decoder 300 includes a geometric arithmetic decoding unit 302, an attribute arithmetic decoding unit 304, an octree synthesis unit 306, an inverse quantization unit 308, a surface approximation synthesis unit 310, a geometric reconstruction unit 312, a RAHT unit 314, an LOD generation unit 316, an inverse lifting unit 318, an inverse coordinate transformation unit 320, and an inverse color transformation unit 322.

[0069] The G-PCC decoder 300 can obtain a geometric bitstream and an attribute bitstream. The geometric arithmetic decoding unit 302 of the G-PCC decoder 300 can apply arithmetic decoding (e.g., context-adaptive binary arithmetic decoding (CABAC) or other types of arithmetic decoding) to the syntax elements in the geometric bitstream. Similarly, the attribute arithmetic decoding unit 304 can apply arithmetic decoding to the syntax elements in the attribute bitstream.

[0070] Octree synthesis unit 306 can synthesize octrees based on syntax elements parsed from the geometric bitstream. In the case of using surface approximation in the geometric bitstream, surface approximation synthesis unit 310 can determine the surface model based on syntax elements parsed from the geometric bitstream and based on the octree.

[0071] Furthermore, the geometric reconstruction unit 312 can perform reconstruction to determine the coordinates of points in the point cloud. The inverse coordinate transformation unit 320 can apply an inverse transformation to the reconstructed coordinates to transform the reconstructed coordinates (positions) of points in the point cloud from the transformation domain back to the initial domain.

[0072] In addition, Figure 3 In the example, the inverse quantization unit 308 reversibly quantizes the attribute value. The attribute value can be based on syntax elements obtained from the attribute bitstream (e.g., including syntax elements decoded by the attribute arithmetic decoding unit 304).

[0073] Depending on how the attribute values ​​are encoded, RAHT unit 314 can perform RAHT decoding to determine the color values ​​of the points in the point cloud based on the inverse quantized attribute values. In some examples, LOD generation unit 316 and inverse lifting unit 318 can use level-of-detail (LMD) based techniques to determine the color values ​​of the points in the point cloud.

[0074] In addition, Figure 3 In the example, the inverse color transformation unit 322 can apply an inverse color transformation to the color value. The inverse color transformation can be the inverse of the color transformation applied by the color transformation unit 204 of the G-PCC encoder 200. For example, the color transformation unit 204 can transform color information from the RGB color space to the YCbCr color space. Therefore, the inverse color transformation unit 322 can transform color information from the YCbCr color space to the RGB color space. The inter-component residual prediction technique described herein can be performed in the RGB color space, meaning that this inverse color transformation can be skipped.

[0075] Figure 2 and Figure 3 The various units are illustrated to aid in understanding the operations performed by the G-PCC encoder 200 and the G-PCC decoder 300. These units can be implemented as fixed-function circuits, programmable circuits, or a combination thereof. A fixed-function circuit is a circuit that provides a specific function and is pre-programmed for the operations it can perform. A programmable circuit is a circuit that can be programmed to perform various tasks and provides flexible functionality within the operations it can perform. For example, a programmable circuit can run software or firmware that causes it to operate in a manner defined by instructions from the software or firmware. A fixed-function circuit can run software instructions (e.g., to receive or output parameters), but the type of operation performed by a fixed-function circuit is typically immutable. In some examples, one or more of the units may be different circuit blocks (fixed-function or programmable), and in some examples, one or more units may be integrated circuits.

[0076] According to the technology disclosed herein, the G-PCC encoder 200 and the G-PCC decoder 300 can be configured to perform cross-component residual prediction for predicting the transform. For decoding RGB content (typically with a decoding order of G, followed by B, then R), there is often correlation between the (dequantized) residuals of different color components. To improve decoding efficiency, the G-PCC decoder 300 can be configured to predict the residuals of the B and R channels from the residuals of the G channel.

[0077] From the perspective of the G-PCC decoder 300, ResX (where X represents a channel that can be G, B, or R) represents the reconstructed sample of the bitstream from channel X. In TMC-v11, the G-PCC decoder 300 determines the final residuals of the B and R channels, namely Res'B and Res'R, as follows:

[0078] Res'B=ResB+ResG

[0079] Res'R=ResR+ResG

[0080] Intercomponent residual prediction can be enabled and disabled via a flag (aps.inter_component_prediction_enabled_flag) included in the Attribute Parameter Set (APS).

[0081] Although current techniques for cross-component residual prediction utilize the correlation between the two channels, they employ a fixed correlation of +1, failing to take advantage of the fact that the difference in residual energy between the two components may differ.

[0082] This disclosure describes a technique for improving cross-component residual prediction by utilizing the difference in residual energy between two components. The techniques described herein can be implemented independently or in combination.

[0083] According to an example technique of this disclosure, the G-PCC encoder 200 and G-PCC decoder 300 can decode a common scaling value for Sb and Sr for each LoD, which can then be used to scale the value of ResG to perform residual prediction. Therefore, the G-PCC decoder 300 can determine the final residuals of the B and R channels, namely Res'B and Res'R, as follows:

[0084] Res'B=ResB+Sb*ResG

[0085] Res'R=ResR+Sr*ResG

[0086] According to another example technique of the present disclosure, the G-PCC encoder 200 and the G-PCC decoder 300 may be configured to decode more diverse values of Sb and Sr for each LoD, where Sb and Sr may be any values with (fractional) precisions p1 and p2, respectively. For example, for p1 = p2 = 1 / 4 (= p), the possible values of Sb and Sr are kb*p = kb / 4 and kr*p = kr / 4, respectively, where kb and kr are integers. The values of p1 and p2 may be fixed, predetermined, or signaled. For simplicity, a common precision referred to as P = (p1 = p2) will be used to describe the techniques of the present disclosure, but these techniques may also be extended to account for different precisions for each component. That is, in some implementations, p1 may not be equal to p2, and the available scaling values for signaling ResB and ResR may be different.

[0087] According to another example technique of the present disclosure, the G-PCC encoder 200 and the G-PCC decoder 300 may be configured to utilize Sb and Sr values with a range [-a, +a] with corresponding precisions. For example, the precision may be represented as P. Thus, for a = 2 and P = 1 / 4, the available scaling factor values are: {-2, -7 / 4, -6 / 4, -5 / 4, -1, -3 / 4, -2 / 4, -1 / 4, 0, 1 / 4, 2 / 4, 3 / 4, 1, 5 / 4, 6 / 4, 7 / 4, 2}. The values of a and p may be fixed, predetermined, or signaled.

[0088] In some examples of the present disclosure, the precision p may be represented as a power of 2, i.e., p = 1 / 2 b , such that the division associated with Sb and Sr may be represented by a right shift:

[0089] Res’B = ResB + (kb*ResG >> b)

[0090] Res’R = ResR + (kr*ResG >> b).

[0091] The value kx (x = B, R) is a scaled version of Sx, e.g., kx = Sx << b (equivalent of Sx = Kx*p). Alternatively, a rounding term may be incorporated for prediction; thus, the residual derivation may be:

[0092] Res’B = ResB + ((kb*ResG + 1 << (b - 1)) >> b)

[0093] Res’R = ResR + ((kr*ResG + 1 << (b - 1)) >> b).

[0094] According to another example technique of this disclosure, the G-PCC encoder 200 and G-PCC decoder 300 can be configured to directly decode the signaling of kx (x = b, r) for each LoD level (kx[lod]) by first decoding the absolute value of kx[lod] and then, where applicable, decoding the symbol (if kx[lod] is not equal to zero). For amplitude decoding, fixed-length codes, whose length is predetermined, fixed, or signaled, can be used, or variable-length codes can be used. The resulting symbols are typically correlated in neighboring LoDs, so the G-PCC encoder 200 and G-PCC decoder 300 can be configured to decode the number using a dedicated context.

[0095] Since the k values ​​of different LoDs are often correlated, according to another example technique of this disclosure, the G-PCC encoder 200 and the G-PCC decoder 300 can be configured to predict kx[lod] based on kx[lod-1], if applicable, and decode the difference (kx[lod] – kx[lod-1]). Decoding of amplitude and symbol can be performed separately.

[0096] According to another example technique of this disclosure, the G-PCC encoder 200 and G-PCC decoder 300 can be configured to signal the value of kx in the attribute data unit header or attribute slice header. In this case, the k value can be decoded using se(v) whether it is direct decoding or differential decoding, meaning that the value can be decoded into a signed integer zero-order exponent Columbus decoding syntax element with the left bit first.

[0097] The resolution of the kx value uses information about the number of LoDs. In G-PCC, the maximum number of LoDs (LoD1) is signaled in the APS. However, the actual number of LoDs (LoD2) can only be derived after LoD reconstruction and can be less than LoD1 in some cases. That is, some slices may have such a sparse arrangement of points or so few points that not all LoDs can be generated. For example, in the example decoding scenario, LoD1 may be equal to 12 and LoD2 may be equal to 10. To avoid the resolution dependency associated with LoD reconstruction, according to another example technique of this disclosure, the G-PCC encoder 200 and the G-PCC decoder 300 may be configured to decode, for example, always the LoD1 scaled value instead of the LoD2 value, and the final (LoD1-LoD2) value may be signaled as zero, since such a value does not need to be used in the decoding process of attribute reconstruction.

[0098] According to another example technique of this disclosure, the G-PCC encoder 200 and the G-PCC decoder 300 can be configured to decode the kx value at the slice level rather than at the LoD level, which simplifies the decoding process and also reduces signaling overhead.

[0099] According to another example technique of this disclosure, the G-PCC encoder 200 and the G-PCC decoder 300 can be configured to apply the decoding tool to a general multidimensional property, wherein for an N-dimensional property having a first (major) component and other (N-1) other components, (N-1) different scaling values ​​can be signaled for each (N-1) other component, which specify the corresponding scaling applied to the major component before performing residual prediction.

[0100] According to the above technology, the G-PCC decoder 300 represents an example of a device for processing point clouds, which can be configured to receive point cloud data representing point clouds; determine the residual value of a first component of the point cloud data; and predict the residual value of a second component of the point cloud data based on the residual value of the first component.

[0101] According to the above technology, the G-PCC encoder 200 represents an example of a device for processing point clouds, which can be configured to obtain point cloud data representing point clouds; determine the residual value of a first component of the point cloud data; determine the residual value of a second component of the point cloud data; and, based on the residual value of the first component, transmit the residual value of the second component of the point cloud data by signaling.

[0102] According to the techniques described above, the G-PCC encoder 200 can be configured to determine a first color value for a first color component (such as a G component) for points in a point cloud, and to encode the first color value of the first color component as a first predicted value and a first residual value. The G-PCC encoder 200 can determine a second color value for a second color component, such as an R or B component, for points in the point cloud. The G-PCC encoder 200 can determine a second predicted value for the second color value of the second color component. The G-PCC encoder 200 can determine a second residual value for the second color value of the second color component, such as Res'B and Res'R as described above. The G-PCC encoder 200 can apply a scaling factor having one or both of an absolute value or a non-integer value greater than one to the first residual value to determine a predicted second residual value. For example, the scaling factor can be applied by multiplying by a scalar value (such as k as described above) and by a bit shift operation (e.g., >>b as described above). The G-PCC encoder 200 can generate syntax elements representing the difference between the second residual value and the predicted second residual value to be included in the point cloud data. This difference may correspond, for example, to ResB or ResR as described above.

[0103] According to the techniques described above, the G-PCC decoder 300 can be configured to determine a first color value for a first color component based on a first predicted value and a first residual value for points in a point cloud. The first color component may be, for example, a green component, and the first residual value may correspond to ResG as described above. The G-PCC decoder 300 may apply a scaling factor having one or both of an absolute value or a non-integer value greater than one to the first residual value to determine a predicted second residual value. For example, the scaling factor may be applied by multiplying by a scalar value (such as k mentioned above) and by a bit shift operation (e.g., >>b mentioned above). For points in the point cloud, the G-PCC decoder 300 may receive a second residual value, such as ResR or ResB mentioned above, in the encoded point cloud data, and determine a final second residual value (e.g., Res'R and Res'B mentioned above) based on the predicted second residual value and the received second residual value. Based on the second predicted value and the final second residual value, the G-PCC decoder 300 may determine a second color value for a second color component, such as a red or blue component.

[0104] Figure 4 This is a flowchart illustrating an example operation of a G-PCC encoder 200 according to one or more techniques of this disclosure. The G-PCC encoder 200 determines a first color value (402) for a first color component for a point in a point cloud. The G-PCC encoder 200 encodes the first color value of the first color component into a first predicted value and a first residual value (404). The first color component may be, for example, a green component.

[0105] The G-PCC encoder 200 determines a second color value (406) for a second color component for a point in the point cloud. The second color component may be, for example, a red component or a blue component.

[0106] The G-PCC encoder 200 determines a second predicted value (408) of the second color value of the second color component. For example, the G-PCC encoder 200 may determine the second predicted value of a point in the point cloud based on the values ​​of different points (such as neighboring points) in the point cloud. The G-PCC encoder 200 determines a second residual value (410) of the second color value of the second color component. The second residual value may, for example, be the difference between the second color value and the second predicted value.

[0107] In this example, the G-PCC encoder 200 applies a scaling factor to a first residual value to determine a predicted second residual value, the scaling factor having one or both of an absolute value greater than one or a non-integer value (412). The G-PCC encoder 200 may, for example, determine and generate a scalar value as an integer to be included in the point cloud data. To generate the scalar value, the G-PCC encoder 200 may generate a difference corresponding to the difference between the scalar value and a previously determined scalar value, and include a variable-length syntax element in the point cloud data indicating the difference. For example, the previously determined scalar value may be a scalar value for a level of detail different from that of the point cloud. The G-PCC encoder 200 may include the scalar value in a syntax structure such as an attribute slice header. To apply the scaling factor, the G-PCC encoder 200 may perform a multiplication with the scalar value and perform a bit shift operation, such as a right shift of 2.

[0108] The G-PCC encoder 200 generates syntax elements representing the difference between the second residual value and the predicted second residual value to be included in the point cloud data (414).

[0109] Figure 5 This is a flowchart illustrating an example operation of a G-PCC decoder 300 according to one or more technologies of this disclosure. Figure 5 In the example, the G-PCC decoder 300 determines a first color value (502) for a point in the point cloud based on a first predicted value and a first residual value. The first color component may be, for example, the green component.

[0110] The G-PCC decoder 300 applies a scaling factor to a first residual value to determine a predicted second residual value, wherein the scaling factor has one or both of an absolute value greater than one or a non-integer value (504). For example, the G-PCC decoder 300 may receive a scalar value in the point cloud data as part of a syntax structure such as an attribute slice header, the scalar value being an integer value, and apply the scaling factor by performing a multiplication with the scalar value and performing a bit shift operation. For example, the bit shift operation could be a right shift by 2 bits. For example, the G-PCC decoder 300 may receive a variable-length syntax element indicating a difference corresponding to the difference between the scalar value and a previously received scalar value from a different level of detail than the level of detail comprising the point cloud.

[0111] In this example, the G-PCC decoder 300 receives second residual values ​​(506) of points in the encoded point cloud data and determines a final second residual value (508) based on the predicted second residual value and the received second residual value. The final second residual value may, for example, be the sum of the received second residual value and the predicted second residual value.

[0112] For a point in the point cloud, the G-PCC decoder 300 determines a second color value for the second color component based on the second predicted value and the final second residual value (510). For example, the G-PCC decoder 300 may determine the second predicted value of a point in the point cloud based on the values ​​of different points in the point cloud. The second color component may be, for example, a red or blue component.

[0113] Figure 6 This is a conceptual diagram illustrating an example ranging system 600 that can be used with one or more technologies disclosed herein. Figure 6 In one example, the ranging system 600 includes an illuminator 602 and a sensor 604. The illuminator 602 may emit light 606. In some examples, the illuminator 602 may emit light 606 as one or more laser beams. Light 606 may be one or more wavelengths, such as infrared wavelengths or visible light wavelengths. In other examples, light 606 is not a coherent laser. When light 606 encounters an object such as object 608, light 606 creates a return light 610. The return light 610 may include backscattered light and / or reflected light. The return light 610 may be guided by a lens 611 to create an image 612 of object 608 on sensor 604. Sensor 604 generates a signal 618 based on image 612. Image 612 may include a set of points (e.g., such as...). Figure 6 (Represented by the points in image 612).

[0114] In some examples, illuminator 602 and sensor 604 can be mounted on a rotating structure, allowing illuminator 602 and sensor 604 to capture a 360-degree view of the environment. In other examples, ranging system 600 may include one or more optical components (e.g., mirrors, collimators, diffraction gratings, etc.) that enable illuminator 602 and sensor 604 to detect objects within a specific range (e.g., up to 360 degrees). Although Figure 6 The example shows only a single illuminator 602 and sensor 604, but the ranging system 600 may include multiple sets of illuminators and sensors.

[0115] In some examples, illuminator 602 generates a structured light pattern. In such examples, ranging system 600 may include multiple sensors 604 on which corresponding images of the structured light pattern are formed. Ranging system 600 can use the parallax between images of the structured light pattern to determine the distance to object 608 from which the structured light pattern is backscattered. When object 608 is relatively close to sensor 604 (e.g., 0.2 meters to 2 meters), the structured light-based ranging system can have a high level of accuracy (e.g., sub-millimeter accuracy). This high level of accuracy can be useful in facial recognition applications, such as unlocking mobile devices (e.g., mobile phones, tablets, etc.) and for security applications.

[0116] In some examples, the ranging system 600 is a time-of-flight (ToF) based system. In some examples of the ToF-based ranging system 600, an illuminator 602 generates a light pulse. In other words, the illuminator 602 can modulate the amplitude of the emitted light 606. In such examples, a sensor 604 detects the return light 610 from the light pulse 606 generated by the illuminator 602. The ranging system 600 can then determine the distance to the object 608 from which the light 606 is backscattered based on the time delay between the time the light 606 is emitted and detected and the known speed of light in air. In some examples, instead of (or in addition to) modulating the amplitude of the emitted light 606, the illuminator 602 can modulate the phase of the emitted light 606. In such examples, the sensor 604 can detect the phase of the return light 610 from the object 608 and use the speed of light and the time difference between the time when the illuminator 602 generates the light 606 at a specific phase and the time when the sensor 604 detects the return light 610 at that specific phase to determine the distance to a point on the object 608.

[0117] In other examples, point clouds can be generated without using illuminator 602. For instance, in some examples, sensor 604 of ranging system 600 may include two or more optical cameras. In such examples, ranging system 600 can use the optical cameras to capture a stereo image of the environment including object 608. Ranging system 600 (e.g., point cloud generator 620) can then calculate the parallax between positions in the stereo image. Ranging system 600 can then use the parallax to determine the distance to the positions shown in the stereo image. Based on these distances, point cloud generator 620 can generate a point cloud.

[0118] Sensor 604 can also detect other properties of object 608, such as color and reflectivity information. Figure 6 In the example, point cloud generator 620 can generate a point cloud based on signal 618 generated by sensor 604. Ranging system 600 and / or point cloud generator 620 can form data source 104 ( Figure 1 Part of ).

[0119] Figure 7 This is a conceptual diagram illustrating an example vehicle-based scenario in which one or more technologies of this disclosure can be used. Figure 7 In the example, vehicle 700 includes laser package 702, such as a LIDAR system. Although in Figure 7 The example is not shown, but vehicle 700 may also include a data source and a G-PCC encoder, such as G-PCC encoder 200. Figure 1 ).exist Figure 7In the example, laser package 702 emits a laser beam 704 reflected from a pedestrian 706 or other object in the road. The data source of vehicle 700 can generate a point cloud based on the signal generated by laser package 702. A G-PCC encoder of vehicle 700 can encode the point cloud to generate a bitstream 708. Bitstream 708 can include significantly fewer bits than the unencoded point cloud obtained by the G-PCC encoder. The output interface of vehicle 700 (e.g., output interface 108) Figure 1 The bitstream 708 can be sent to one or more other devices. Therefore, the vehicle 700 may be able to send the bitstream 708 to other devices faster than uncoded point cloud data. In addition, the bitstream 708 may require less data storage capacity.

[0120] exist Figure 7 In the example, vehicle 700 can send bitstream 708 to another vehicle 710. Vehicle 710 may include a G-PCC decoder, such as G-PCC decoder 300. Figure 1 The G-PCC decoder of vehicle 710 can decode bitstream 708 to reconstruct a point cloud. Vehicle 710 can use the reconstructed point cloud for various purposes. For example, vehicle 710 can determine that pedestrian 706 is in the road ahead of vehicle 700 based on the reconstructed point cloud, and therefore begin to slow down even before, for example, the driver of vehicle 710 becomes aware that pedestrian 706 is in the road. Thus, in some examples, vehicle 710 can perform autonomous navigation operations, generate notifications or warnings, or perform other actions based on the reconstructed point cloud.

[0121] Alternatively, vehicle 700 may send bitstream 708 to server system 712. Server system 712 may use bitstream 708 for various purposes. For example, server system 712 may store bitstream 708 for subsequent reconstruction of the point cloud. In this example, server system 712 may use the point cloud along with other data (e.g., vehicle telemetry data generated by vehicle 700) to train an autonomous driving system. In other examples, server system 712 may store bitstream 708 for subsequent reconstruction for forensic crash investigations (e.g., if vehicle 700 collides with pedestrian 706), or may send notifications or instructions for navigation to vehicle 700 or vehicle 710.

[0122] Figure 8 This is a conceptual diagram illustrating an example extended reality system in which one or more technologies of this disclosure may be used. Extended reality (XR) is a term used to cover a range of technologies including augmented reality (AR), mixed reality (MR), and virtual reality (VR). Figure 8In the example, a first user 800 is located at a first position 802. User 800 wears an XR headset 804. Alternatively, user 800 may use a mobile device (e.g., a mobile phone, tablet, etc.). The XR headset 804 includes a depth sensor, such as a LiDAR system, which detects the position of a point on an object 806 at the first position 802. The data source of the XR headset 804 may use signals generated by the depth sensor to generate a point cloud representation of the object 806 at position 802. The XR headset 804 may include a G-PCC encoder (e.g., [unclear text - likely a typo]) configured to encode the point cloud to generate a bitstream 808. Figure 1 (G-PCC encoder 200).

[0123] XR headset 804 can send a bitstream 808 (e.g., via a network such as the Internet) to XR headset 810 worn by user 812 at a second location 814. XR headset 810 can decode bitstream 808 to reconstruct a point cloud. XR headset 810 can use the point cloud to generate an XR visualization (e.g., AR, MR, VR visualization) representing object 806 at location 802. Thus, in some examples, such as when XR headset 810 generates a VR visualization, user 812 at location 814 can have a 3D immersive experience at location 802. In some examples, XR headset 810 can determine the location of a virtual object based on the reconstructed point cloud. For example, XR headset 810 can determine that the environment (e.g., location 802) includes a flat surface based on the reconstructed point cloud, and then determine that a virtual object (e.g., a cartoon character) will be positioned on that flat surface. XR headset 810 can generate an XR visualization where the virtual object is at the determined location. For example, the XR Headset 810 can display a cartoon character sitting on a flat surface.

[0124] Figure 9 This is a conceptual diagram illustrating an example mobile device system in which one or more technologies of this disclosure may be used. Figure 9 In the example, mobile device 900, such as a mobile phone or tablet computer, includes a depth detection sensor, such as a LiDAR system, which detects the location of points on object 902 in the environment of mobile device 900. The data source of mobile device 900 can use signals generated by the depth detection sensor to generate a point cloud representation of object 902. Mobile device 900 may include a G-PCC encoder (e.g., [missing information]) configured to encode the point cloud to generate a bitstream 904. Figure 1 (G-PCC encoder 200). In Figure 9In the example, mobile device 900 can send a bitstream to remote device 906, such as a server system or other mobile device. Remote device 906 can decode bitstream 904 to reconstruct a point cloud. Remote device 906 can use the point cloud for various purposes. For example, remote device 906 can use the point cloud to generate a map of the environment of mobile device 900. For instance, remote device 906 can generate a map of the interior of a building based on the reconstructed point cloud. In another example, remote device 906 can generate an image (e.g., computer graphics) based on the point cloud. For instance, remote device 906 can use points in the point cloud as vertices of polygons and use the color attributes of the points as the basis for coloring the polygons. In some examples, remote device 906 can use the point cloud to perform facial recognition.

[0125] Examples in various aspects of this disclosure may be used individually or in any combination.

[0126] The following numbered clauses describe one or more aspects of the devices and technologies described in this disclosure.

[0127] Clause 1A. A method for decoding point cloud data, the method comprising: receiving point cloud data representing a point cloud; determining a residual value of a first component of the point cloud data; and predicting a residual value of a second component of the point cloud data based on the residual value of the first component.

[0128] Clause 2A. The method according to Clause 1A, wherein predicting the residual value of a second component of the point cloud data based on the residual value of the first component comprises: determining a difference corresponding to the difference between the residual value of the second component and the residual value of the first component.

[0129] Clause 3A. The method according to Clause 1A, wherein predicting the residual value of a second component of point cloud data based on the residual value of a first component comprises: determining a scaling value; and determining a difference corresponding to the difference between the residual value of the second component and the scaled residual value of the first component.

[0130] Clause 4A. The method according to any of Clauses 1A-3A, wherein the point cloud data includes RGB point cloud data, the first component includes the G component of the RGB point cloud data, and the second component includes the B component of the RGB point cloud data.

[0131] Clause 5A. The method according to any of Clauses 1A-3A, wherein the point cloud data includes RGB point cloud data, the first component includes the G component of the RGB point cloud data, and the second component includes the R component of the RGB point cloud data.

[0132] Clause 6A. The method pursuant to any of Clauses 1A-5A further includes: generating a point cloud based on a first component of the point cloud data and a second component of the point cloud.

[0133] Clause 7A. An apparatus for processing point clouds, the apparatus comprising one or more components for performing a method pursuant to any one of Clauses 1A-6A.

[0134] Clause 8A. A device pursuant to Clause 7A, wherein one or more components include one or more processors implemented in a circuit.

[0135] Clause 9A. The device pursuant to any of Clauses 7A or 8A further includes: a memory for storing data representing the point cloud.

[0136] Clause 10A. A device pursuant to any of Clauses 7A-9A, wherein the device includes a decoder.

[0137] Clause 11A. Devices pursuant to any of Clauses 7A-10A, wherein the device includes an encoder.

[0138] Clause 12A. The apparatus pursuant to any of Clauses 7A-11A further includes: an apparatus for generating point clouds.

[0139] Clause 13A. A device pursuant to any of Clauses 7-12 further includes: a display for rendering images based on point clouds.

[0140] Clause 14A. A computer-readable storage medium having instructions stored thereon, which, when executed, cause one or more processors to perform the method of any one of Clauses 1A-6A.

[0141] Clause 1B. An apparatus for decoding encoded point cloud data, the apparatus comprising: a memory configured to store the encoded point cloud data; and one or more processors implemented in circuitry and configured to: determine, for points in the point cloud, a first color value for a first color component based on a first predicted value and a first residual value; apply a scaling factor to the first residual value to determine a predicted second residual value, wherein the scaling factor has one or both of an absolute value greater than one or a non-integer value; receive, for points in the point cloud, a second residual value in the encoded point cloud data; determine a final second residual value based on the predicted second residual value and the received second residual value; and determine, for points in the point cloud, a second color value for a second color component based on the second predicted value and the final second residual value.

[0142] Clause 2B. The device according to Clause 1B, wherein the one or more processors are further configured to: receive information in point cloud data indicating a scalar value, wherein the scalar value is an integer value, and wherein, in order to apply the scaling factor, the one or more processors are further configured to perform a multiplication of a first residual value with the scalar value.

[0143] Clause 3B. A device pursuant to Clause 2B, wherein, in order to apply the scaling factor, the one or more processors are further configured to perform a bit shift operation including a right shift of 2.

[0144] Clause 4B. For devices pursuant to Clause 2B, the information indicating the scalar value includes the difference corresponding to the difference between the scalar value and a previously determined scalar value.

[0145] Clause 5B. A device according to Clause 4B, wherein the point cloud includes a point cloud for a first level of detail and a previously determined scalar value is used for a second level of detail different from the first level of detail.

[0146] Clause 6B. In accordance with Clause 2B, the device wherein, in order to receive information indicating the scalar value, the one or more processors are further configured to receive information in an attribute slice header syntax structure indicating the scalar value.

[0147] Clause 7B. A device according to Clause 1B, wherein the one or more processors are further configured to: determine a second predicted value for a point in the point cloud based on the values ​​of different points in the point cloud.

[0148] Clause 8B. An apparatus according to Clause 1B, wherein the point cloud comprises a point cloud for a first level of detail, and wherein, in order to determine the scaling factor, the one or more processors are further configured to determine the scaling factor based on a scaling factor for a second level of detail different from the first level of detail.

[0149] Clause 9B. A device pursuant to Clause 1B, wherein the first color component comprises a green component and the second color component comprises a red component.

[0150] Clause 10B. A device pursuant to Clause 1B, wherein the first color component includes a green component and the second color component includes a blue component.

[0151] Clause 11B. The apparatus according to Clause 1B, wherein the one or more processors are further configured to: apply a second scaling factor to the first residual value to determine a predicted third residual value, wherein the second scaling factor has one or both of an absolute value greater than one or a non-integer value; for points in the point cloud, receive the third residual value in the encoded point cloud data; determine a final third residual value based on the predicted third residual value and the received third residual value; and for points in the point cloud, determine a third color value for a third color component based on the third predicted value and the final third residual value.

[0152] Clause 12B. A device pursuant to Clause 1B, wherein the one or more processors are further configured to reconstruct the point cloud.

[0153] Clause 13B. A device according to Clause 12B, wherein the one or more processors are further configured to generate a map of the building's interior based on the reconstructed point cloud.

[0154] Clause 14B. In a device pursuant to Clause 12B, one or more processors are further configured to perform autonomous navigation operations based on the reconstructed point cloud.

[0155] Clause 15B. A device according to Clause 12B, wherein the one or more processors are further configured to generate computer graphics based on a reconstructed point cloud.

[0156] Clause 16B. A device according to Clause 12B, wherein the one or more processors are configured to: determine the position of a virtual object based on the reconstructed point cloud; and generate an extended reality (XR) visualization of the virtual object at the determined position.

[0157] Clause 17B. The device pursuant to Clause 12B also includes: a display for rendering an image based on the reconstructed point cloud.

[0158] Clause 18B. Devices pursuant to Clause 1B, wherein the device includes one or more of a mobile phone, tablet computer, vehicle, or extended reality device.

[0159] Clause 19B. A method for decoding encoded point cloud data, the method comprising: for points in the point cloud, determining a first color value for a first color component based on a first predicted value and a first residual value; applying a scaling factor to the first residual value to determine a predicted second residual value, wherein the scaling factor has one or both of an absolute value greater than one or a non-integer value; for points in the point cloud, receiving a second residual value in the encoded point cloud data; determining a final second residual value based on the predicted second residual value and the received second residual value; and for points in the point cloud, determining a second color value for a second color component based on the second predicted value and the final second residual value.

[0160] Clause 20B. The method according to Clause 19B further includes: receiving information in the point cloud data indicating a scalar value, wherein the scalar value is an integer value, and wherein applying the scaling factor includes performing a multiplication of the first residual value with the scalar value.

[0161] Clause 21B. A device pursuant to Clause 20B, wherein applying the scaling factor includes performing a bit shift operation comprising a right shift of 2.

[0162] Clause 22B. In accordance with Clause 20B, the information indicating the scalar value includes a difference corresponding to the difference between the scalar value and a previously determined scalar value.

[0163] Clause 23B. A device according to Clause 22B, wherein the point cloud comprises a point cloud for a first level of detail and the previously determined scalar value is for a second level of detail different from the first level of detail.

[0164] Clause 24B. In accordance with Clause 20B, the information for receiving an indication scalar value includes information in the attribute slice header syntax structure indicating the scalar value.

[0165] Clause 25B. The method according to Clause 19B further includes: determining a second predicted value for a point in the point cloud based on the values ​​of different points in the point cloud.

[0166] Clause 26B. The method according to Clause 19B further includes: determining the scaling factor based on the scaling factor used for the second level of detail.

[0167] Clause 27B. The method according to Clause 19B, wherein the first color component includes a green component and the second color component includes a red component.

[0168] Clause 28B. The method according to Clause 19B, wherein the first color component includes a green component and the second color component includes a blue component.

[0169] Clause 29B. The method according to Clause 19B further includes: applying a second scaling factor to the first residual value to determine a predicted third residual value, wherein the second scaling factor has one or both of an absolute value greater than one or a non-integer value; receiving the third residual value in the encoded point cloud data for a point in the point cloud; determining a final third residual value based on the predicted third residual value and the received third residual value; and determining a third color value for a third color component for a point in the point cloud based on the third predicted value and the final third residual value.

[0170] Clause 30B. A computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to: determine a first color value for a first color component based on a first predicted value and a first residual value for points in a point cloud; apply a scaling factor to the first residual value to determine a predicted second residual value, wherein the scaling factor has one or both of an absolute value greater than one or a non-integer value; receive, for points in the point cloud, a second residual value in encoded point cloud data; determine a final second residual value based on the predicted second residual value and the received second residual value; and determine a second color value for a second color component based on the second predicted value and the final second residual value for points in the point cloud.

[0171] Clause 31B. An apparatus for encoding point cloud data, the apparatus comprising: a memory configured to store the encoded point cloud data; and one or more processors implemented in circuitry and configured to: determine, for points in the point cloud, a first color value of a first color component; encode the first color value of the first color component into a first predicted value and a first residual value; for points in the point cloud, determine a second color value of a second color component; determine a second predicted value of the second color value of the second color component; determine a second residual value of the second color value of the second color component; apply a scaling factor to the first residual value to determine a predicted second residual value, wherein the scaling factor has one or both of an absolute value greater than one or a non-integer value; and generate a syntax element representing the difference between the second residual value and the predicted second residual value to be included in the point cloud data.

[0172] Clause 32B. A device according to Clause 31B, wherein the one or more processors are further configured to: generate information indicating a scalar value to be included in the point cloud data, wherein the scalar value is an integer value, and wherein, in order to apply the scaling factor, the one or more processors are further configured to perform a multiplication of the first residual value with the scalar value.

[0173] Clause 33B. A device according to Clause 32B, wherein, in order to apply the scaling factor, the one or more processors are further configured to perform a bit shift operation including a right shift of 2.

[0174] Clause 34B. In accordance with Clause 32B, the information indicating the scalar value includes a difference corresponding to the difference between the scalar value and a previously determined scalar value.

[0175] Clause 35B. A device according to Clause 34B, wherein the point cloud comprises a point cloud for a first level of detail and the previously determined scalar value is for a second level of detail different from the first level of detail.

[0176] Clause 36B. According to Clause 32B, in order to include information indicating the scalar value in the point cloud data, the one or more processors are further configured to include the information indicating the scalar value in the attribute slice header syntax structure.

[0177] Clause 37B. A device pursuant to Clause 32B, wherein, in order to generate the scalar value, the one or more processors are further configured to include the scalar value in an attribute slice header syntax structure.

[0178] Clause 38B. A device according to Clause 31B, wherein the one or more processors are further configured to: determine a second predicted value for a point in the point cloud based on the values ​​of different points in the point cloud.

[0179] Clause 39B. A device according to Clause 31B, wherein the point cloud includes a point cloud for a first level of detail, and wherein, in order to determine the scaling factor, the one or more processors are further configured to determine the scaling factor based on a scaling factor for a second level of detail different from the first level of detail.

[0180] Clause 40B. A device pursuant to Clause 31B, wherein the first color component includes a green component and the second color component includes a red component.

[0181] Clause 41B. A device pursuant to Clause 31B, wherein the first color component includes a green component and the second color component includes a blue component.

[0182] Clause 42B. The device pursuant to Clause 31B also includes: a sensor for generating point cloud data.

[0183] Clause 1C. An apparatus for decoding encoded point cloud data, the apparatus comprising: a memory configured to store the encoded point cloud data; and one or more processors implemented in circuitry and configured to: determine, for points in the point cloud, a first color value for a first color component based on a first predicted value and a first residual value; apply a scaling factor to the first residual value to determine a predicted second residual value, wherein the scaling factor has one or both of an absolute value greater than one or a non-integer value; receive, for points in the point cloud, a second residual value in the encoded point cloud data; determine a final second residual value based on the predicted second residual value and the received second residual value; and determine, for points in the point cloud, a second color value for a second color component based on the second predicted value and the final second residual value.

[0184] Clause 2C. The device according to Clause 1C, wherein the one or more processors are further configured to: receive information in the point cloud data indicating a scalar value, wherein the scalar value is an integer value, and wherein, in order to apply the scaling factor, the one or more processors are further configured to perform a multiplication of the first residual value with the scalar value.

[0185] Clause 3C. A device according to Clause 2C, wherein, in order to apply the scaling factor, the one or more processors are further configured to perform a bit shift operation including a right shift of 2.

[0186] Clause 4C. Equipment pursuant to Clause 2C or 3C, wherein the information indicating the scalar value includes a difference corresponding to the difference between the scalar value and a previously determined scalar value.

[0187] Clause 5C. A device according to Clause 4C, wherein the point cloud includes a point cloud for a first level of detail and a previously determined scalar value is used for a second level of detail different from the first level of detail.

[0188] Clause 6C. A device pursuant to any of Clauses 2C-5C, wherein, in order to receive information indicating the scalar value, the one or more processors are further configured to receive information in an attribute slice header syntax structure indicating the scalar value.

[0189] Clause 7C. An apparatus pursuant to any of Clauses 1C-6C, wherein the one or more processors are further configured to: determine a second predicted value for a point in the point cloud based on the values ​​of different points in the point cloud.

[0190] Clause 8C. An apparatus pursuant to any of Clauses 1C-7C, wherein the point cloud comprises a point cloud for a first level of detail, and wherein, in order to determine the scaling factor, the one or more processors are further configured to determine the scaling factor based on a scaling factor for a second level of detail different from the first level of detail.

[0191] Clause 9C. A device pursuant to any of Clauses 1C-8C, wherein the first color component comprises a green component and the second color component comprises a red component.

[0192] Clause 10C. A device pursuant to any of Clauses 1C-8C, wherein the first color component comprises a green component and the second color component comprises a blue component.

[0193] Clause 11C. An apparatus pursuant to any of Clauses 1C-10C, wherein the one or more processors are further configured to: apply a second scaling factor to the first residual value to determine a predicted third residual value, wherein the second scaling factor has one or both of an absolute value greater than one or a non-integer value; for points in the point cloud, receive the third residual value in the encoded point cloud data; determine a final third residual value based on the predicted third residual value and the received third residual value; and for points in the point cloud, determine a third color value for a third color component based on the third predicted value and the final third residual value.

[0194] Clause 12C. A device pursuant to any of Clauses 1C-12C, wherein the one or more processors are further configured to reconstruct the point cloud.

[0195] Clause 13C. A device according to Clause 12C, wherein the one or more processors are further configured to generate a map of the building's interior based on the reconstructed point cloud.

[0196] Clause 14C. In a device pursuant to Clause 12C, one or more processors are further configured to perform autonomous navigation operations based on the reconstructed point cloud.

[0197] Clause 15C. A device according to Clause 12C, wherein the one or more processors are further configured to generate computer graphics based on a reconstructed point cloud.

[0198] Clause 16C. A device according to Clause 12C, wherein the one or more processors are configured to: determine the position of a virtual object based on the reconstructed point cloud; and generate an extended reality (XR) visualization of the virtual object at the determined position.

[0199] Clause 17C. An apparatus pursuant to any of Clauses 12C-16C further includes a display for rendering an image based on the reconstructed point cloud.

[0200] Clause 18C. Devices pursuant to any of Clauses 12C, 14C, or 17C, wherein the device includes one or more of a mobile phone, tablet computer, vehicle, or extended reality device.

[0201] Clause 19C. A method for decoding encoded point cloud data, the method comprising: for points in the point cloud, determining a first color value for a first color component based on a first predicted value and a first residual value; applying a scaling factor to the first residual value to determine a predicted second residual value, wherein the scaling factor has one or both of an absolute value greater than one or a non-integer value; for points in the point cloud, receiving a second residual value in the encoded point cloud data; determining a final second residual value based on the predicted second residual value and the received second residual value; and for points in the point cloud, determining a second color value for a second color component based on the second predicted value and the final second residual value.

[0202] Clause 20C. The method according to Clause 19C further includes: receiving information in the point cloud data indicating a scalar value, wherein the scalar value is an integer value, and wherein applying the scaling factor includes performing a multiplication of the first residual value with the scalar value.

[0203] Clause 21C. A device according to Clause 20C, wherein applying the scaling factor includes performing a bit shift operation including a right shift of 2.

[0204] Clause 22C. For devices pursuant to Clause 20C or 21C, the information indicating the scalar value includes a difference corresponding to the difference between the scalar value and a previously determined scalar value.

[0205] Clause 23C. A device according to Clause 22C, wherein the point cloud comprises a point cloud for a first level of detail and the previously determined scalar value is for a second level of detail different from the first level of detail.

[0206] Clause 24C. An apparatus pursuant to any of Clauses 20C-23C, wherein information indicating a scalar value is received includes information indicating a scalar value in the Receive Attribute Slice Header syntax structure.

[0207] Clause 25C. The method pursuant to any of Clauses 19C-24C further includes: determining a second predicted value for a point in the point cloud based on the values ​​of different points in the point cloud.

[0208] Clause 26C. The method pursuant to any of Clauses 19C-25C, wherein determining the scaling factor includes determining the scaling factor based on the scaling factor used for the second level of detail.

[0209] Clause 27C. A method pursuant to any of Clauses 19C-26C, wherein the first color component comprises a green component and the second color component comprises a red component.

[0210] Clause 28C. A method pursuant to any of Clauses 19C-26C, wherein the first color component comprises a green component and the second color component comprises a blue component.

[0211] Clause 29C. The method according to any one of Clauses 19C-28C further includes: applying a second scaling factor to the first residual value to determine a predicted third residual value, wherein the second scaling factor has one or both of an absolute value greater than one or a non-integer value; receiving the third residual value in the encoded point cloud data for a point in the point cloud; determining a final third residual value based on the predicted third residual value and the received third residual value; and determining a third color value for a third color component for a point in the point cloud based on the third predicted value and the final third residual value.

[0212] Clause 30C. A computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to: determine a first color value for a first color component based on a first predicted value and a first residual value for points in a point cloud; apply a scaling factor to the first residual value to determine a predicted second residual value, wherein the scaling factor has one or both of an absolute value greater than one or a non-integer value; receive, for points in the point cloud, a second residual value in encoded point cloud data; determine a final second residual value based on the predicted second residual value and the received second residual value; and determine a second color value for a second color component based on the second predicted value and the final second residual value for points in the point cloud.

[0213] Clause 31C. An apparatus for encoding point cloud data, the apparatus comprising: a memory configured to store the encoded point cloud data; and one or more processors implemented in circuitry and configured to: determine, for points in the point cloud, a first color value of a first color component; encode the first color value of the first color component into a first predicted value and a first residual value; for points in the point cloud, determine a second color value of a second color component; determine a second predicted value of the second color value of the second color component; determine a second residual value of the second color value of the second color component; apply a scaling factor to the first residual value to determine a predicted second residual value, wherein the scaling factor has one or both of an absolute value greater than one or a non-integer value; and generate a syntax element representing the difference between the second residual value and the predicted second residual value to be included in the point cloud data.

[0214] Clause 32C. A device according to Clause 31C, wherein the one or more processors are further configured to: generate information indicating a scalar value to be included in the point cloud data, wherein the scalar value is an integer value, and wherein, in order to apply the scaling factor, the one or more processors are further configured to perform a multiplication of the first residual value with the scalar value.

[0215] Clause 33C. A device according to Clause 32C, wherein, in order to apply the scaling factor, the one or more processors are further configured to perform a bit shift operation including a right shift of 2.

[0216] Clause 34C. For devices pursuant to Clause 32C or 33C, the information indicating the scalar value includes a difference corresponding to the difference between the scalar value and a previously determined scalar value.

[0217] Clause 35C. A device according to Clause 34C, wherein the point cloud comprises a point cloud for a first level of detail and the previously determined scalar value is for a second level of detail different from the first level of detail.

[0218] Clause 36C. A device pursuant to any of Clauses 32C-35C, wherein, in order to include information indicating the scalar value in the point cloud data, the one or more processors are further configured to include information indicating the scalar value in an attribute slice header syntax structure.

[0219] Clause 37C. A device pursuant to any of Clauses 32C-36C, wherein, in order to generate the scalar value, the one or more processors are further configured to include the scalar value in an attribute slice header syntax structure.

[0220] Clause 38C. An apparatus pursuant to any of Clauses 31C-37C, wherein the one or more processors are further configured to: determine a second predicted value for a point in the point cloud based on the values ​​of different points in the point cloud.

[0221] Clause 39C. An apparatus pursuant to any of Clauses 31C-38C, wherein the point cloud comprises a point cloud for a first level of detail, and wherein, in order to determine the scaling factor, the one or more processors are further configured to determine the scaling factor based on a scaling factor for a second level of detail different from the first level of detail.

[0222] Clause 40C. A device pursuant to any of Clauses 31C-39C, wherein the first color component comprises a green component and the second color component comprises a red component.

[0223] Clause 41C. A device pursuant to any of Clauses 31C-39C, wherein the first color component comprises a green component and the second color component comprises a blue component.

[0224] Clause 42C. Devices pursuant to any of Clauses 31C-41C also include sensors for generating point cloud data.

[0225] Examples in various aspects of this disclosure may be used individually or in any combination.

[0226] It should be recognized that, depending on the example, certain actions or events of any of the techniques described herein may be performed in a different sequence, and may be added, combined, or omitted together (e.g., not all described actions or events are necessary for the practice of the technique). Furthermore, in some examples, actions or events may be performed concurrently, for example, through multithreaded processing, interrupt handling, or multiple processors, rather than sequentially.

[0227] In one or more examples, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, these functions may be stored on or transmitted via a computer-readable medium as one or more instructions or code, and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium corresponding to a tangible medium such as a data storage medium, or a communication medium that includes, for example, any medium facilitating the transfer of a computer program from one place to another according to a communication protocol. In this manner, a computer-readable medium may generally correspond to (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium such as a signal or carrier wave. A data storage medium may be any available medium accessible by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this disclosure. Computer program products may include computer-readable media.

[0228] By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage, disk storage or other magnetic storage devices, flash memory, or any other medium that can be used to store required program code in the form of instructions or data structures and that can be accessed by a computer. Furthermore, any connection is properly referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies (such as infrared, radio, and microwave), then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies (such as infrared, radio, and microwave) are included in the definition of medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but rather refer to non-transient tangible storage media. As used herein, disks and optical discs include compact optical discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, where disks generally reproduce data magnetically, while optical discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0229] Instructions can be executed by one or more processors, such as one or more DSPs, general-purpose microprocessors, ASICs, FPGAs, or other equivalent integrated or discrete logic circuits. Therefore, the terms "processor" and "processing circuit" as used herein can refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described herein can be provided in dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into combined codecs. Moreover, these techniques can be implemented entirely within one or more circuit or logic elements.

[0230] The techniques disclosed herein can be implemented in a variety of devices or apparatuses, including wireless mobile phones, integrated circuits (ICs), or IC sets (e.g., chipsets). Various components, modules, or units are described in this disclosure to highlight functional aspects of a device configured to perform the disclosed techniques, but implementation by different hardware units is not necessarily required. Rather, as described above, various units may be combined with suitable software and / or firmware within a codec hardware unit, or provided by a set of interoperable hardware units including one or more processors as described above.

[0231] Various examples have been described. These and other examples are all within the scope of the appended claims.

Claims

1. A device for decoding encoded point cloud data, the device comprising: a memory configured to store the encoded point cloud data; and one or more processors implemented in circuitry and configured to: determine, for a point in a point cloud, a first attribute value of a first component based on a first predicted value and a first residual value; receive, in the encoded point cloud data, a first syntax element indicative of a scalar value, wherein the scalar value is an integer value; apply a scaling factor to the first residual value to determine a predicted second residual value, wherein the scaling factor has a non-integer value, and wherein applying the scaling factor to the first residual value, the one or more processors are further configured to: perform a multiplication by the scalar value; and perform a right shift operation, wherein a result of the multiplication by the scalar value and the right shift operation in combination is equivalent to the scaling factor; receive, in the encoded point cloud data, a second syntax element; determine, for the point in the point cloud, a second residual value based on the second syntax element; and determine, for the point in the point cloud, a second attribute value of a second component based on a second predicted value, the second residual value, and the predicted second residual value.

2. The device of claim 1, wherein to perform the right shift operation includes the one or more processors being further configured to perform a bit shift operation that includes a right shift 2.

3. The device of claim 1, wherein the first syntax element comprises: a difference value corresponding to a difference between the scalar value and a previously determined scalar value.

4. The device of claim 1, wherein the point cloud comprises: the point cloud is for a first level of detail, and the previously determined scalar value is for a second level of detail different than the first level of detail.

5. The device of claim 1, wherein to receive the first syntax element, the one or more processors are further configured to receive the first syntax element in an attribute slice header syntax structure.

6. The device of claim 1, wherein the one or more processors are further configured to determine the second predicted value for the point in the point cloud based on values of different points in the point cloud.

7. The device of claim 1, wherein the point cloud comprises a point cloud for a first level of detail, and wherein to determine the scaling factor, the one or more processors are further configured to determine the scaling factor based on a scaling factor for a second level of detail different than the first level of detail.

8. The device of claim 1, wherein the first component comprises a green component and the second component comprises a red component.

9. The device of claim 1, wherein the first component comprises a green component and the second component comprises a blue component.

10. The device of claim 1, wherein the one or more processors are further configured to: apply a second scaling factor to the first residual value to determine a predicted third residual value, wherein the second scaling factor has one or both of an absolute value greater than one or a non-integer value; receive, in the encoded point cloud data, a third syntax element; determine, for the point in the point cloud, a third residual value based on the third syntax element; and determining, for the point in the point cloud, a third attribute value of a third component based on a third prediction value, the third residual value, and the predicted third residual value.

11. The device of claim 1, wherein the one or more processors are further configured to: reconstruct the point cloud.

12. The device of claim 11, wherein the one or more processors are further configured to: generate a map of an interior of a building based on the reconstructed point cloud.

13. The device of claim 11, wherein the one or more processors are further configured to: perform an autonomous navigation operation based on the reconstructed point cloud.

14. The device of claim 11, wherein the one or more processors are further configured to: generate computer graphics based on the reconstructed point cloud.

15. The device of claim 11, wherein the one or more processors are further configured to: determine a position of a virtual object based on the reconstructed point cloud; and generate an extended reality (XR) visualization with the virtual object at the determined position.

16. The apparatus of claim 10, further comprising: a display for presenting images based on the reconstructed point cloud.

17. The apparatus of claim 1, wherein the apparatus comprises: one or more of a mobile phone, a tablet computer, a vehicle, or an extended reality device.

18. A method of decoding encoded point cloud data, the method comprising: determining, for a point in a point cloud, a first attribute value of a first component based on a first prediction value and a first residual value; receiving, in the encoded point cloud data, a first syntax element indicative of a scalar value, wherein the scalar value is an integer value; applying a scaling factor to the first residual value to determine a predicted second residual value, wherein the scaling factor has a non-integer value, and wherein applying the scaling factor to the first residual value comprises: performing a multiplication with the scalar value; and performing a right shift operation, wherein a result of a combination of the multiplication with the scalar value and the right shift operation is equivalent to the scaling factor; receiving, in the encoded point cloud data, a second syntax element; determining, for the point in the point cloud, a second residual value based on the second syntax element; and determining, for the point in the point cloud, a second attribute value of a second component based on a second prediction value, the second residual value, and the predicted second residual value.

19. The method of claim 18, wherein performing the right shift operation comprises: performing a bit shift operation comprising a right shift 2.

20. The method of claim 18, wherein the point cloud comprises: the point cloud is for a first level of detail, and the previously determined scalar value is for a second level of detail different than the first level of detail.

21. The method of claim 18, wherein receiving the first syntax element comprises: receiving the first syntax element in an attribute slice header syntax structure.

22. The method of claim 18, further comprising: determining the second prediction value for the point in the point cloud based on values of different points in the point cloud.

23. The method of claim 18, further comprising: determining the scaling factor based on a scaling factor for a second level of detail.

24. The method of claim 18, wherein the first component comprises a green component and the second component comprises a red component.

25. The method of claim 18, wherein the first component comprises a green component and the second component comprises a blue component.

26. The method of claim 18, further comprising: applying a second scaling factor to the first residual value to determine a predicted third residual value, wherein the second scaling factor has one or both of an absolute value greater than one or a non-integer value; receiving, in the encoded point cloud data, a third syntax element; determining, for the point in the point cloud, a third residual value based on the third syntax element; and determining, for the point in the point cloud, a third attribute value of a third component based on a third predicted value, the third residual value, and the predicted third residual value.

27. A computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to: determine, for a point in a point cloud of encoded point cloud data, a first attribute value of a first component based on a first predicted value and a first residual value; receive, in the encoded point cloud data, a first syntax element indicating a scalar value, wherein the scalar value is an integer value; apply a scaling factor to the first residual value to determine a predicted second residual value, wherein the scaling factor has a non-integer value, and wherein applying the scaling factor to the first residual value, the instructions cause the one or more processors to: perform a multiplication by the scalar value; and perform a right-shift operation, wherein a result of the multiplication by the scalar value and the right-shift operation in combination is equivalent to the scaling factor; receive, in the encoded point cloud data, a second syntax element; determine, for the point in the point cloud, a second residual value based on the second syntax element; and determine, for the point in the point cloud, a second attribute value of a second component based on a second predicted value, the second residual value, and the predicted second residual value.

28. A device for encoding point cloud data, the device comprising: a memory configured to store encoded point cloud data; and one or more processors implemented in circuitry and configured to: determine, for a point in a point cloud, a first attribute value of a first component; encode the first attribute value of the first component as a first predicted value and a first residual value; determine, for the point in the point cloud, a second attribute value of a second component; determine a second predicted value of the second attribute value of the second component; determine a second residual value of the second attribute value of the second component; apply a scaling factor to the first residual value to determine a predicted second residual value, wherein the scaling factor has a non-integer value, and wherein applying the scaling factor to the first residual value, the one or more processors are further configured to: perform a multiplication by a scalar value; and perform a right-shift operation, wherein a result of the multiplication by the scalar value and the right-shift operation in combination is equivalent to the scaling factor; generate, in the encoded point cloud data, a first syntax element indicating the scalar value, wherein the scalar value is an integer value; and generate a second syntax element representing a difference between the second residual value and the predicted second residual value to include in the encoded point cloud data. ​ ​ 29. The device of claim 28, wherein to perform the right shift operation, the one or more processors are further configured to perform a bit shift operation comprising a right shift by 2.

30. The device of claim 28, wherein the point cloud comprises a point cloud for a first level of detail, and the previously determined scalar value is for a second level of detail different than the first level of detail.

31. The device of claim 28, wherein to include the first syntax element in the encoded point cloud data, the one or more processors are further configured to include the first syntax element in an attribute slice header syntax structure.

32. The device of claim 28, wherein to generate the scalar value, the one or more processors are further configured to include the scalar value in an attribute slice header syntax structure.

33. The device of claim 28, wherein the one or more processors are further configured to determine the second prediction value for the point in the point cloud based on values for different points in the point cloud.

34. The device of claim 28, wherein the point cloud comprises: for a first level of detail, and wherein to determine the scaling factor, the one or more processors are further configured to determine the scaling factor based on a scaling factor for a second level of detail different than the first level of detail.

35. The device of claim 28, wherein the first component comprises a green component and the second component comprises a red component.

36. The device of claim 28, wherein the first component comprises a green component and the second component comprises a blue component.

37. The device of claim 28, further comprising: a sensor for generating the encoded point cloud data.

38. A computer program product comprising computer readable instructions, which, when executed by a processor, cause the processor to perform the method of any of claims 18-26.

Images