Adaptive mode selection for point cloud compression

Abstract

An electronic device and method for adaptive mode selection for point cloud compression is provided. The electronic device receives a 3D point cloud geometry and divides the 3D point cloud geometry into a set of 3D blocks. Mode decision information is determined for a 3D block of the set of 3D blocks. The mode decision information includes class information of the 3D point cloud geometry, an operating condition associated with an encoding stage of the 3D point cloud geometry, or mode related information associated with one or more 3D blocks of the set of 3D blocks. Based on the mode decision information, one or more modes are selected for the 3D block from a plurality of modes. Each mode corresponds to a function used to encode the 3D block. The 3D block is encoded based on the one or more modes.

Description

[Technical Field]

[0001] [CROSS-REFERENCE / INCORPORATION BY REFERENCE TO RELATED APPLICATIONS]

[0001] This application claims the benefit of priority to U.S. Patent Application No. 17 / 739,484, filed with the U.S. Patent and Trademark Office on May 9, 2021, which claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 262,135, filed on October 5, 2021, the entire contents of which are incorporated herein by reference.

[0002] Various embodiments of the present disclosure relate to three-dimensional (3D) point cloud compression (PCC). More particularly, various embodiments of the present disclosure relate to adaptive mode selection for point cloud compression. [Background technology]

[0003] Advances in the field of three-dimensional (3D) scanning have led to the ability to create 3D geometric representations of 3D objects. 3D point clouds are an example of 3D geometric representations, which have been adopted for different applications, such as free-viewpoint viewing for sports or live event broadcasts, geographic information systems, cultural heritage reconstruction, or autonomous vehicle navigation. Typically, a 3D point cloud includes a large number of unstructured 3D points (e.g., each point has X, Y, and Z coordinates) and associated attributes (e.g., texture, including color or reflectance). A 3D image of an object represented by a 3D point cloud may contain a significant amount of raw data, making data storage, processing, and transmission challenging tasks for any practical application. Therefore, it may be desirable to have an efficient point cloud compression (PCC) technique. Some conventional PCC techniques require multiple encoding and decoding operations to compress even a portion of a 3D point cloud. Such techniques can be computationally expensive and may constrain available memory resources for computation.

[0004]

[0004] Further limitations and disadvantages of conventional methods will become apparent to those skilled in the art by comparing the described system with certain aspects of the present disclosure illustrated in the remainder of this application with reference to the drawings. Summary of the Invention [Problem to be solved by the invention]

[0005]

[0005] An electronic device and method for adaptive mode selection for point cloud compression is provided, as substantially shown in and / or described in connection with at least one of the figures and more fully set forth in the claims.

[0006]

[0006] These and other features and advantages of the present disclosure can be understood by considering the following detailed description of the disclosure in conjunction with the accompanying drawings in which like elements are designated by like reference numerals throughout. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram illustrating an example environment for adaptive mode selection for point cloud compression, according to an embodiment of the present disclosure. [Figure 2] FIG. 2 is a block diagram illustrating the example electronic device of FIG. 1, according to an embodiment of the present disclosure. [Figure 3] 1A and 1B are block diagrams of an example encoder and an example decoder for adaptive mode selection for point cloud compression, according to an embodiment of the present disclosure. [Figure 4] FIG. 1 illustrates an exemplary processing pipeline for adaptive mode selection for point cloud compression, according to an embodiment of the present disclosure. [Figure 5] FIG. 1 illustrates an exemplary processing pipeline for adaptive mode selection for point cloud compression, according to an embodiment of the present disclosure. [Figure 6] FIG. 1 illustrates an exemplary processing pipeline for adaptive mode selection for point cloud compression, according to an embodiment of the present disclosure. [Figure 7] FIG. 1 illustrates an exemplary processing pipeline for adaptive mode selection for point cloud compression, according to an embodiment of the present disclosure. [Figure 8A] FIG. 1 illustrates an exemplary processing pipeline for adaptive mode selection for point cloud compression, according to an embodiment of the present disclosure. [Figure 8B] FIG. 1 illustrates an exemplary processing pipeline for adaptive mode selection for point cloud compression, according to an embodiment of the present disclosure. [Figure 9A] FIG. 1 illustrates an exemplary processing pipeline for adaptive mode selection for point cloud compression, according to an embodiment of the present disclosure. [Figure 9B] FIG. 1 illustrates an exemplary processing pipeline for adaptive mode selection for point cloud compression, according to an embodiment of the present disclosure. [Figure 10] 10 is a flowchart illustrating an example operation for adaptive mode selection for point cloud compression, according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0008]

[0019] Implementations described below can be found in the disclosed electronic device and method for adaptive mode selection for point cloud compression. An exemplary embodiment of the electronic device can include circuitry that can be configured to receive a three-dimensional (3D) point cloud geometry of a 3D point cloud associated with at least one object in 3D space. The electronic device can be configured to divide the 3D point cloud geometry into a set of 3D blocks and determine mode decision information for the 3D blocks of the set of 3D blocks. The mode decision information can include, for example, class information associated with the 3D point cloud geometry, one or more operational conditions associated with an encoding stage of the 3D point cloud geometry, or mode-related information associated with one or more 3D blocks of the set of 3D blocks. The electronic device can be further configured to select one or more modes for the 3D blocks from a plurality of modes based on the mode decision information. Each mode of the plurality of modes can correspond to a function that can be used to encode the 3D block. For example, the function can correspond to a deep neural network (DNN) model that can be trained to encode the 3D blocks of the 3D point cloud geometry to generate the encoded 3D blocks. Each mode of the plurality of modes may correspond, for example, to an alpha parameter of a focal loss function used in a training phase of the DNN model. The focal loss function may be configured to penalize the removal of non-empty voxels from a 3D block of the 3D point cloud geometry. The electronic device may be further configured to encode the 3D block based on the selected one or more modes.

[0009]

[0020] Some conventional point cloud compression (PCC) techniques require a full mode search operation to find the optimal mode for encoding each 3D block of a 3D point cloud geometry. In the full mode search operation, the 3D block can be encoded based on each possible mode, and the encoded 3D block can be reconstructed (i.e., decoded) to determine the rate-distortion costs for all possible modes. The mode corresponding to the lowest rate-distortion cost can be selected as the optimal mode associated with the 3D block. Finally, the 3D block can be encoded based on the selected optimal mode. Because determining the rate-distortion cost may require multiple encoding and decoding operations per block, conventional PCC techniques can be computationally expensive and may have high memory usage during the encoding phase of the point cloud geometry.

[0010]

[0021] In contrast, the disclosed technique for point cloud compression does not require a full mode search operation to determine the optimal mode for encoding each 3D block of a 3D point cloud geometry. Instead of encoding each 3D block using all possible modes (i.e., a full mode search), the disclosed technique uses mode decision information to select one or more modes and then searches for the optimal mode from the selected one or more modes. The number of encoding and decoding operations required to determine the final mode for encoding each 3D block can be significantly reduced compared to a full mode search for each 3D block. Therefore, the disclosed technique for point cloud compression is less computationally intensive and requires fewer memory resources than conventional PCC techniques.

[0011]

[0022] FIG. 1 is a block diagram illustrating an exemplary environment for adaptive mode selection for point cloud compression, according to an embodiment of the present disclosure. Referring to FIG. 1, a network environment 100 is shown. The network environment 100 may include an electronic device 102, a server 104, a scanning setup 106, and multiple external devices 108A...108N. The scanning setup 106 may include one or more image sensors (not shown) and one or more depth sensors (not shown) associated with the one or more image sensors. The electronic device 102 may be communicatively coupled to the server 104, the scanning setup 106, and the multiple external devices 108A...108N via a communications network 110. Also shown is a three-dimensional (3D) point cloud geometry 112 of a 3D point cloud associated with at least one object (e.g., a person) in 3D space.

[0012]

[0023] The electronic device 102 may include suitable logic, circuitry, interfaces, and / or code that can be configured to encode and / or decode 3D point cloud geometry (e.g., 3D point cloud geometry 112). A 3D point cloud may include multiple points that together represent an object in 3D space. As an example, each point may be represented as (x, y, z, r, g, b, α), where (x, y, z) represent the 3D coordinates of the point on the object, (r, g, and b) represent the red, green, and blue values ​​of the point, and (α) represents the transparency value of the point. In some embodiments, the electronic device 102 may be configured to generate a 3D point cloud of an object or multiple objects (e.g., a 3D scene including objects in the foreground and background). The electronic device 102 may obtain the 3D point cloud geometry 112 of the object (or multiple objects) from the 3D point cloud. Examples of electronic devices 102 may include, but are not limited to, computing devices, video conferencing systems, augmented reality (AR) devices, virtual reality (VR) devices, mixed reality (MR) devices, game consoles, smart wearable devices, mainframe machines, servers, computer workstations, and / or consumer electronics (CE) devices.

[0013]

[0024] The server 104 may include suitable logic, circuitry, interfaces, and / or code that may be configured to generate a 3D point cloud of an object and store 3D point cloud geometry 112 and information associated with the 3D point cloud. Examples of the server 104 may include, but are not limited to, an application server, a cloud server, a web server, a database server, a file server, a game server, a mainframe server, or combinations thereof.

[0014]

[0025] The scanning setup 106 may include suitable logic, circuitry, interfaces, and / or code that can be configured to scan a 3D environment containing an object and generate a raw 3D scan (also referred to as a raw 3D point cloud). According to certain embodiments, the scanning setup 106 may include a single image capture device or multiple image capture devices (positioned at multiple viewpoints) to capture multiple color images. In certain examples, the scanning setup 106 may include an additional depth sensor to capture depth information of the object. The multiple color images and depth information of the object may be captured from different viewpoints. In such cases, a 3D point cloud may be generated based on the captured multiple color images and corresponding depth information of the object.

[0015]

[0026] According to an embodiment, the scanning setup 106 can be configured to perform a 3D scan of an object in 3D space and generate a dynamic 3D point cloud (i.e., a point cloud sequence) that can capture different attributes and changes in the geometry of the 3D points at different time steps. The scanning setup 106 can be configured to communicate the generated 3D point cloud, the plurality of color images, and / or the corresponding depth information to the electronic device 102 via the communication network 110.

[0016]

[0027] According to an embodiment, the scanning setup 106 may include multiple sensors, such as a combination of a depth sensor and a color sensor (such as a red-green-blue (RGB) sensor) and / or a combination of an infrared (IR) projector and an IR sensor. For example, the depth sensor may capture information associated with the point cloud geometry (e.g., the 3D position of the points), and the RGB and IR sensors may capture information associated with the point cloud attributes (e.g., color and temperature). In an embodiment, the IR projector and IR sensor may be used to estimate depth information. The combination of the depth sensor, RGB, and IR sensors may be used to capture a point cloud frame (a single static point cloud) or multiple point cloud frames (3D video) along with associated geometry and attributes.

[0017]

[0028] According to an embodiment, the scanning setup 106 can include an active 3D scanner that relies on radiation or light to capture the 3D structure of an object in 3D space. The scanning setup 106 can also include an image sensor that can capture color information associated with the object. For example, the active 3D scanner can be a time-of-flight (TOF)-based 3D laser scanner, a laser range finder, a TOF camera, a handheld laser scanner, a structured light 3D scanner, a modulated light 3D scanner, a CT scanner that outputs point cloud data, an airborne Light Detection and Ranging (LiDAR) scanner, a 3D LiDAR, a 3D motion sensor, etc.

[0018]

[0029] 1, scanning setup 106 is shown as separate from electronic device 102. However, the present disclosure is not so limited, and in some embodiments, scanning setup 106 may be integrated into electronic device 102. Alternatively, the entire functionality of scanning setup 106 may be incorporated into electronic device 102 without departing from the scope of the present disclosure. Examples of scanning setup 106 may include, but are not limited to, a depth sensor, an RGB sensor, an IR sensor, an image sensor, and / or a motion detector device.

[0019]

[0030] Each of the plurality of external devices 108A...108N may include suitable logic, circuitry, interfaces, and / or code that may be configured to communicate with the electronic device 102 via the communications network 110. According to an embodiment, each of the plurality of external devices 108A...108N may be configured to receive encoded 3D point cloud geometry (as multimedia content) from the electronic device 102 via the communications network 110. The plurality of external devices 108A...108N may be configured to decode the encoded 3D point cloud geometry to render a 3D model of the object. Examples of the plurality of external devices 108A...108N may include, but are not limited to, a display system, a computing device, a gaming device, a mobile phone, a television, or an electronic device capable of storing or rendering multimedia content.

[0020]

[0031] The communication network 110 can include a communication medium through which the electronic device 102 can be communicatively coupled to the server 104, the scanning setup 106, and multiple external devices 108A...108N. Examples of the communication network 110 can include, but are not limited to, the Internet, a cloud network, a cellular or wireless mobile network (such as Long-Term Evolution and 5G New Radio), a Wireless Fidelity (Wi-Fi) network, a personal area network (PAN), a local area network (LAN), or a metropolitan area network (MAN). The various devices in the network environment 100 can be configured to connect to the communication network 110 according to various wired and wireless communication protocols. Examples of such wired and wireless communication protocols may include, but are not limited to, at least one of Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), ZigBee, EDGE, IEEE 802.11, Light Fidelity (Li-Fi), 802.16, IEEE 802.11s, IEEE 802.11g, multi-hop communication, wireless access point (AP), device-to-device communication, cellular communication protocols, and Bluetooth (BT) communication protocols.

[0021]

[0032] In operation, the electronic device 102 can be configured to receive 3D point cloud geometry 112 associated with at least one object in 3D space. For example, the 3D point cloud data can be obtained from a 3D point cloud (or a 3D scan) that includes geometry and attributes. In one embodiment, the 3D point cloud can be a static point cloud. In another embodiment, the 3D point cloud can be a frame of a dynamic 3D point cloud (i.e., a point cloud sequence). In general, a 3D point cloud is a representation of geometric information (e.g., 3D coordinates of points) and attribute information of an object in 3D space. The attribute information can include, but is not limited to, color information, reflectance information, opacity information, normal vector information, material identifier information, or texture information associated with the object in 3D space. The texture information can represent the spatial arrangement of colors or intensities in a multi-color image of the object. The reflectance information can represent information associated with an empirical model of local illumination (e.g., a Phong shading model or a Gouraud shading model) of feature points of the 3D point cloud. The empirical model of local lighting can correspond to the reflectance of the object's surface (rough or shiny surface areas). The opacity information can represent the transparency of a point. The normal vector information can represent the direction perpendicular to the tangent plane at the point. The material identifier information can represent an identifier that indicates a material from a set of available materials.

[0022]

[0033] The electronic device 102 may be further configured to generate a plurality of voxels from the 3D point cloud geometry 112. The generation of voxels may be referred to as voxelizing the 3D point cloud geometry 112. Conventional techniques for voxelizing 3D point clouds may be well known to those skilled in the art. Accordingly, details of voxelization are omitted from this disclosure for the sake of brevity.

[0023]

[0034] The 3D point cloud geometry 112 is a set of a large number (e.g., 10 4 Because a 3D point cloud geometry 112 may contain data points (on the order of 1000 or more), transmitting / receiving the data points may consume high network bandwidth. Similarly, the data points may consume more storage than necessary in their uncompressed state. The 3D point cloud geometry 112 may need to be encoded so that the encoded point cloud data uses less network bandwidth for transmission / reception and requires less storage space. The encoding process for the 3D point cloud geometry 112 is described herein.

[0024]

[0035] The electronic device 102 may be configured to divide the 3D point cloud geometry 112 into a set of 3D blocks. The electronic device 102 may be configured to determine mode decision information for a 3D block of the set of 3D blocks. The mode decision information may include, for example, class information associated with the 3D point cloud geometry 112, one or more operating conditions associated with an encoding stage of the 3D point cloud geometry 112, or mode-related information associated with one or more 3D blocks of the set of 3D blocks. For example, the class information associated with the 3D point cloud geometry may include a geometry bit depth, density, or point distribution associated with the 3D point cloud geometry 112. Similarly, the one or more operating conditions associated with an encoding stage of the 3D point cloud geometry 112 may include a target rate-distortion cost associated with the 3D point cloud geometry 112.

[0025]

[0036] The electronic device 102 may be further configured to select one or more modes for the 3D block from the plurality of modes based on the mode decision information. Each mode may correspond to a function capable of encoding the 3D block. For example, the function may correspond to a deep neural network (DNN) model that can be trained to encode the 3D block of the 3D point cloud geometry 112 to generate the encoded 3D block. According to an embodiment, each mode may correspond to an alpha parameter of a focal loss function used in a training phase of the DNN model. The focal loss function may be configured to penalize the removal of non-empty voxels from the 3D block of the 3D point cloud geometry 112. The electronic device 102 may include multiple DNN models, each of which may be configured for a particular mode of the plurality of modes. For example, a first DNN model of the multiple DNN models may correspond to a first mode associated with a first value of the alpha parameter. Similarly, a second DNN model of the multiple DNN models may correspond to a second mode associated with a second value of the alpha parameter, etc. Selection of one or more modes is further described, for example, in Figures 4, 5, 6, 7, 8A, 8B, 9A, and 9B.

[0026]

[0037] The electronic device 102 may be further configured to encode the 3D block based on the selected one or more modes. In some embodiments, the electronic device 102 may determine an optimal mode for encoding the 3D block from the selected one or more modes. In certain scenarios, the selected one or more modes may include a single mode. In such scenarios, the optimal mode may be the same as the selected mode. Determining the optimal mode is further described, for example, in Figures 4, 5, 6, 7, 8A, 8B, 9A, and 9B.

[0027]

[0038] The electronic device 102 may also determine a mode(s) for all remaining blocks of the 3D point cloud geometry 112 and may encode the remaining blocks of the 3D point cloud geometry 112 to generate an encoded 3D point cloud geometry (which may include encoded 3D blocks).

[0028]

[0039] In an embodiment, the electronic device 102 can generate supplemental information associated with the encoded 3D point cloud geometry. Examples of the supplemental information can include, but are not limited to, coding tables, mode decision information, weight information, index values ​​for geometric information, and quantization parameters. The electronic device 102 can be further configured to transmit the encoded 3D point cloud geometry to at least one of the plurality of external devices 108A...108N. The supplemental information can be transmitted together with the encoded 3D point cloud geometry.

[0029]

[0040] Conventional point cloud compression (PCC) techniques require a mode search operation to search for an optimal mode for encoding each 3D block of the 3D point cloud geometry 112. In the mode search operation, each 3D block may be encoded based on all possible modes, and the encoded 3D block may be reconstructed (i.e., decoded) to determine a rate-distortion cost for each possible mode. The mode corresponding to the lowest rate-distortion cost may be selected as the optimal mode associated with the 3D block. The 3D block may be encoded based on the selected optimal mode. Because determining the rate-distortion cost may require multiple encoding and decoding operations for each 3D block of the 3D point cloud geometry 112, conventional PCC techniques may be computationally expensive and may require high memory usage compared to desired usage. In contrast, the disclosed electronic device 102 may select one or more modes from multiple possible modes (i.e., multiple modes) to encode a 3D block of the 3D point cloud geometry 112. Because not all of the multiple possible modes have a significant effect on the rate-distortion performance of the encoding of the 3D block, encoding the 3D block using one or more selected modes (instead of all possible modes) does not degrade the compression quality of the 3D point cloud geometry 112. Furthermore, because the rate-distortion cost of the 3D block may need to be determined for a fewer number of modes, the number of encoding and decoding operations associated with determining the rate-distortion cost may be reduced. Thus, the computational and memory requirements associated with compressing the 3D block may be reduced, thereby improving the efficiency associated with compressing the 3D point cloud geometry 112.

[0030]

[0041] FIG. 2 is a block diagram illustrating the exemplary electronic device of FIG. 1 in accordance with an embodiment of the present disclosure. The description of FIG. 2 will be provided with reference to the elements of FIG. 1. Referring to FIG. 2, a block diagram 200 of the electronic device 102 is shown. The electronic device 102 may include a circuit 202. The circuit may include a processor 204 and an encoder 206. In some embodiments, the circuit 202 may also include a decoder 208. The electronic device 102 may further include a memory 210, an input / output (I / O) device 212, and a network interface 214. The I / O device 212 may include a display device 212A, which may be utilized to render multimedia content, such as a 3D point cloud or a 3D graphics model rendered from the 3D point cloud. The circuit 202 may be communicatively coupled to the memory 210, the I / O device 212, and the network interface 214. The circuitry 202 can be configured to communicate with the server 104, the scanning setup 106, and a number of external devices 108A...108N using a network interface 214.

[0031]

[0042] The processor 204 may include suitable logic, circuitry, and / or interfaces that can be configured to execute instructions associated with encoding a 3D point cloud of an object. The processor 204 may also be configured to execute instructions associated with generating a 3D point cloud of an object in 3D space and / or receiving multiple color images and corresponding depth information. The processor 204 may further be configured to perform various operations related to transmitting and / or receiving the 3D point cloud (as multimedia content) to at least one of the multiple external devices 108A...108N. Examples of the processor 204 may include a graphics processing unit (GPU), a central processing unit (CPU), a tensor processing unit (TPU), a reduced instruction set computer (RISC) processor, an application specific integrated circuit (ASIC) processor, a complex instruction set computer (CISC) processor, a coprocessor, other processors, and / or combinations thereof. According to an embodiment, the processor 204 may be configured to assist the encoder 206 in encoding the 3D point cloud and the decoder 208 in decoding the encoded 3D point cloud and other functions of the electronic device 102.

[0032]

[0043] The encoder 206 may include suitable logic, circuitry, and / or interfaces that may be configured to encode 3D point cloud geometry corresponding to an object in 3D space. In one embodiment, the encoder 206 may encode the 3D point cloud by encoding each 3D block associated with the 3D point cloud geometry. In one embodiment, the encoder 206 may generate supplemental information associated with the encoded 3D point cloud geometry.

[0033]

[0044] In particular embodiments, the encoder 206 is configured to manage the storage of the encoded 3D point cloud geometry in memory 210 and / or the transfer of the encoded 3D point cloud geometry to another media device (e.g., a portable media player) via the communications network 110.

[0034]

[0045] In some embodiments, the encoder 206 may be implemented as a deep neural network (in the form of computer-executable code) on a GPU, CPU, TPU, RISC processor, ASIC processor, CISC processor, coprocessor, other processor, and / or combinations thereof. In some other embodiments, the encoder 206 may be implemented as a deep neural network on dedicated hardware interfaced with other computational circuitry of the electronic device 102. In such implementations, the encoder 206 may be associated with a particular form factor on a particular computational circuit. Examples of particular computational circuitry include, but are not limited to, field programmable gate arrays (FPGAs), programmable logic devices (PLDs), ASICs, programmable ASICs (PL-ASICs), application specific integrated components (ASSPs), and systems-on-chips (SOCs) based on standard microprocessors (MPUs) or digital signal processors (DSPs). According to some embodiments, the encoder 206 may also be interfaced with a GPU to parallelize the operation of the encoder 206. According to another embodiment, the encoder 206 may be implemented as a combination of programmable instructions stored in the memory 210 and logic units (or programmable logic units) on the hardware circuitry of the electronic device 102 .

[0035]

[0046] The decoder 208 may include suitable logic, circuitry, and / or interfaces that may be configured to decode encoded information that may represent geometric information of an object. The encoded information may also include supplemental information, such as coding tables, weight information, mode information, index values ​​for the geometric information, and quantization parameters, to assist the decoder 208. As an example, the encoded information may include encoded 3D point cloud geometry. The decoder 208 may be configured to reconstruct the 3D point cloud geometry by decoding the encoded 3D point cloud geometry. According to an embodiment, the decoder 208 may reside in at least one of multiple external devices 108A...108N.

[0036]

[0047] The memory 210 may include suitable logic, circuitry, and / or interfaces that may be configured to store instructions executable by the circuit 202. The memory 210 may be configured to store an operating system and associated applications. The memory 210 may further be configured to store a 3D point cloud (including the 3D point cloud geometry 112) corresponding to an object. According to an embodiment, the memory 210 may be configured to store information related to a plurality of modes and a table that maps the plurality of modes to classes and operating conditions. Example implementations of the memory 210 may include, but are not limited to, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), hard disk drive (HDD), solid-state drive (SSD), CPU cache, and / or a secure digital (SD) card.

[0037]

[0048] The I / O devices 212 may include suitable logic, circuitry, interfaces, and / or code that may be configured to receive user input. The I / O devices 212 may be further configured to provide output in response to the user input. The I / O devices 212 may include a variety of input and output devices, which may be configured to communicate with the circuit 202. Examples of input devices may include, but are not limited to, a touchscreen, a keyboard, a mouse, a joystick, and / or a microphone. Examples of output devices may include, but are not limited to, a display device 212A and / or a speaker.

[0038]

[0049] The display device 212A may include suitable logic, circuitry, interfaces, and / or code that may be configured to render a 3D point cloud on a display screen of the display device 212A. According to some embodiments, the display device 212A may include a touch screen and may receive user input. The display device 212A may be implemented through several known technologies, such as, but not limited to, liquid crystal display (LCD) display, light emitting diode (LED) display, plasma display, and / or organic LED (OLED) display technology, and / or other display technologies. According to some embodiments, the display device 212A may represent the display screen of a smart glasses device, a 3D display, a see-through display, a projection display, an electrochromic display, and / or a transparent display.

[0039]

[0050] The network interface 214 may include suitable logic, circuitry, interfaces, and / or code that may be configured to establish communications between the electronic device 102, the server 104, the scanning setup 106, and the plurality of external devices 108A...108N over the communications network 110. The network interface 214 may be implemented using various known technologies that support wired or wireless communications between the electronic device 102 and the communications network 110. The network interface 214 may include, but is not limited to, an antenna, a radio frequency (RF) transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a coder-decoder (CODEC) chipset, a subscriber identity module (SIM) card, and / or a local buffer.

[0040]

[0051] The network interface 214 may communicate via wireless communication with networks such as the Internet, an intranet, and / or a wireless network such as a cellular telephone network, a wireless local area network (LAN) and / or a metropolitan area network (MAN). The wireless communication may use any of a number of communication standards, protocols, and technologies, such as Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (W-CDMA), Long Term Evolution (LTE), Fifth Generation (5G) New Radio (NR), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Light Fidelity (Li-Fi), Wi-MAX, protocols for email, instant messaging, and / or Short Message Service (SMS). Operation of circuit 202 is described in more detail, for example, in Figures 3, 4, 5, 6, 7, 8A, 8B, 9A, 9B, and 10.

[0041]

[0052] Figure 3 is a block diagram of an example encoder and an example decoder for adaptive mode selection for point cloud compression, according to an embodiment of the present disclosure. The description of Figure 3 is provided with reference to elements of Figures 1 and 2. Referring to Figure 3, a block diagram 300 is shown that includes an encoder 302A and a decoder 302B. The encoder 302A may be an example implementation of the encoder 206 of Figure 2, and the decoder 302B may be an example implementation of the decoder 208 of Figure 2.

[0042]

[0053] In one embodiment, the encoder 302A and the decoder 302B may be implemented on separate electronic devices. In another embodiment, both the encoder 302A and the decoder 302B may be implemented on the electronic device 102. The decoder 302B may also be implemented on each of one or more external devices 108A...108N.

[0043]

[0054] The encoder 302A may include a set of encoders, such as a first encoder (e.g., encoder-1 304A), ..., and an Nth encoder (e.g., encoder-N 304N). Each encoder in the set of encoders of the encoder 302A may include an associated neural network model. For example, encoder-1 304A may include a first deep neural network (DNN) model, such as DNN model-1 306A. Further, encoder-N 304N may include an Nth DNN model, such as DNN model-N 306N. The encoder 302A may further include a mode selector 308, which may be communicatively coupled to each of the encoder-1 304A, ..., and encoder-N 304N.

[0044]

[0055] Each deep neural network model (e.g., DNN Model-1 306A) can be a neural network model including a computational network or system of artificial neurons arranged as nodes in multiple layers. The multiple layers of the neural network model can include an input layer, one or more hidden layers, and an output layer. Each of the multiple layers can include one or more nodes (or artificial neurons, represented by circles, for example). The output of every node in the input layer can be connected to at least one node in the hidden layer(s). Similarly, the input of each hidden layer can be connected to the output of at least one node in another layer of the neural network model. The output of each hidden layer can be connected to the input of at least one node in another layer of the neural network model. The node(s) in the final layer can receive inputs from at least one hidden layer and output results. The number of layers and the number of nodes in each layer can be determined from hyperparameters of the neural network model. Such hyperparameters can be set before or after training the neural network model with a training dataset.

[0045]

[0056] Each node of a neural network model can correspond to a mathematical function (e.g., a sigmoid function or a rectified linear unit) having a set of parameters that can be adjusted during training of the network. The parameter set can include, for example, weight parameters, regularization parameters, etc. Each node can use the mathematical function to calculate an output based on one or more inputs from nodes in other layer(s) (e.g., previous layer(s)) of the neural network model. All or some of the nodes of a neural network model can correspond to the same or different mathematical functions.

[0046]

[0057] In training a neural network model, one or more parameters of each node of the neural network can be updated based on whether the output of the final layer for a given input (from the training dataset) matches the correct result based on a loss function for the neural network model. The above process can be repeated for the same or different inputs until a minimum value of the loss function can be reached and the training error can be minimized. Several training methods are known in the art, such as gradient descent, stochastic gradient descent, batch gradient descent, gradient boosting, metaheuristics, etc.

[0047]

[0058] The neural network model may include electronic data that may be implemented, for example, as a software component of an application executable on an electronic device (e.g., electronic device 102). The neural network model may rely on libraries, external scripts, or other logic / instructions executed by a processing device such as circuit 202. The neural network model may include code and routines configured to enable a computing device such as circuit 202 to perform one or more operations for encoding or decoding 3D blocks associated with a 3D point cloud geometry. Additionally or alternatively, the neural network model may be implemented using hardware, including a processor, a microprocessor (e.g., performing or controlling the execution of one or more operations), a field programmable gate array (FPGA), or an application-specific integrated circuit (ASIC). Alternatively, in some embodiments, the neural network model may be implemented using a combination of hardware and software.

[0048]

[0059] The decoder 302B may include a set of decoders, such as a first decoder (e.g., decoder-1 310A), ..., and an Nth decoder (e.g., decoder-N 310N). Each decoder in the set of decoders may include an associated neural network model. For example, decoder-1 310A may include a first DNN model, such as DNN model-1 306A. Further, decoder-N 310N may include an Nth DNN model, such as DNN model-N 306N. FIG. 3 illustrates a block segmenter 312A associated with the encoder 302A and a binarizer and merger 312B associated with the decoder 302B. Also illustrated are an encoded bitstream and supplemental information 314A, a signaling bitstream 314B, an input point cloud 316A, a reconstructed point cloud 316N, and a set of 3D blocks 318.

[0049]

[0060] In operation, an input point cloud 316A including the 3D point cloud geometry 112 may be received for compression. The block divider 312A may receive the input point cloud 316A and divide the input point cloud 316A into a set of 3D blocks 318. The encoder 302A may receive the set of 3D blocks 318 from the block divider 312A. Each 3D block from the set of 3D blocks 318 may be input to the mode selector 308 and each of the encoders-1 304A, ..., and encoder-N 304N. The encoder-1 304A may include encoder circuitry and / or software (such as the encoder 206) and may encode the 3D block to determine a first encoded block based on application of the DNN model-1 306A to the 3D block. The encoding of the 3D block based on the DNN model-1 306A may correspond to encoding using a first mode from a plurality of modes. For example, the first mode may correspond to a first value (e.g., 0.5) of an alpha parameter of a focal loss function that can be used to train the DNN model-1 306A. Here, the focal loss function may be configured to penalize the removal of non-empty voxels from a 3D block of the 3D point cloud geometry 112. Similarly, the encoder-N 304N may include encoder circuitry and / or software (e.g., encoder 206) and encode the 3D block to determine an Nth encoded block based on application of the DNN model-N 306N to the 3D block. The encoding of the 3D block based on the DNN model-N 306N may correspond to encoding using an Nth mode from multiple modes.

[0050]

[0061] The encoder-1 304A may further include a decoder circuit and / or software (e.g., decoder 208) and may decode the first encoded block to determine a first decoded block based on application of the DNN model-1 306A to the first encoded block. The decoding of the first encoded block based on the DNN model-1 306A may correspond to decoding using a first mode. Similarly, the encoder-N 304N may include a circuit and / or software (e.g., decoder 208) and may decode the Nth encoded block to determine an Nth decoded block based on application of the DNN model-N 306N to the Nth encoded block. The decoding of the Nth encoded block based on the DNN model-N 306N may correspond to decoding using an Nth mode.

[0051]

[0062] The mode selector 308 may receive the 3D block and may further receive a first decoded block from encoder-1 304A, ..., and an Nth decoded block from encoder-N 304N. The mode selector 308 may compare the 3D block and the first decoded block to determine a first rate-distortion cost associated with encoding the 3D block using the first mode. Similarly, the mode selector 308 may compare the 3D block and the Nth decoded block to determine an Nth rate-distortion cost associated with encoding the 3D block using the Nth mode. The mode selector 308 may then compare the first rate-distortion cost, ..., and the Nth rate-distortion cost with each other. Based on the comparison between the rate-distortion costs, the mode selector 308 may determine a mode corresponding to the smallest rate-distortion cost. The mode selector 308 may select the determined mode as the mode selected for encoding the 3D block.

[0052]

[0063] The encoder 302A may be configured to encode the 3D blocks based on the mode selected by the mode selector 308. Encoding the 3D blocks included in the 3D point cloud geometry 112 may convert the 3D point cloud geometry 112 into a bitstream of compressed point cloud data (i.e., an encoded bitstream). The encoder 302A may transmit the bitstream of compressed point cloud data together with supplemental information associated with the 3D point cloud geometry 112 as an encoded bitstream and supplemental information 314A. Examples of the supplemental information may include, but are not limited to, mode decision information, coding tables, weight information, index values ​​for geometric information, and quantization parameters. The encoder 302A and the block divider 312A may together transmit a signaling bitstream 314B to the decoder 302B. The signaling bitstream 314B may include, but is not limited to, information associated with dividing the 3D point cloud geometry 112 into a set of 3D blocks 318 and mode decision information (e.g., the mode selected by the mode selector 308 for encoding the 3D blocks).

[0053]

[0064] The decoder 302B may be configured to receive the encoded bitstream and supplemental information 314A and the signaling bitstream 314B from the encoder 302A. In an embodiment, the decoder 302B may determine a mode for decoding the encoded bitstream based on mode decision information in the signaling bitstream 314B. In one example, the decoder 302B may determine that a first mode can be used to decode the encoded bitstream. The decoder-1 310A may include decoder circuitry and / or software (e.g., decoder 208) and may decode a first encoded block (from the encoded bitstream) to determine a first decoded block based on application of the DNN model-1 306A to the first encoded block. The decoding of the first encoded block based on the DNN model-1 306A may correspond to decoding the first encoded block using the first mode. Similarly, decoder-N 310N may include circuitry and / or software (such as decoder 208) to decode the Nth coded block (from the coded bitstream) to determine the Nth decoded block based on application of DNN model-N 306N to the Nth coded block. The decoding of the Nth coded block based on DNN model-N 306N may correspond to decoding the Nth coded block using the Nth mode.

[0054]

[0065] The decoder 302B may be configured to select a mode for decoding a particular coded block from the received coded bitstream (in the coded bitstream and supplemental information 314A) based on mode decision information associated with the coded block (in the signaling bitstream 314B). Based on the mode selected for the coded block, the decoder 302B may decode the coded block to determine a decoded block. For example, if a first mode is selected for decoding the coded block, the decoder 302B may decode the coded block based on the first mode. In such a case, the decoder 302B may select decoder-1 310A (which may be associated with DNN model-1 306A) to decode the coded block. The decoder 302B may be configured to determine a set of decoded blocks 318B based on decoding each coded block in the coded bitstream.

[0055]

[0066] The binarizer and merger 312B may receive the set of decoded blocks 318B from the decoder 302B. Additionally, the binarizer and merger 312B may receive the signaling bitstream 314B from the encoder 302A. Based on the received signaling bitstream 314B, the binarizer and merger 312B may binarize and merge the received set of decoded blocks 318B to obtain the reconstructed point cloud 316N. In one example, the binarizer and merger 312B may binarize and merge the set of decoded blocks 318B using information associated with the segmentation of the 3D point cloud geometry 112 in the signaling bitstream 314B.

[0056]

[0067] Figure 4 illustrates an exemplary processing pipeline for adaptive mode selection for point cloud compression, according to an embodiment of the present disclosure. Figure 4 is described with reference to elements of Figures 1, 2, and 3. Referring to Figure 4, a processing pipeline 400 for adaptive mode selection for point cloud compression is shown. The processing pipeline 400 illustrates a sequence of operations that may begin at 402 and end at 414. The sequence of operations may be performed by the circuitry 202 of the electronic device 102. The resulting outputs at each stage of the operations are also shown.

[0057]

[0068] At 402, a data acquisition operation may be performed. In the data acquisition operation, the processor 204 may be configured to acquire a 3D point cloud 402A, which may correspond to one or more objects (e.g., people) in 3D space. The 3D point cloud 402A may be a representation of geometric information and attribute information of one or more objects in 3D space. The geometric information may indicate 3D coordinates (e.g., XYZ coordinates) of individual feature points of the 3D point cloud 402A. In the absence of attribute information, the 3D point cloud 402A may be represented as a 3D point cloud geometry (e.g., 3D point cloud geometry 112) associated with one or more objects. The attribute information may include, for example, color information, reflectance information, opacity information, normal vector information, material identifier information, and texture information of the one or more objects. According to an embodiment, the 3D point cloud 402A may be received from the scanning setup 106 via the communication network 110 or acquired directly from an internal scanner, which may have the same functionality as the scanning setup 106.

[0058]

[0069] Each feature point in the 3D point cloud 402A is (x, y, z, Y, Cb, Cr, α, a1,...a n), where (x, y, z) can be 3D coordinates that can represent geometric information, (Y, Cb, Cr) can be the luma, chroma-blue difference, and chroma-red difference components (in YCbCr or YUV color space) of the feature point, α can be the transparency value of the feature point, and a1 to a n represents one-dimensional or multi-dimensional attributes such as material identifiers and normal vectors. Y, Cb, Cr, α and a1 to a n can jointly represent the attribute information of each feature point in the 3D point cloud 402A.

[0059]

[0070] Pre-encoding stage operations may be performed at 404. The pre-encoding stage operations may include operations such as operations 406 and 408, which may be performed by processor 204, as described herein.

[0060]

[0071] At 406, a voxelization operation may be performed on the 3D point cloud 402A. In the voxelization operation, the processor 204 may be configured to generate a plurality of voxels 406A from the 3D point cloud 402A. Each generated voxel may represent a volume element of one or more objects in 3D space. The volume element may indicate attribute information and geometric information corresponding to a group of feature points in the 3D point cloud 402A.

[0061]

[0072] An example of a voxelization process for a 3D point cloud 402A is presented herein. The 3D space corresponding to the 3D point cloud 402A can be thought of as a cube that can be recursively divided into multiple sub-cubes (e.g., octants). The size of each sub-cube can be based on the density of feature points in the 3D point cloud 402A. The feature points of the 3D point cloud 402A can occupy different sub-cubes. Each sub-cube can correspond to a voxel, and can include a set of feature points of the 3D point cloud 402A within a particular volume of the corresponding sub-cube. The processor 204 can be configured to calculate an average of attribute information associated with the set of feature points of the corresponding voxel. The processor 204 can also be configured to calculate a center coordinate for each voxel of the multiple voxels 406A based on geometric information associated with the set of feature points in the corresponding voxel. Each voxel among the generated plurality of voxels 406A can be represented by the center coordinates and the average of the attribute information associated with the corresponding set of feature points.

[0062]

[0073] According to an embodiment, the process of voxelizing the 3D point cloud 402A can be performed using conventional techniques that may be known to those skilled in the art. Therefore, details of the conventional techniques are omitted from this disclosure for the sake of brevity. The plurality of voxels 406A can represent geometric information and attribute information of one or more objects in 3D space. The plurality of voxels 406A can include occupied voxels and unoccupied voxels. Unoccupied voxels cannot represent geometric information and attribute information of one or more objects in 3D space. Only occupied voxels can represent geometric information and attribute information (e.g., color information) of one or more objects. According to an embodiment, the processor 204 can be configured to identify occupied voxels from the plurality of voxels 406A.

[0063]

[0074] At 408, a block division operation may be performed on the plurality of voxels 406A. In the block division operation, the processor 204 may be configured to divide the plurality of voxels 406A of the 3D point cloud geometry 112 into a set of 3D blocks (e.g., the set of 3D blocks 408A). For example, the processor 204 may divide the 3D point cloud geometry 112 into 3D blocks, each of which may have a predetermined size, such as 64x64x64. In one embodiment, the 3D point cloud geometry 112 may be divided into 3D blocks of the same size. In another embodiment, the 3D point cloud geometry 112 may be divided into 3D blocks of different sizes. For example, the plurality of voxels 406A may include a first set of voxels that may be densely populated and a second set of voxels that may be sparsely populated. A portion of the 3D point cloud geometry 112 that includes densely occupied voxels may be partitioned into a first set of 3D blocks of size 32x32x32, while another portion of the 3D point cloud geometry 112 that includes sparsely occupied voxels may be partitioned into a second set of 3D blocks of size 64x64x64. According to an embodiment, the processor 204 may select block sizes for partitioning different portions of the 3D point cloud geometry 112 based on a trade-off between the computational cost associated with the partitioning operation and the occupancy density of the partitioned 3D blocks.

[0064]

[0075] Encoding stage operations may be performed at 410. The encoding stage operations may include operations 412-414, which may be performed by the encoder 206, as described herein.

[0065]

[0076] At 412, a mode selection operation may be performed. In the mode selection operation, the processor 204 may be configured to determine mode decision information for a 3D block 416 of the set of 3D blocks 408A. In an alternative embodiment, the mode selection operation may be performed by the encoder 206. The mode decision information 412A may include, for example, class information associated with the 3D point cloud geometry 112, one or more operating conditions associated with an encoding stage (e.g., encoding stage 410) of the 3D point cloud geometry 112, or mode-related information associated with one or more 3D blocks of the set of 3D blocks 408A. Further, the processor 204 may be configured to select one or more modes (e.g., selected one or more modes 418) for the 3D block 416 from a plurality of modes based on the mode decision information.

[0066]

[0077] In one embodiment, one or more modes may be selected based on a lookup from a table that can map modes to classes and operating conditions. In another embodiment, one or more modes may be selected for a first number (e.g., “N”) of current 3D blocks based on a mode used to encode a second number (e.g., “M”) of previous 3D blocks. The second number of previous 3D blocks may precede the first number of current 3D blocks based on scanning the set of 3D blocks 408A in a defined scan order (e.g., Z-scan order). In another embodiment, one or more modes may be selected based on a mode used by 3D blocks adjacent to the current 3D block in the spatial arrangement of the set of 3D blocks 408A in the 3D point cloud geometry 112. In another embodiment, one or more modes may be selected based on application of a classifier to the current 3D block (and / or its neighboring 3D blocks) and point cloud metrics associated with the current 3D block (and / or its neighboring 3D blocks). In another embodiment, one or more modes may be selected based on applying a convolutional neural network directly to the current 3D block (and / or neighboring 3D blocks of the current 3D block).

[0067]

[0078] Depending on the mode decision information, one or more modes may be selected. If the one or more modes include more than one mode, the processor 204 may determine a rate-distortion cost associated with each of the selected one or more modes and compare the determined rate-distortion costs with each other. Based on the comparison of the determined rate-distortion costs, the processor 204 may select a mode with the smallest rate-distortion cost from the selected one or more modes as an optimal mode to encode the current 3D block. In another scenario, if the one or more modes include a single mode, the rate-distortion costs of the modes cannot be determined. Instead, the single mode may itself be the optimal mode and may be used to encode the current 3D block. Determining the mode decision information and selecting one or more modes is further described, for example, in Figures 5, 6, 7, 8A, 8B, 9A, and 9B.

[0068]

[0079] At 414, a mode-based encoding operation may be performed. In the mode-based encoding operation, the encoder 206 may be configured to encode the 3D block 416 based on one or more selected modes 418. For example, the encoder 206 may encode the 3D block 416 based on the one or more selected modes 418 to obtain an encoded 3D block 420. The encoding of 3D blocks is further described in, e.g., Figures 5, 6, 7, 8A, 8B, 9A, and 9B.

[0069]

[0080] Figure 5 illustrates an exemplary processing pipeline for adaptive mode selection for point cloud compression, according to an embodiment of the present disclosure. Figure 5 is described with reference to elements of Figures 1, 2, 3, and 4. Referring to Figure 5, a processing pipeline 500 for adaptive mode selection for point cloud compression is shown. The processing pipeline 500 illustrates a sequence of operations that may begin at 502 and end at 510. The sequence of operations may be performed by the circuitry 202 of the electronic device 102. The resulting outputs at each stage of the operations are also shown.

[0070]

[0081] At 502, a block splitting operation may be performed. In the block splitting operation, the processor 204 may be configured to split an input point cloud geometry (e.g., 3D point cloud geometry 512) into a set of 3D blocks (e.g., set of 3D blocks 408A). Set of 3D blocks 408A may include 3D block 502A. The block splitting operation is further described, for example, in FIG. 4 (at 408). Encoding of 3D block 502A according to embodiments of the present disclosure is described next herein.

[0071]

[0082] At 504, a class identification operation may be performed. In the class identification operation, the processor 204 may be configured to determine class information associated with the 3D point cloud geometry 512 based on a priori characteristics 514 associated with the 3D point cloud geometry 512. For example, the a priori characteristics 514 may include a geometry bit depth associated with the 3D point cloud geometry 512. Based on the a priori characteristics 514, the processor 204 may determine class information (e.g., a point cloud class 516). By way of example and not limitation, the class information may include at least one of a geometry bit depth, density, or point distribution associated with the 3D point cloud geometry 512. The processor 204 may also receive one or more operating conditions, such as operating conditions 518. In some embodiments, the one or more operating conditions may be received via user input from a user associated with the electronic device 102. Alternatively, the one or more operating conditions may be predetermined and stored in the memory 210. In such a case, the processor 204 may retrieve one or more operating conditions from the memory 210. The one or more operating conditions may be associated with the encoding stage of the 3D point cloud geometry 512 and may include a target rate-distortion cost associated with the 3D point cloud geometry 512. The rate-distortion cost may correspond to a distortion between the original point cloud block and the reconstructed point cloud block. The rate-distortion cost may be determined based on a point-to-point distance or a plane-to-plane distance (or any other objective or subjective distortion metric) between corresponding points in the original point cloud block and the reconstructed point cloud block and an estimated number of bits required to encode the corresponding block.

[0072]

[0083] The processor 204 may be configured to determine the mode decision information 412A based on the class information and one or more operating conditions. For example, the processor 204 may load a table (e.g., table 520) that may map a plurality of modes to classes and operating conditions. The processor 204 may search the table 520 using the class information and the one or more operating conditions to select one or more modes, where each mode of the plurality of modes may correspond to a function that may be used to encode a 3D block, such as the 3D block 502A. The table 520 may be generated and loaded prior to encoding the 3D block 502A. Generating the table 520 is further described in 506. Searching the table 520 to select one or more modes is further described in 508.

[0073]

[0084] At 506, a mode search operation can be performed. Prior to encoding the 3D blocks of the 3D point cloud geometry 512, the mode search operation can be used to generate the table 520. In the mode search operation, the processor 204 can use the encoder 206 and decoder 208 to determine an optimal mode from multiple modes to encode the 3D blocks of the calibration point cloud. The calibration point cloud can be a point cloud for which the class and operating conditions are known. The processor 204 can be configured to partition the calibration point cloud into multiple 3D blocks. The partitioning of the calibration point cloud can be similar to the partitioning of the 3D point cloud geometry 512, for example, as further described at 502.

[0074]

[0085] The encoder 206 may be further configured to encode each 3D block of the plurality of 3D blocks based on a plurality of modes to generate a plurality of encoded 3D blocks corresponding to each 3D block. Here, each mode may correspond to a function that can be used to encode the 3D block. A function that may be associated with a mode (e.g., a first mode) may be a deep neural network (DNN) model (e.g., DNN model-1 306A of encoder-1 304A in FIG. 3) that can be trained to encode the 3D block to generate the encoded 3D block. In an embodiment, each mode of the plurality of modes may correspond to an alpha parameter of a focal loss function used in the training phase of the DNN model. Here, the focal loss function may be configured to penalize the removal of non-empty voxels from the 3D block. By way of example and not limitation, the focal loss function may be expressed as follows: TIFF0007797633000001.tif24163In the above formula, FL() can represent the focal loss function, p t can represent the DNN model's estimated probability of a voxel being occupied, TIFF0007797633000002.tif7150 can represent an adjustable focusing parameter (where TIFF0007797633000003.tif7150), TIFF0007797633000004.tif7150 can represent the alpha parameter, which can be a balancing factor for the focal loss function (where: TIFF0007797633000005.tif7150).

[0075]

[0086] In one embodiment, a DNN model associated with each mode can be trained based on attributes and geometric characteristics (such as density) of 3D blocks of the input point cloud, e.g., 3D point cloud geometry 512. In such a case, each DNN model (which can be associated with a particular mode) can be trained using 3D blocks corresponding to different levels of density. Thus, different DNNs representing different modes can be derived from different training sets, each of which can correspond to a different density level. For example, the different density levels can include, but are not limited to, a solid level, a dense level, a sparse level, a scant level, and an insufficient level. Each density level can correspond to the median of a distribution of local densities associated with each 3D point in the 3D block. The local density can be, for example, a number of neighboring points within a spherical volume around each point.

[0076]

[0087] As an example, the multiple modes may include five modes, such as a first mode (e.g., mode “0”), a second mode (e.g., mode “1”), a third mode (e.g., mode “2”), a fourth mode (e.g., mode “3”), and a fifth mode (e.g., mode “4”). Each mode may correspond to a particular value of the alpha parameter. As an example, the first mode (i.e., mode “0”) may correspond to a value of “0.5” for the alpha parameter. Similarly, the second mode (i.e., mode “1”) may correspond to a value of “0.6” for the alpha parameter. Furthermore, each mode may correspond to a particular density level of the 3D point cloud geometry 512. As an example, the first mode (i.e., mode “0”) may correspond to a density level of “solid.” Similarly, the second mode (i.e., mode “1”) may correspond to a density level of “dense.” Table 1 provides exemplary values ​​of the alpha parameter for each of the multiple modes, and Table 2 provides exemplary values ​​of the density level for each of the multiple modes, as follows: Example values ​​for the alpha parameter for each mode [Table 1] Example values ​​of density levels for each mode [Table 2]

[0077]

[0088] Referring to Tables 1 and 2, the encoder 206 may encode multiple 3D blocks of the calibration point cloud based on each of the five modes. Each 3D block may be encoded to generate five encoded 3D blocks. The five encoded 3D blocks may include a first encoded 3D block (which may be encoded based on the first mode), a second encoded 3D block (which may be encoded based on the second mode), ..., a fifth encoded 3D block (which may be encoded based on the fifth mode). Each 3D block of the calibration point cloud may be encoded five times (i.e., once for each mode) to generate five versions of the encoded 3D block (i.e., one encoded block version for each mode). Note that the data shown in Tables 1 and 2 may be provided merely as experimental data and should not be construed as limiting the present disclosure.

[0078]

[0089] The encoder 206 can be configured to determine a rate-distortion cost associated with each of the generated multiple encoded 3D blocks. The rate-distortion cost can correspond to the distortion between the original point cloud block and the reconstructed point cloud block. The rate-distortion cost can be determined based on the point-to-point distance or plane-to-plane distance (or any other objective or subjective distortion metric) between corresponding points in the original point cloud and the reconstructed point cloud, and the estimated number of bits required to encode the corresponding block.

[0079]

[0090] The encoder 206 may determine, for each mode among the plurality of modes, statistical information that may indicate a fraction of the generated plurality of encoded 3D blocks that may have the lowest rate-distortion cost. For example, the plurality of modes may include mode “0,” mode “1,” mode “2,” mode “3,” and mode “4.” From among all available modes (i.e., modes “0,” “1,” “2,” “3,” and “4”), the encoder 206 may determine that the lowest rate-distortion cost for 90% of the encoded 3D blocks of the calibration point cloud is associated with mode “0.” Similarly, the encoder 206 may determine that the lowest rate-distortion cost for 5% of the encoded 3D blocks of the calibration point cloud may be associated with mode “1.” For each mode, the statistical information may include such a fraction of the generated plurality of encoded 3D blocks that may have the lowest rate-distortion cost. Example statistical information for the plurality of modes is shown in Table 3 as follows: Example statistics for each mode [Table 3] The data presented in Table 3 should be presented as experimental data only and should not be construed as limiting the present disclosure.

[0080]

[0091] Based on the determined statistical information, the encoder 206 may determine a subset of coded 3D blocks from the plurality of coded 3D blocks, where the percentage of the plurality of coded 3D blocks exceeds a threshold. For example, referring to Table 3, for 90% of the plurality of 3D blocks of the calibration point cloud, mode "0" may be the optimal mode. Furthermore, for 5% of the plurality of 3D blocks, mode "1" may be the optimal mode. In such a case, if the threshold is 94%, the subset of coded 3D blocks may be determined as mode "0" and mode "1", because the total percentage of coded 3D blocks for which modes "0" and "1" are the optimal modes is 95% (greater than the threshold).

[0081]

[0092] The encoder 206 may further determine a subset of modes from the plurality of modes that can be used to generate the subset of encoded 3D blocks. The subset of modes, when used to encode the 3D blocks, may result in a relatively small distortion cost compared to distortion costs corresponding to other modes of the plurality of modes. Furthermore, the subset of modes may encode a statistically significant or majority (e.g., 95%) of the plurality of 3D blocks of the calibration point cloud as an optimal mode. Thus, the subset of modes may encode the 3D blocks of the calibration point cloud as a more optimal mode than other modes of the plurality of modes. The subset of modes may be selected as modes corresponding to a class and operating condition associated with the calibration point cloud. The selected mode may be inserted into the tablet as a record corresponding to the class and operating condition associated with the calibration point cloud.

[0082]

[0093] The encoder 206 may generate a table (e.g., table 520) based on the determined subset of modes, classes (e.g., including classes of calibration point clouds), and operating conditions. For example, the encoder 206 may select different operating conditions (i.e., target rate-distortion costs) and determine a subset of modes for each operating condition for a particular class of calibration point cloud, as described above. The encoder 206 may repeat the process of determining a subset of modes for sets of calibration point clouds of different classes under various operating conditions. The encoder 206 may determine statistical information associated with mode usage for different point cloud classes and different operating conditions. The statistical information may indicate statistics associated with a subset of 3D blocks that can be encoded using the subset of modes. The encoder 206 may determine the subset of modes as an entry corresponding to the class and operating condition in table 520 based on the statistical information. In one example, if the statistical information indicates that the subset of 3D blocks includes 90% or more of the 3D blocks from the plurality of 3D blocks, the encoder 206 can determine a subset of modes (which can be associated with the subset of 3D blocks) as entries corresponding to a particular class and operating condition of the calibration point cloud. Because the subset of modes can be associated with 90% or more of the 3D blocks from the plurality of 3D blocks, the encoder 206 need only check the subset of modes to encode any 3D blocks of the 3D point cloud geometry that have the same class and under the same operating condition. Therefore, it may not be necessary to perform a full search (which may require encoding 3D blocks based on each of multiple modes) to determine a suitable mode for encoding the 3D blocks of the 3D point cloud geometry, thereby saving time and computation. The encoder 206 can search the table 520 using the class information and operating conditions of the 3D point cloud geometry 512 to select one or more modes.

[0083]

[0094] As shown in FIG. 5 , table 520 may include entries for subsets of modes corresponding to different point cloud classes and different operating conditions. For example, various point cloud classes may include class “b1” (e.g., 10-bit depth) and class “b2” (e.g., 11-bit depth). Furthermore, the operating conditions may correspond to a target rate-distortion cost and may include five operating conditions, such as “a1,” “a2,” “a3,” “a4,” and “a5.” As an example, a first subset of modes 520A corresponding to operating condition “a3” and point cloud class “b2” may include mode “2” and mode “3.” Note that the data shown in table 520 may be provided merely as experimental data and should not be construed as limiting the present disclosure.

[0084]

[0095] At 508, a mode selection operation may be performed. In the mode selection operation, the encoder 206 may be configured to select one or more modes for the 3D block 502A from a plurality of modes based on mode decision information. Here, the mode decision information may include class information (i.e., point cloud class 516) and operating conditions 518 associated with the 3D point cloud geometry. The encoder 206 may search a table 520 based on the point cloud class 516 and the operating conditions 518 to select one or more modes, as described herein. With reference to the table 520, the encoder 206 may select a first subset of modes 520A (i.e., mode “2” and mode “3”) as one or more modes to encode the 3D block 502A when the operating conditions 518 are “a3” and the point cloud class 516 is “b2.”

[0085]

[0096] At 510, a mode-based encoding operation may be performed. In the mode-based encoding operation, the encoder 206 may encode the 3D block 502A based on one or more selected modes to generate an encoded 3D block 522. If the selected one or more modes include a single mode (e.g., mode "4" for operating condition "a5" and point cloud class "b2"), the single mode itself may correspond to an optimal mode, which may be used to encode the 3D block 502A. In another scenario, if the selected one or more modes include multiple modes, the encoder 206 may encode the 3D block based on each of the one or more selected modes to determine one or more encoded 3D blocks. For example, if the one or more modes include three modes, the encoder 206 may encode each 3D block to generate three encoded 3D block versions (one for each mode). The encoder 206 may determine a rate-distortion cost associated with each of the selected one or more modes. The encoder 206 may determine a mode of the selected one or more modes as an optimal mode for the encoding step based on a determination that the rate-distortion cost associated with the mode corresponds to the smallest of the determined rate-distortion costs. In other words, the encoder 206 may compare the determined rate-distortion costs and select the mode with the smallest rate-distortion cost as the optimal mode, which may be used to encode the 3D block 502A.

[0086]

[0097] For example, if the selected one or more modes are the first subset of modes 520A, the encoder 206 may encode the 3D block 502A based on mode “2” and based on mode “3” to determine a first encoded 3D block and a second encoded 3D block, respectively. The encoder 206 may determine a first rate-distortion cost for the first encoded 3D block and a second rate-distortion cost for the second encoded 3D block. In an embodiment, the encoder 206 may compare the first rate-distortion cost and the second rate-distortion cost and determine the mode corresponding to the lower of the two rate-distortion costs as the optimal mode. For example, if the first rate-distortion cost associated with the first encoded 3D block (encoded using mode “2”) is less than the second rate-distortion cost associated with the second encoded 3D block (encoded using mode “3”), the encoder 206 may determine the optimal mode as mode “2.” In other words, the encoder 206 can determine an optimal mode from the one or more selected modes (e.g., modes “2” and “3”) as one that minimizes the rate-distortion cost associated with the 3D block when encoded using a particular mode (i.e., mode “2”).

[0087]

[0098] Once the optimal mode is determined, the encoder 206 can encode the 3D block 502A based on the determined optimal mode to generate an encoded 3D block 522. In one embodiment, the encoder 206 can encode the 3D block 502A using a DNN model corresponding to the determined optimal mode. For example, if mode “2” is determined as the optimal mode, the encoder 206 can encode the 3D block 502A using a DNN model (e.g., DNN model-N 306N) corresponding to mode “2.” Here, the DNN model-N 306N can be a DNN model that can be trained based on a focal loss function with an alpha parameter value of 0.7, because the alpha parameter value corresponding to mode “2” can be 0.7. In another example, the DNN model-N 306N can be a DNN model that can be trained with a training set including 3D blocks of a point cloud with a “sparse” density level. If the selected one or more modes include two modes (e.g., mode “2” and mode “3”) and the plurality of modes includes five different modes, the encoder 206 may need to perform two encoding operations and two decoding operations to encode each 3D block. In a conventional solution implementing a full mode search, the encoder may need to perform five encoding operations and five decoding operations to encode each 3D block. Using the table 520, the disclosed electronic device 102 can achieve a coding efficiency gain of 2.5 times (i.e., (5+5) / (2+2)) over that provided by conventional solutions.

[0088]

[0099] Figure 6 illustrates an exemplary processing pipeline for adaptive mode selection for point cloud compression, according to an embodiment of the present disclosure. Figure 6 is described with reference to elements in Figures 1, 2, 3, 4, and 5. Referring to Figure 6, a processing pipeline 600 for adaptive mode selection for point cloud compression is shown. The processing pipeline 600 illustrates a sequence of operations that may begin at 602 and end at 608. The sequence of operations may be performed by the circuitry 202 of the electronic device 102. The resulting outputs of each stage of the operations are also shown.

[0089]

[0100] At 602, a block splitting operation may be performed. In the block splitting operation, the encoder 206 may be configured to split an input point cloud geometry (e.g., 3D point cloud geometry 610) into a set of 3D blocks (e.g., set of 3D blocks 408A). Set of 3D blocks 408A may include 3D block 416. The block splitting operation is further described (at 408), for example, in FIG. 4.

[0090]

[0101] In one embodiment, the encoder 206 can be configured to determine a subset of the set of 3D blocks 408A based on scanning the set of 3D blocks 408A in a predetermined scan order. For example, the set of 3D blocks 408A can be scanned according to Morton order (or Z-order). The Morton order for the set of 3D blocks 408A can map multidimensional coordinate values ​​to 1D while preserving the locality of each of the voxels 406A in the set of 3D blocks 408A. The Morton order in 3D space can be generated by interleaving binary representations of the geometric information (i.e., 3D coordinates) of the voxels 406A. In one embodiment, the coordinates of the origins of each of the voxels 406A in the set of 3D blocks 408A can be sorted based on the Morton order for the encoding stage 410. Furthermore, during the decoding stage, the decoder can re-sort the voxels 406A in the set of 3D blocks 408A to obtain a reconstructed point cloud.

[0091]

[0102] Based on the determined scan order, the encoder 206 may determine a first subset of 3D blocks 602A and a second subset of 3D blocks 602B from the subsets of the set of 3D blocks 408A. That is, the set of 3D blocks 408A may include the first subset of 3D blocks 602A and the second subset of 3D blocks 602B. Here, the second subset of 3D blocks 602B may follow the first subset of 3D blocks 602A in the scan order. For example, the first subset of 3D blocks 602A may be a subset of “N” consecutive 3D blocks in the scan order. Furthermore, the second subset of 3D blocks 602B may be a subset of “M” consecutive 3D blocks in the scan order.

[0092]

[0103] The values ​​of "M" and "N" can vary during the encoding of the set of 3D blocks 408A based on local characteristics of the encoding operation. The local characteristics can include, for example, an increase in reconstruction distortion above a certain threshold. In one embodiment, the second subset of 3D blocks 602B can include the 3D block 416. The encoding of the 3D block 416 is described herein.

[0093]

[0104] At 604, a mode search operation may be performed. In the mode search operation, the encoder 206 may be configured to encode each 3D block of a first subset of the determined subsets (i.e., the first subset of 3D blocks 602A) based on each of a plurality of modes to generate a plurality of encoded 3D blocks. For example, with reference to Table 1, the encoder 206 may encode each 3D block of the first subset of 3D blocks 602A based on five modes (such as mode "0," mode "1," mode "2," mode "3," and mode "4") to generate five corresponding encoded 3D blocks. The encoder 206 may further be configured to determine a rate-distortion cost associated with each encoded 3D block of the plurality of encoded 3D blocks. The encoder 206 may be further configured to determine mode usage statistics (e.g., mode usage statistics 612) associated with the first subset (i.e., the first subset of 3D blocks 602A) based on the determined rate-distortion cost associated with each coded 3D block of the plurality of coded 3D blocks. The mode usage statistics 612 may include information associated with an optimal mode selected for the first subset of 3D blocks 602A. For example, the mode usage statistics 612 may indicate that mode "2" and mode "3" may be optimal modes for a particular percentage of the 3D blocks in the first subset of 3D blocks 602A. Here, the mode-related information may include the determined mode usage statistics 612 associated with the first subset of 3D blocks 602A.

[0094]

[0105] At 606, a mode selection operation may be performed. In the mode selection operation, the encoder 206 may be configured to select one or more modes for the second subset of 3D blocks 602B, which may include the 3D blocks 416. The selection of the one or more modes may be based on mode decision information that includes mode-related information in the form of mode usage statistics 612. For example, if the mode usage statistics indicate that mode "2" and mode "3" are optimal modes for more than 90% of the 3D blocks from the first subset of 3D blocks 602A, the encoder 206 may select mode "2" and mode "3" as the one or more modes for the second subset of 3D blocks 602B.

[0095]

[0106] At 608, a mode-based encoding operation may be performed. In the mode-based encoding operation, the encoder 206 may encode the 3D block 416 based on one or more selected modes to generate an encoded 3D block 614. In one embodiment, the encoder 206 may encode the 3D block 416 based on each of the one or more selected modes to determine one or more encoded 3D blocks. The encoder 206 may then be configured to determine a rate-distortion cost associated with each of the determined one or more encoded 3D blocks. The encoder 206 may determine a mode of the selected one or more modes as an optimal mode for the encoding stage based on a determination that the rate-distortion cost associated with the determined mode is below a threshold. Mode-based encoding operations are further described (at 510), for example, in FIG. 5.

[0096]

[0107] If the selected one or more modes include two modes (e.g., mode "2" and mode "3") and the plurality of modes includes five modes, then encoder 206 only needs to perform 2*M encoding operations and 2*M decoding operations to encode each 3D block. In this case, there is no need to generate table 520. In conventional systems, each of the M+N 3D blocks may be encoded and decoded in all five modes, so a total of 5*(M+N) encoding and decoding operations may be required. For the current embodiment of the present disclosure, 5*N encoding and decoding operations may be required to encode N blocks in the calibration process, and 2*M encoding and decoding operations may be required to encode M blocks following the N blocks (which may be used in the calibration process). The coding efficiency gain may be determined as 5*(M+N) / (2*M+5*N). For example, if M and N have the same value (i.e., M=N), the coding efficiency gain may be, for example, 5*2 / 7 (i.e., 10 / 7~1.43). In such a case, only two candidate modes can be selected, and M=N.

[0097]

[0108] Figure 7 illustrates an exemplary processing pipeline for adaptive mode selection for point cloud compression, according to an embodiment of the present disclosure. Figure 7 is described with reference to elements in Figures 1, 2, 3, 4, 5, and 6. Referring to Figure 7, a processing pipeline 700 for adaptive mode selection for point cloud compression is shown. The processing pipeline 700 illustrates a sequence of operations that may begin at 702 and end at 706. The sequence of operations may be performed by the circuitry 202 of the electronic device 102. The resulting outputs of each stage of the operations are also shown.

[0098]

[0109] At 702, a block splitting operation may be performed. In the block splitting operation, the processor 204 may be configured to split an input point cloud geometry (e.g., 3D point cloud geometry 708) into a set of 3D blocks (e.g., set of 3D blocks 408A). Set of 3D blocks 408A may include 3D block 710 (e.g., block "C" as shown in FIG. 7). The block splitting operation is further described (at 408), for example, in FIG. 4.

[0099]

[0110] The processor 204 may be further configured to determine a subset of 3D blocks from the set of 3D blocks 408A that reside in a neighborhood of the 3D block 710 based on the spatial arrangement of the set of 3D blocks 408A in the 3D point cloud geometry 708. For example, a subset of four 3D blocks may reside in a 3D neighborhood of the 3D block 710 (i.e., block "C"). As shown in FIG. 7, the subset of four 3D blocks may include a first 3D block 712A (e.g., block "A1"), a second 3D block 712B (e.g., block "A2"), a third 3D block 714A (e.g., block "B1"), and a fourth 3D block 714B (e.g., block "B2"). The encoding of the 3D block 710 is described herein.

[0100]

[0111] At 704, a mode selection operation may be performed. In the mode selection operation, the processor 204 may be configured to determine the use of one or more modes for encoding each 3D block of the subset of 3D blocks to generate a respective encoded 3D block. In some embodiments, the use of one or more modes for encoding each 3D block of the subset of 3D blocks may be predetermined based on the encoding of a subset of 3D blocks prior to the 3D block 710. As an example, the one or more modes used to encode both the first 3D block 712A and the second 3D block 712B may be mode "2" and mode "3." Furthermore, the one or more modes used to encode both the third 3D block 714A and the fourth 3D block 714B may be mode "3" and mode "4."

[0101]

[0112] The processor 204 can be configured to perform a neighborhood analysis operation on each 3D block of the subset. The neighborhood analysis operation can include determining a relative position of each 3D block of the neighborhood subset of the 3D block 710. As shown, for example, based on the neighborhood analysis operation, the processor 204 can determine that an edge of both the first 3D block 712A and the second 3D block 712B can be directly adjacent to an edge of the 3D block 710. Furthermore, the processor 204 can determine that a face of both the third 3D block 714A and the fourth 3D block 714B can be directly adjacent to a face of the 3D block 710. Based on the neighborhood analysis operation, it can be determined that both the third 3D block 714A and the fourth 3D block 714B are closer to the 3D block 710 than either the first 3D block 712A or the second 3D block 712B.

[0102]

[0113] By way of example and not limitation, based on the neighborhood analysis operation, processor 204 may select a frequently used mode from among neighboring 3D blocks of 3D block 710 (e.g., first 3D block 712A, second 3D block 712B, third 3D block 714A, and fourth 3D block 714B) to encode 3D block 710. If the most frequent mode does not exist, processor 204 may select a default mode associated with the closest 3D block (e.g., third 3D block 714A) to encode 3D block 710. In another example, processor 204 may select the mode of the closest 3D block (e.g., third 3D block 714A) in the neighborhood of 3D block 710 to encode 3D block 710. If there are two or more closest 3D blocks (such as a third 3D block 714A and a fourth 3D block 714B) in the neighborhood of the 3D block 710, the mode of one of these closest 3D blocks may be randomly selected to encode the 3D block 710. In another example, the processor 204 may perform a mode search operation (similar to 506 in FIG. 5 and 604 in FIG. 6) to select a mode from the modes of all the neighboring 3D blocks (or the nearest 3D blocks in the neighborhood) that corresponds to the smallest rate-distortion cost relative to the other modes. The selected mode (i.e., the mode with the smallest rate-distortion cost characteristics) may be used to encode the 3D block 710.

[0103]

[0114] The processor 204 may select an optimal mode for the 3D block 710 based on the use of the mode for encoding each 3D block of the subset of 3D blocks to generate a respective encoded 3D block. For example, because the third 3D block 714A and the fourth 3D block 714B may be closer to the 3D block 710, the encoder 206 may select the optimal mode to be used to encode the 3D block 710 as the mode used to encode each of the third 3D block 714A and the fourth 3D block 714B. Specifically, both the third 3D block 714A and the fourth 3D block 714B may be encoded based on mode "4."

[0104]

[0115] At 706, a mode-based encoding operation may be performed. In the mode-based encoding operation, the encoder 206 may encode the 3D block 710 based on one or more selected modes to generate an encoded 3D block 716. In some embodiments, the selected one or more modes may include only one mode. In such a case, rate-distortion-based optimization may not be necessary, and the encoder 206 may directly encode the 3D block 710 based on the single selected mode. As an example, if the selected one or more modes include a single mode (e.g., mode "4") and the multiple modes include five modes, the encoder 206 may only need to perform one encoding operation to encode each 3D block. Furthermore, generating the table 520 may not be necessary. However, in a conventional system, because the multiple modes include five modes, the encoder may need to perform five encoding operations and five decoding operations to encode each 3D block. Therefore, the disclosed electronic device 102 may achieve a coding efficiency gain of 10 times (i.e., (5 + 5) / 1) compared to a conventional system.

[0105]

[0116] Figure 8A illustrates an exemplary processing pipeline for adaptive mode selection for point cloud compression, according to an embodiment of the present disclosure. Figure 8A is described with reference to elements of Figures 1, 2, 3, 4, 5, 6, and 7. Referring to Figure 8A, a processing pipeline 800A for adaptive mode selection for point cloud compression is shown. The processing pipeline 800A illustrates a sequence of operations including 802A, 804A, and 806A. The sequence of operations may be performed by the circuitry 202 of the electronic device 102. The resulting outputs of each stage of the operations are also shown.

[0106]

[0117] At 802A, a block splitting operation may be performed. In the block splitting operation, the processor 204 may be configured to split an input point cloud geometry (e.g., 3D point cloud geometry 808) into a set of 3D blocks (e.g., set of 3D blocks 408A). The 3D point cloud geometry 808 may include information associated with the 3D point cloud geometry 112. The set of 3D blocks 408A may include a 3D block 810. The block splitting operation is further described (at 408) in, for example, FIG. 4.

[0107]

[0118] The processor 204 may further be configured to determine a point cloud metric (e.g., point cloud metric 812A) that includes class information associated with the 3D block 810. In some cases, the point cloud metric 812A may include, for example, a density or point distribution associated with the 3D block 810. The encoding of the 3D block 810 is described herein.

[0108]

[0119] At 804A, a mode selection operation may be performed. In the mode selection operation, the processor 204 may be configured to select one or more modes (e.g., predicted modes 814A) for the 3D block 810 based on application of a classifier model to the point cloud metrics 812A. According to one embodiment, the classifier model may be a machine learning model that may be trained on the task of mode prediction. A training dataset for the mode prediction task may include point cloud metrics associated with a 3D block of a point cloud and one or more modes that may be used to encode the corresponding 3D block. In one embodiment, the classifier model may be stored in the memory 210 and used by the processor 204 to select one or more modes for the 3D block 810. According to another embodiment, the classifier model may be a neural network model. The neural network model may be similar to the DNN model further described in FIG. 3, for example.

[0109]

[0120] At 806A, a mode-based encoding operation may be performed. In the mode-based encoding operation, the encoder 206 may encode the 3D block 810 based on one or more selected modes (e.g., predicted mode 814A) to generate an encoded 3D block 816A. In some embodiments, the selected one or more modes may include only one mode. In such a case, rate-distortion-based optimization may not be necessary, and the encoder 206 may directly encode the 3D block 810 based on the single selected mode. As an example, if the selected one or more modes include a single mode (e.g., mode "4") and the multiple modes include five modes, the encoder 206 may only need to perform one encoding operation to encode each 3D block. In this case, generating the table 520 may not be necessary. In a conventional solution that implements a full mode search for all modes (e.g., five modes), the encoder may perform five encoding operations and five decoding operations to encode each 3D block. Thus, the disclosed electronic device 102 can achieve a coding efficiency gain of more than 5 (i.e., (5*E+5*D) / E or 5+5*(D / E)) over that provided by conventional solutions when the encoding and decoding complexities are different. On the other hand, when the encoding and decoding complexities are equal, the disclosed electronic device 102 can achieve a coding efficiency gain of 10 over that provided by conventional solutions, where E and D represent the encoding complexity and the decoding complexity, respectively.

[0110]

[0121] FIG. 8B illustrates an exemplary processing pipeline for adaptive mode selection for point cloud compression, according to an embodiment of the present disclosure. The description of FIG. 8B is provided with reference to elements of FIGS. 1, 2, 3, 4, 5, 6, 7, and 8A. Referring to FIG. 8B, a processing pipeline 800B for adaptive mode selection for point cloud compression is shown. The processing pipeline 800B illustrates a sequence of operations including 802B, 804B, and 806B. The sequence of operations may be performed by the circuitry 202 of the electronic device 102. The resulting outputs of each stage of the operations are also illustrated.

[0111]

[0122] At 802B, a block splitting operation may be performed. In the block splitting operation, the processor 204 may be configured to split the input point cloud geometry (e.g., 3D point cloud geometry 808) into a set of 3D blocks (e.g., set of 3D blocks 408A). Set of 3D blocks 408A may include 3D block 810. The block splitting operation is further described (at 408), for example, in FIG. 4.

[0112]

[0123] The processor 204 may be further configured to determine a subset of neighboring 3D blocks of the 3D block 810 (e.g., the subset of 3D blocks 818). The encoder 206 may be further configured to determine a point cloud metric (e.g., the point cloud metric 812B) including class information associated with the 3D block 810 and the subset of neighboring 3D blocks 818 of the 3D block 810 of the 3D point cloud geometry 808. For example, the point cloud metric 812B may include, but is not limited to, a density or point distribution associated with the 3D block 810 and the subset of 3D blocks 818. The encoding of the 3D block 810 according to embodiments of the present disclosure is described herein below.

[0113]

[0124] At 804B, a mode selection operation may be performed. In the mode selection operation, the processor 204 may be configured to select one or more modes (e.g., predicted modes 814B) for the 3D block 810 based on application of a classifier model to the point cloud metrics 812B. In one embodiment, the classifier model may be a machine learning model that may be trained on the task of mode prediction. In one example, the classifier model may be a neural network model. A training dataset for the mode prediction task may include information associated with point cloud metrics associated with 3D blocks of the point cloud and one or more modes in which the corresponding 3D blocks may be encoded. In one embodiment, the classifier model may be stored in the memory 210 and used by the encoder 206 to select one or more modes for the 3D block 810. More details regarding the classifier model are described further, for example, in FIG. 8A .

[0114]

[0125] At 806B, a mode-based encoding operation may be performed. In the mode-based encoding operation, the encoder 206 may encode the 3D block 810 based on one or more selected modes (e.g., predicted mode 814B) to generate an encoded 3D block 816B. In some embodiments, the selected one or more modes may include only one mode. In such a case, rate-distortion-based optimization may not be necessary, and the encoder 206 may directly encode the 3D block 810 based on the single selected mode. As an example, if the selected one or more modes include a single mode (e.g., mode "4") and the multiple modes include five modes, the encoder 206 may only need to perform one encoding operation to encode each 3D block. Furthermore, generating the table 520 may not be necessary here. However, in a conventional system, because the multiple modes include five modes, the encoder may need to perform five encoding operations and five decoding operations to encode each 3D block. Thus, the disclosed electronic device 102 can achieve a coding efficiency gain of more than 5 (i.e., (5*E+5*D) / E or 5+5*(D / E)) over that provided by conventional solutions when the encoding and decoding complexities are different. On the other hand, when the encoding and decoding complexities are equal, the disclosed electronic device 102 can achieve a coding efficiency gain of 10 over that provided by conventional solutions, where E and D represent the encoding complexity and the decoding complexity, respectively.

[0115]

[0126] FIG. 9A illustrates an exemplary processing pipeline for adaptive mode selection for point cloud compression, according to an embodiment of the present disclosure. The description of FIG. 9A is provided with reference to elements of FIGS. 1, 2, 3, 4, 5, 6, 7, 8A, and 8B. Referring to FIG. 9A, a processing pipeline 900A for adaptive mode selection for point cloud compression is shown. The processing pipeline 900A illustrates a sequence of operations including 902A, 904A, and 906A. The sequence of operations may be performed by the circuitry 202 of the electronic device 102. The resulting outputs of each stage of the operations are also shown.

[0116]

[0127] At 902A, a block splitting operation may be performed. In the block splitting operation, the processor 204 may be configured to split an input point cloud geometry (e.g., 3D point cloud geometry 908) into a set of 3D blocks (e.g., set of 3D blocks 408A). Set of 3D blocks 408A may include a 3D block 910 (e.g., current 3D block "C"). The block splitting operation is further described, for example, in FIG. 4 (at 408). Encoding of the 3D blocks 910 according to embodiments of the present disclosure is described next herein.

[0117]

[0128] At 904A, a mode selection operation may be performed. In the mode selection operation, the processor 204 may be configured to apply a convolutional neural network (CNN) to the 3D block 910 to generate a mode prediction (e.g., predicted mode 912A) for the 3D block 910. The convolutional neural network may be trained on the task of mode prediction. A training dataset for the mode prediction task may include 3D blocks of sample point clouds and one or more modes that can be used to encode the corresponding 3D block. In one embodiment, the convolutional neural network (CNN) may be stored in the memory 210 and used to select one or more modes for the 3D block 910. The mode prediction may be included in the mode decision information (i.e., determined before the mode(s) are selected), and one or more modes may be selected based on the mode prediction. The CNN may be similar to the DNN model further described in FIG. 3, for example.

[0118]

[0129] At 906A, a mode-based encoding operation may be performed. In the mode-based encoding operation, the encoder 206 may encode the 3D block 910 based on one or more selected modes to generate an encoded 3D block 914A. In some embodiments, the selected one or more modes may include only one mode. In such a case, rate-distortion-based optimization may not be necessary, and the encoder 206 may directly encode the 3D block 910 based on the single selected mode. As an example, if the selected one or more modes include a single mode (e.g., mode "4") and the multiple modes include five modes, the encoder 206 may only need to perform one encoding operation to encode each 3D block. In this case, the table 520 may not be necessary. In a conventional solution implementing multiple modes (e.g., five modes), the encoder must perform at least five encoding operations and five decoding operations to encode each 3D block. Thus, using CNN, the disclosed electronic device 102 can achieve a coding efficiency gain of more than 5 (i.e., (5*E+5*D) / E or 5+5*(D / E)) over that provided by conventional solutions when the encoding and decoding complexities are different. On the other hand, when the encoding and decoding complexities are equal, the disclosed electronic device 102 can achieve a coding efficiency gain of 10 over that provided by conventional solutions, where E and D represent the encoding complexity and the decoding complexity, respectively.

[0119]

[0130] FIG. 9B illustrates an exemplary processing pipeline for adaptive mode selection for point cloud compression, according to an embodiment of the present disclosure. The description of FIG. 9B is provided with reference to elements of FIGS. 1, 2, 3, 4, 5, 6, 7, 8A, 8B, and 9A. Referring to FIG. 9B, a processing pipeline 900B for adaptive mode selection for point cloud compression is shown. The processing pipeline 900B illustrates a sequence of operations including 902B, 904B, and 906B. The sequence of operations may be performed by the circuitry 202 of the electronic device 102. The resulting outputs of each stage of the operations are also shown.

[0120]

[0131] At 902B, a block splitting operation may be performed. In the block splitting operation, the processor 204 may be configured to split the input point cloud geometry (e.g., 3D point cloud geometry 908) into a set of 3D blocks (e.g., set of 3D blocks 408A). The set of 3D blocks 408A may include 3D block 910. The block splitting operation is further described, for example, in FIG. 4 (at 408). The processor 204 may be further configured to determine a subset of neighboring 3D blocks of 3D block 910 (e.g., subset of 3D blocks 916). Encoding of 3D block 910 according to embodiments of the present disclosure is described next herein.

[0121]

[0132] At 904B, a mode selection operation may be performed. In the mode selection operation, the processor 204 may be configured to apply a convolutional neural network (CNN) to the 3D block 910 and a subset 916 of the 3D block 910's neighboring 3D blocks of the 3D point cloud geometry 908. Based on the application of the convolutional neural network, the encoder 206 may generate a mode prediction (e.g., predicted mode 912B) for the 3D block 910. The convolutional neural network may be trained on the task of mode prediction. A training data set for the mode prediction task may include 3D blocks of the sample point cloud and one or more modes that can be used to encode the corresponding 3D block. In one embodiment, the convolutional neural network (CNN) may be stored in the memory 210 and used to select one or more modes for the 3D block 910. The mode prediction may be included in the mode decision information (i.e., determined before the mode(s) are selected), and one or more modes may be selected based on the mode prediction. In the present case, one or more modes may be selected based on a mode prediction included in the mode decision information, which may be a single mode. The 3D block 910 may be coded based on the single mode indicated by the mode prediction in the mode decision information. More details regarding the convolutional neural network model are described further in, for example, FIG. 9A.

[0122]

[0133] At 906B, a mode-based encoding operation may be performed. In the mode-based encoding operation, the encoder 206 may encode the 3D block 910 based on one or more selected modes to generate an encoded 3D block 914B. In some embodiments, the selected one or more modes may include only one mode. In such a case, rate-distortion-based optimization may not be necessary, and the encoder 206 may directly encode the 3D block 910 based on the single selected mode. As an example, if the selected one or more modes include a single mode (e.g., mode "4") and the multiple modes include five modes, the encoder 206 may only need to perform one encoding operation to encode each 3D block. Furthermore, here, it may not be necessary to generate the table 520. However, in a conventional solution, because the multiple modes include five modes, the encoder may need to perform at least five encoding operations and five decoding operations to encode each 3D block. Thus, the disclosed electronic device 102 can achieve a coding efficiency gain of more than 5 (i.e., (5*E+5*D) / E or 5+5*(D / E)) over that provided by conventional solutions when the encoding and decoding complexities are different. On the other hand, when the encoding and decoding complexities are equal, the disclosed electronic device 102 can achieve a coding efficiency gain of 10 over that provided by conventional solutions, where E and D represent the encoding complexity and the decoding complexity, respectively.

[0123]

[0134] 10 is a flowchart illustrating an example operation for adaptive mode selection for point cloud compression, according to an embodiment of the present disclosure. Referring to FIG. 10, a flowchart 1000 is shown. The flowchart 1000 will be described with reference to elements in FIGS. 1, 2, 3, 4, 5, 6, 7, 8A, 8B, 9A, and 9B. Operations 1002-1012 may be implemented in the electronic device 102. The method described in flowchart 1000 may start at 1002 and proceed to 1004.

[0124]

[0135] At 1004, a 3D point cloud geometry (e.g., 3D point cloud geometry 112) may be received. In an embodiment, the circuit 202 may be configured to receive the 3D point cloud geometry 112. The 3D point cloud geometry 112 may be received from the scanning setup 106 via the communication network 110. Receiving the 3D point cloud geometry is further described, for example, in FIG. 4.

[0125]

[0136] At 1006, the 3D point cloud geometry 112 may be divided into a set of 3D blocks (e.g., set of 3D blocks 408A). In one embodiment, the circuitry 202 may be configured to divide the 3D point cloud geometry 112 into set of 3D blocks 408A. Division of the 3D point cloud geometry is further described, for example, in FIG. 4.

[0126]

[0137] At 1008, mode decision information (e.g., mode decision information 412A) may be determined for a 3D block of the set of 3D blocks, where the mode decision information includes at least one of, but is not limited to, class information associated with the 3D point cloud geometry 112, one or more operating conditions associated with an encoding stage of the 3D point cloud geometry 112, or mode-related information associated with one or more 3D blocks of the set of 3D blocks 408A. In an embodiment, the mode decision information 412A may include at least one of, but is not limited to, class information associated with the 3D point cloud geometry 112, one or more operating conditions associated with an encoding stage of the 3D point cloud geometry 112, or mode-related information associated with one or more 3D blocks of the set of 3D blocks 408A. In an embodiment, the circuit 202 may be configured to determine the mode decision information 412A for a 3D block 416 of the set of 3D blocks 408A. Determining the mode decision information is further described, for example, in Figures 4, 5, 6, 7, 8A, 8B, 9A, and 9B.

[0127]

[0138] At 1010, one or more modes for the 3D block may be selected from a plurality of modes based on the mode decision information, where each mode of the plurality of modes corresponds to a function for encoding the 3D block. Here, each mode of the plurality of modes may correspond to a function by which the 3D block 416 may be encoded. In an embodiment, the circuit 202 may be configured to select one or more modes for the 3D block 416 from a plurality of modes based on the mode decision information 412A. The selection of one or more modes is further described in, e.g., Figures 4, 5, 6, 7, 8A, 8B, 9A, and 9B.

[0128]

[0139] At 1012, the 3D block 416 may be encoded based on the selected one or more modes. In one embodiment, the circuit 202 may be configured to encode the 3D block 416 based on the selected one or more modes. Encoding of the 3D block based on the selected one or more modes is further described in, for example, Figures 4, 5, 6, 7, 8A, 8B, 9A, and 9B. Control may proceed to an end.

[0129]

[0140] Although flowchart 1000 is shown as individual operations such as 1004, 1006, 1008, 1010, and 1012, the disclosure is not limited in this respect. Thus, in particular embodiments, such individual operations may be further divided into additional operations, combined into fewer operations, or eliminated, depending on the particular implementation, without departing from the essence of the disclosed embodiments.

[0130]

[0141] Various embodiments of the present disclosure may provide a non-transitory computer-readable medium and / or storage medium having stored thereon computer-executable instructions that a machine and / or computer can execute to operate an electronic device (e.g., electronic device 102). Such instructions may cause electronic device 102 to perform operations including receiving a three-dimensional (3D) point cloud geometry (e.g., 3D point cloud geometry 112). The operations may further include dividing 3D point cloud geometry 112 into a set of 3D blocks (e.g., set of 3D blocks 408A). The operations may further include determining mode decision information (e.g., mode decision information 412A) for 3D block 416 of set of 3D blocks 408A. The mode decision information 412A may include at least one of, but is not limited to, class information associated with the 3D point cloud geometry 112, one or more operating conditions associated with an encoding stage of the 3D point cloud geometry 112, or mode-related information associated with one or more 3D blocks of the set of 3D blocks 408A. The operations may further include selecting one or more modes for the 3D block 416 from a plurality of modes based on the mode decision information 412A, where each mode among the plurality of modes may correspond to a function capable of encoding the 3D block 416. The operations may further include encoding the 3D block 416 based on the selected one or more modes.

[0131]

[0142] An example aspect of the present disclosure may provide a first electronic device (such as the first electronic device 102 of FIG. 1 ) including a circuit (such as the circuit 202). The circuit 202 may be configured to receive a three-dimensional (3D) point cloud geometry (e.g., the 3D point cloud geometry 112). The circuit 202 may be further configured to divide the 3D point cloud geometry 112 into a set of 3D blocks (e.g., the set 408A of 3D blocks). The circuit 202 may be further configured to determine mode decision information (e.g., the mode decision information 412A) for a 3D block 416 of the set 408A of 3D blocks. The mode decision information 412A may include, but is not limited to, at least one of class information associated with the 3D point cloud geometry 112, one or more operating conditions associated with an encoding stage of the 3D point cloud geometry 112, or mode-related information associated with one or more 3D blocks of the set 408A of 3D blocks. The circuitry 202 may be further configured to select one or more modes for the 3D block 416 from a plurality of modes based on the mode decision information 412A, where each mode of the plurality of modes may correspond to a function capable of encoding the 3D block 416. The circuitry 202 may be further configured to encode the 3D block 416 based on the selected one or more modes.

[0132]

[0143] In an embodiment, the class information associated with the 3D point cloud geometry 112 may include, but is not limited to, at least one of a geometry bit depth, density, or point distribution associated with the 3D point cloud geometry 112. Additionally, the one or more operating conditions associated with the encoding stage of the 3D point cloud geometry 112 may include, but is not limited to, a target rate-distortion cost associated with the 3D point cloud geometry 112.

[0133]

[0144] In one embodiment, circuitry 202 can be configured to load a table that can map multiple modes to classes and operating conditions, and circuitry 202 can be configured to search the table using class information and one or more operating conditions to select one or more modes.

[0134]

[0145] In one embodiment, the circuit 202 may be configured to divide the calibration point cloud into a plurality of 3D blocks. The circuit 202 may be further configured to encode the plurality of 3D blocks based on each of a plurality of modes to generate a plurality of encoded 3D blocks. The circuit 202 may be further configured to determine a rate-distortion cost associated with each of the generated plurality of encoded 3D blocks. The circuit 202 may further be configured to determine, for each of the plurality of modes, statistical information that may indicate a proportion of the plurality of encoded 3D blocks that has the lowest rate-distortion cost for the plurality of modes. The circuit 202 may be configured to determine, from the generated plurality of encoded 3D blocks, a subset of encoded 3D blocks in which a proportion of the plurality of encoded 3D blocks exceeds a threshold based on the determined statistical information. The circuit 202 may further be configured to determine, from the plurality of modes, a subset of modes used to generate the subset of encoded 3D blocks. The circuit 202 may be configured to generate a table based on the determined subset of modes, classes, and operating conditions.

[0135]

[0146] In one embodiment, the circuit 202 may be configured to encode the 3D block 416 based on each of the one or more selected modes to determine one or more encoded 3D blocks. Furthermore, the circuit 202 may be configured to determine a rate-distortion cost associated with the determined one or more encoded 3D blocks. The circuit 202 may be configured to determine a mode from the one or more selected modes as an optimal mode for the encoding stage based on a determination that the rate-distortion cost associated with the mode corresponds to the minimum of the determined rate-distortion costs. In other words, the circuit 202 may determine a mode from the one or more selected modes as an optimal mode that minimizes the rate-distortion cost for encoding the 3D block. The circuit 202 may further be configured to encode the 3D block based on the determined mode to generate an encoded 3D block.

[0136]

[0147] In one embodiment, the function corresponds to a deep neural network (DNN) model that can be trained to encode the 3D blocks 416 of the 3D point cloud geometry 112 to generate encoded 3D blocks. Each mode of the multiple modes can correspond to an alpha parameter of a focal loss function used in the training phase of the DNN model. Further, the focal loss function can be configured to penalize the removal of non-empty voxels from the 3D blocks 416 of the 3D point cloud geometry 112.

[0137]

[0148] In one embodiment, the circuit 202 may be configured to determine a subset of the set of 3D blocks 408A based on scanning the set of 3D blocks 408A in a predetermined scan order. The circuit 202 may be configured to encode each 3D block of a first subset of the determined subsets based on a plurality of modes to generate a plurality of encoded 3D blocks. The circuit 202 may further be configured to determine a rate-distortion cost associated with each encoded 3D block of the plurality of encoded 3D blocks. The circuit 202 may be configured to determine mode usage statistics associated with the first subset based on the determined rate-distortion cost associated with each encoded 3D block of the plurality of encoded 3D blocks. Here, the mode-related information may include the determined mode usage statistics associated with the first subset. The circuit 202 may further be configured to select one or more modes for a second subset including the 3D block 416. Here, the second subset may be included in the determined subset, and the second subset may follow the first subset according to the scan order.

[0138]

[0149] In an embodiment, the circuitry 202 may be configured to determine a subset of 3D blocks from the set of 3D blocks 408A that may be in a neighborhood of the 3D block 416 based on a spatial arrangement of the set of 3D blocks 408A in the 3D point cloud geometry 112. Here, the selection of the one or more modes may be based on using the one or more modes for encoding each 3D block of the subset of 3D blocks into a respective encoded 3D block.

[0139]

[0150] In one embodiment, the circuit 202 can be configured to determine a point cloud metric that includes class information associated with the 3D block 416, where the one or more modes can be selected further based on application of a classifier model to the point cloud metric, where the classifier model can be a machine learning model that can be trained on the task of mode prediction.

[0140]

[0151] In an embodiment, the circuitry 202 can be configured to determine a point cloud metric that includes class information associated with the 3D block and a subset of neighboring 3D blocks of the 3D block 416 of the 3D point cloud geometry 112. Here, the one or more modes can be selected further based on application of a classifier model to the point cloud metric, where the classifier model can be a machine learning model that can be trained on the task of mode prediction.

[0141]

[0152] In one embodiment, the circuitry 202 may be configured to apply a convolutional neural network to the 3D block 416 to generate a mode prediction for the 3D block 416. Here, the mode prediction may be included in the mode decision information 412A, and one or more modes may be selected based on the mode prediction.

[0142]

[0153] In one embodiment, the circuitry 202 may be configured to apply a convolutional neural network to the 3D block 416 and a subset of the 3D blocks neighboring the 3D block 416 in the 3D point cloud geometry 112 to generate a mode prediction for the 3D block 416. Here, the mode prediction may be included in the mode decision information 412A, and one or more modes may be selected based on the mode prediction.

[0143]

[0154] The present disclosure can be implemented in the form of hardware or in the form of a combination of hardware and software. The present disclosure can be implemented in a centralized manner in at least one computer system, or in a distributed manner where different elements can be distributed across several interconnected computer systems. Any computer system or other apparatus adapted to perform the methods described herein can be suitable. The combination of hardware and software can be a general-purpose computer system including a computer program that, when loaded and executed, can control the computer system to perform the methods described herein. The present disclosure can be implemented in the form of hardware including portions of integrated circuits that also perform other functions.

[0144]

[0155] The present disclosure may also be embodied in a computer program product, which includes all features that enable the implementation of the methods described herein and which is capable of executing these methods when loaded into a computer system. A computer program in this context means any expression, in any language, code or notation, of a set of instructions intended to cause a system having information processing capabilities to perform a particular function, either directly, or after a) conversion into another language, code or notation, or b) reproduction in a different content form, or both.

[0145]

[0156] While the present disclosure has been described with reference to several embodiments, those skilled in the art will recognize that various modifications can be made and equivalents substituted without departing from the scope of the disclosure. Additionally, many modifications may be made to adapt a particular situation or material to the teachings of the disclosure without departing from the scope of the disclosure. Therefore, it is not intended that the disclosure be limited to the particular embodiments disclosed, but rather, it is intended to include all embodiments falling within the scope of the appended claims. [Explanation of symbols]

[0146] 100 Network Environment 102 Electronic Devices 104 Server 106 Scanning Setup 108A...108N Multiple External Devices 110 Communication Network 112 Three-dimensional (3D) point cloud geometry 200 Block Diagram 202 circuits 204 processors 206 Encoder 208 Decoder 210 memory 212 Input / Output (I / O) Devices 212A Display Device 214 Network Interface 300 Block Diagram 302A Encoder 302B decoder 304A Encoder-1 304N Encoder-N 306A DNN Model-1 306N DNN Model-N 308 Mode Selector 310A Decoder-1 310N Decoder-N 312A Block division section 312B Binarization and Merging Section 314A Encoded bitstream and supplementary information 314B Signaling Bit Stream 316A Input point cloud 316N Reconstructed point cloud Set of 318 3D blocks 318B Set of decoded blocks 400 Processing Pipeline 402 Data Acquisition Phase 402A 3D Point Cloud 404 Pre-encoding stage 406 Voxelization 406A Multiple Voxels 408 Block Division 408A 3D Block Set 410 Encoding Stage 412 Mode Selection 412A Mode Determination Information 414 Mode-Based Encoding 416 3D Blocks 418 Selected mode or modes 420 coded 3D blocks 500 Processing Pipeline 502 Block Division 502A 3D Block 504 Class Identification 506 Mode Search 508 Mode Selection 510 Mode-Based Encoding 512 3D Point Cloud Geometry 514 A priori properties 516 Point Cloud Classes 518 Operating Conditions 520 Table First subset of 520A modes 522 coded 3D blocks 600 Processing Pipelines 602 Block Division 602A First subset of 3D blocks 602B Second subset of 3D blocks 604 Mode Search 606 Mode Selection 608 Mode-Based Encoding 610 3D Point Cloud Geometry 612 Mode Usage Statistics 614 coded 3D blocks 700 Processing Pipelines 702 Block Division 704 Mode Selection 706 mode-based encoding 708 3D Point Cloud Geometry 710 3D Blocks 712A First 3D Block 712B Second 3D Block 714A Third 3D Block 714B 4th 3D Block 716 coded 3D blocks 800A Processing Pipeline 800B Processing Pipeline 802A Block Division 802B Block Division 804A Mode Selection 804B Mode Selection 806A mode-based coding 806B mode-based encoding 808 3D Point Cloud Geometry 810 3D Blocks 812A Point Cloud Metrics 812B Point Cloud Metrics 814A Predicted Mode 814B Predicted Mode 816A Encoded 3D Block 816B coded 3D blocks Subset of 818 3D blocks 900A Processing Pipeline 900B Processing Pipeline 902A Block Division 902B Block division 904A Mode Selection 904B Mode Selection 906A Mode-Based Coding 906B Mode-Based Coding 908 3D Point Cloud Geometry 910 3D Blocks 912A Predicted Mode 912B Predicted Mode 914A Encoded 3D Block 914B Encoded 3D Block Subset of 916 3D blocks 1000 Flowchart 1002 start 1004 Receive three-dimensional (3D) point cloud geometry 1006 Split a 3D point cloud geometry into a set of 3D blocks 1008. Determine mode decision information for a 3D block of the set of 3D blocks, the mode decision information including at least one of class information associated with the 3D point cloud geometry, one or more operating conditions associated with an encoding stage of the 3D point cloud geometry, or mode related information associated with one or more 3D blocks of the set of 3D blocks. 1010 selecting one or more modes for the 3D block from a plurality of modes based on the mode decision information, each mode of the plurality of modes corresponding to a function for encoding the 3D block; 1012 Encode the 3D block based on one or more selected modes

Claims

1. 1. An electronic device comprising: receiving a three-dimensional (3D) point cloud geometry; Dividing the 3D point cloud geometry into a set of 3D blocks; determining mode decision information for a 3D block of the set of 3D blocks, the mode decision information comprising: class information associated with the 3D point cloud geometry; one or more operational conditions associated with the encoding of the 3D point cloud geometry; or mode-related information associated with one or more 3D blocks of said set of 3D blocks; and selecting one or more modes for the 3D block from a plurality of modes based on the mode determination information; each mode of the plurality of modes corresponds to a function used to encode the 3D block; and encoding the 3D block based on the selected one or more modes; a circuit configured to: Including, The circuit comprises: loading a table mapping the plurality of modes to classes and operating conditions; searching the table using the class information and the one or more operating conditions to select the one or more modes; The electronic device further comprises:

2. 2. The electronic device of claim 1, wherein the class information associated with the 3D point cloud geometry includes at least one of a geometry bit depth, density, or point distribution associated with the 3D point cloud geometry.

3. 2. The electronic device of claim 1, wherein the one or more operating conditions associated with the encoding of the 3D point cloud geometry include a target rate-distortion cost associated with the 3D point cloud geometry.

4. The circuit comprises: Dividing the calibration point cloud into multiple 3D blocks; encoding the plurality of 3D blocks based on each of the plurality of modes to generate a plurality of encoded 3D blocks; determining a rate-distortion cost associated with each of the generated plurality of coded 3D blocks; determining, for each mode among the plurality of modes, statistical information indicative of a fraction of the plurality of coded 3D blocks for which the rate-distortion cost for the plurality of modes is minimum; determining, from the generated plurality of coded 3D blocks, a subset of coded 3D blocks in which the proportion of the plurality of coded 3D blocks exceeds a threshold based on the determined statistical information; determining a subset of modes from the plurality of modes to be used to generate the subset of encoded 3D blocks; generating the table based on the determined subset of modes, the classes, and the operating conditions; further configured as follows:

4. The electronic device according to claim 3, wherein the electronic device comprises:

5. The circuit comprises: encoding the 3D block based on each of the selected one or more modes to determine one or more encoded 3D blocks; determining a rate-distortion cost associated with the determined one or more coded 3D blocks; determining a mode of the selected one or more modes as an optimal mode for the encoding step based on a determination that a rate-distortion cost associated with the mode corresponds to a minimum of the determined rate-distortion costs; encoding the 3D block based on the determined mode to generate an encoded 3D block. further configured as follows:

2. The electronic device according to claim 1 .

6. 2. The electronic device of claim 1, wherein the function corresponds to a deep neural network (DNN) model that is trained to encode the 3D blocks of the 3D point cloud geometry to generate encoded 3D blocks.

7. Each mode of the plurality of modes corresponds to an alpha parameter of a focal loss function used in a training phase of the DNN model; the focal loss function is configured to penalize the removal of non-empty voxels from the 3D block of the 3D point cloud geometry.

7. The electronic device according to claim 6, wherein the electronic device comprises:

8. An electronic device, receiving a three-dimensional (3D) point cloud geometry; Dividing the 3D point cloud geometry into a set of 3D blocks; determining mode decision information for a 3D block of the set of 3D blocks, the mode decision information comprising: class information associated with the 3D point cloud geometry; one or more operational conditions associated with the encoding of the 3D point cloud geometry; or mode-related information associated with one or more 3D blocks of said set of 3D blocks; and selecting one or more modes for the 3D block from a plurality of modes based on the mode determination information; each mode of the plurality of modes corresponds to a function used to encode the 3D block; and encoding the 3D block based on the selected one or more modes; a circuit configured to: Including, The circuit comprises: determining a subset of the set of 3D blocks based on scanning the set of 3D blocks in a prescribed scan order; encoding each 3D block of a first subset of the determined subsets based on the plurality of modes to generate a plurality of encoded 3D blocks; determining a rate-distortion cost associated with each coded 3D block of the plurality of coded 3D blocks; determining mode usage statistics associated with the first subset based on the determined rate-distortion cost associated with each coded 3D block of the plurality of coded 3D blocks. further configured as follows: the mode-related information includes the determined mode usage statistics associated with the first subset. An electronic device characterized by:

9. the circuitry is further configured to select the one or more modes for a second subset that includes the 3D block; the second subset is included in the determined subset; the second subset follows the first subset in the scanning order; 9. The electronic device according to claim 8.

10. An electronic device, receiving a three-dimensional (3D) point cloud geometry; Dividing the 3D point cloud geometry into a set of 3D blocks; determining mode decision information for a 3D block of the set of 3D blocks, the mode decision information comprising: class information associated with the 3D point cloud geometry; one or more operational conditions associated with the encoding of the 3D point cloud geometry; or mode-related information associated with one or more 3D blocks of said set of 3D blocks; and selecting one or more modes for the 3D block from a plurality of modes based on the mode determination information; each mode of the plurality of modes corresponds to a function used to encode the 3D block; and encoding the 3D block based on the selected one or more modes; a circuit configured to: Including, the circuitry is further configured to determine, from the set of 3D blocks, a subset of 3D blocks that are in a neighborhood of the 3D block based on a spatial arrangement of the set of 3D blocks in the 3D point cloud geometry; selecting the one or more modes is based on using the one or more modes for encoding each 3D block of the subset of 3D blocks into a respective encoded 3D block. An electronic device characterized by:

11. An electronic device, receiving a three-dimensional (3D) point cloud geometry; Dividing the 3D point cloud geometry into a set of 3D blocks; determining mode decision information for a 3D block of the set of 3D blocks, the mode decision information comprising: class information associated with the 3D point cloud geometry; one or more operational conditions associated with the encoding of the 3D point cloud geometry; or mode-related information associated with one or more 3D blocks of said set of 3D blocks; and selecting one or more modes for the 3D block from a plurality of modes based on the mode determination information; each mode of the plurality of modes corresponds to a function used to encode the 3D block; and encoding the 3D block based on the selected one or more modes; a circuit configured to: Including, the circuitry is further configured to determine a point cloud metric including the class information associated with the 3D block; the one or more modes are selected further based on application of a classifier model to the point cloud metrics; the classifier model is a machine learning model trained on the task of mode prediction; An electronic device characterized by:

12. An electronic device comprising: receiving a three-dimensional (3D) point cloud geometry; Dividing the 3D point cloud geometry into a set of 3D blocks; determining mode decision information for a 3D block of the set of 3D blocks, the mode decision information comprising: class information associated with the 3D point cloud geometry; one or more operational conditions associated with the encoding of the 3D point cloud geometry; or mode-related information associated with one or more 3D blocks of said set of 3D blocks; and selecting one or more modes for the 3D block from a plurality of modes based on the mode determination information; each mode of the plurality of modes corresponds to a function used to encode the 3D block; and encoding the 3D block based on the selected one or more modes; a circuit configured to: Including, the circuitry is further configured to determine a point cloud metric including the class information associated with the 3D block and a subset of the 3D block's neighboring 3D blocks of the 3D point cloud geometry; the one or more modes are selected further based on application of a classifier model to the point cloud metrics; the classifier model is a machine learning model trained on the task of mode prediction; An electronic device characterized by:

13. the circuitry is further configured to apply a convolutional neural network to the 3D block to generate a mode prediction for the 3D block; the mode prediction is included in the mode decision information, and the one or more modes are selected based on the mode prediction.

2. The electronic device according to claim 1 .

14. An electronic device comprising: receiving a three-dimensional (3D) point cloud geometry; Dividing the 3D point cloud geometry into a set of 3D blocks; determining mode decision information for a 3D block of the set of 3D blocks, the mode decision information comprising: class information associated with the 3D point cloud geometry; one or more operational conditions associated with the encoding of the 3D point cloud geometry; or mode-related information associated with one or more 3D blocks of said set of 3D blocks; and selecting one or more modes for the 3D block from a plurality of modes based on the mode determination information; each mode of the plurality of modes corresponds to a function used to encode the 3D block; and encoding the 3D block based on the selected one or more modes; a circuit configured to: Including, the circuitry is further configured to apply a convolutional neural network to the 3D block and a subset of the 3D block's neighboring 3D blocks of the 3D point cloud geometry to generate a mode prediction for the 3D block; the mode prediction is included in the mode decision information, and the one or more modes are selected based on the mode prediction. An electronic device characterized by:

15. 1. A method comprising: In electronic devices, receiving a three-dimensional (3D) point cloud geometry; Dividing the 3D point cloud geometry into a set of 3D blocks; determining mode decision information for a 3D block of the set of 3D blocks, the mode decision information comprising: class information associated with the 3D point cloud geometry; one or more operating conditions associated with encoding the 3D point cloud geometry; or mode-related information associated with one or more 3D blocks of said set of 3D blocks; and selecting one or more modes for the 3D block from a plurality of modes based on the mode determination information, each mode of the plurality of modes corresponding to a function used to encode the 3D block; encoding the 3D block based on the selected one or more modes; Including, loading a table mapping the plurality of modes to classes and operating conditions; searching the table using the class information and the one or more operating conditions to select the one or more modes; The method further comprising:

16. 16. The method of claim 15, wherein the function corresponds to a deep neural network (DNN) model that is trained to encode the 3D blocks of the 3D point cloud geometry to generate encoded 3D blocks.

17. each mode of the plurality of modes corresponds to an alpha parameter of a focal loss function used in a training phase of the DNN model; the focal loss function is configured to penalize the removal of non-empty voxels from the 3D block of the 3D point cloud geometry.

17. The method of claim 16.

18. A non-transitory computer-readable medium having stored thereon computer-executable instructions that, when executed by an electronic device, cause the electronic device to perform operations, the operations including: receiving a three-dimensional (3D) point cloud geometry; Dividing the 3D point cloud geometry into a set of 3D blocks; determining mode decision information for a 3D block of the set of 3D blocks, the mode decision information comprising: class information associated with the 3D point cloud geometry; one or more operating conditions associated with encoding the 3D point cloud geometry; or mode-related information associated with one or more 3D blocks of said set of 3D blocks; and selecting one or more modes for the 3D block from a plurality of modes based on the mode determination information; each mode of the plurality of modes corresponds to a function used to encode the 3D block; and encoding the 3D block based on the selected one or more modes; Including, The operation is loading a table mapping the plurality of modes to classes and operating conditions; searching the table using the class information and the one or more operating conditions to select the one or more modes; 10. A non-transitory computer-readable medium, further comprising:

Images