Octree features for depth feature based point cloud compression

Abstract

Some embodiments of an encoding method can include obtaining information comprising a point cloud; passing the point cloud through a feature extraction process to generate first features; obtaining octree features from an octree encoding process; generating second features by fusing the octree features with the first features; and aggregating features based on the second features. Some embodiments of a decoding method can include obtaining octree features from an octree encoding process; obtaining input features; determining new input features by fusing the octree features with the input features; and aggregating features based on the new input features.

Description

[0001] Cross-reference to related applications This application claims the benefit of U.S. Provisional Patent Application No. 63 / 543,466, entitled “OCTREE FEATURE FOR DEEP-FEATURE BASED POINT CLOUD COMPRESSION”, filed October 10, 2023, which is incorporated herein by reference in its entirety.

[0002] By incorporating references This application incorporates, in its entirety, the following applications: International Application No. PCT / US2022 / 045790 (“'790 Application”), entitled “METHOD AND APPARATUS FOR POINT CLOUD COMPRESSION USING HYBRID DEEP ENTROPY CODING”, filed October 5, 2022, which claims the benefit of U.S. Provisional Patent Application Serial No. 63 / 252,482 (“'482 Application”), entitled “METHOD AND APPARATUS FOR POINT CLOUD COMPRESSION USING HYBRID DEEP ENTROPY CODING”, filed October 5, 2021, under 35 USC § 119(e); and “LEARNING BASED OCTREE ENTROPY CODING COMPRESSION AND PROCESSING IN LIGHT DETECTION AND RANGING (LIDAR) AND OTHER”. The international application filed on July 12, 2023, with the international application number PCT / US2023 / 027509 (“'609 application”), claims the benefits under 35 U.S.SC § 119(e) to the following: “LEARNING BASED BITWISE OCTREE ENTROPY CODING COMPRESSION AND PROCESSING IN LIGHT DETECTION AND RANGING (LIDAR) AND OTHERSYSTEMS”, filed on July 12, 2022, with the U.S. Provisional Patent Application Serial No. 63 / 388,462 (“'462 application”); “Deep Distribution-Aware Point Feature Extractor for AI-Based PointCloud Compression”, filed on July 12, 2022, with the U.S. Provisional Patent Application Serial No. 63 / 388,600 (“'600 application”); and “Context-Aware Voxel-Based Upsampling for Point…”. "CloudProcessing" and U.S. Provisional Patent Application Serial No. 63 / 438,212 ("212 Application") filed on January 10, 2023. Background Technology

[0003] Point clouds are a data format used across several commercial sectors, from autonomous driving, robotics, AR / VR, civil engineering, and computer graphics to animation / film. 3D LiDAR sensors have already been deployed in self-driving cars, and affordable LiDAR sensors have been released for applications such as the Velodyne Velabit, Apple iPad Pro 2020, and Intel RealSense LiDAR camera L515. With advancements in sensing technology, 3D point cloud data is becoming more practical than ever before and is expected to be the ultimate enabler for these applications.

[0004] Point cloud data is also considered to consume a significant portion of network traffic, for example in immersive communication (VR / AR) and in cars connected via 5G networks. Efficient representation formats can be used for point clouds and communication. In particular, raw point cloud data is organized and processed for world modeling and sensing purposes. Compression of raw point clouds can be used for data storage and transmission within relevant scenes. Summary of the Invention

[0005] A first example method according to some embodiments may include: obtaining information including a point cloud; passing the point cloud through a feature extraction process to generate a first feature; obtaining an octree feature from an octree encoding process; generating a second feature by fusing the octree feature with the first feature; and aggregating the feature based on the second feature.

[0006] In some embodiments of the first example method, the octree feature is used on at least one of the encoder and decoder processes.

[0007] Some embodiments of the first example method may further include: a first bitstream of features aggregated by transport encapsulation; and a second bitstream of block map transport encapsulated from the octree coding process.

[0008] In some embodiments of the first example method, octree features are obtained from the octree encoding process based on block graphs corresponding to the first features.

[0009] For some embodiments of the first example method, obtaining octree features from the octree encoding process includes: using a block graph corresponding to the first feature as input; and passing the block graph through the octree encoding process to generate octree features.

[0010] For some embodiments of the first example method, aggregating features based on a second feature includes: passing the second feature through one or more convolutional neural network (CNN) blocks; and passing the output of one or more CNN blocks through a transformer process to generate aggregated features.

[0011] In some embodiments of the first example method, the converter process includes one or more multilayer perceptron (MLP) blocks.

[0012] For some embodiments of the first example method, aggregating features based on the second feature includes: passing the second feature through one or more convolutional neural network (CNN) blocks; and passing the output of one or more CNN blocks through an Inception-ResNet (IRN) process to generate aggregated features.

[0013] In some embodiments of the first example method, the IRN process includes one or more multilayer perceptron (MLP) blocks.

[0014] For some embodiments of the first example method, aggregating features based on a second feature includes: performing one or more iterative processes, wherein the iterative process includes: passing a fused feature through a multilayer perceptron (MLP) block; passing the output of the MLP through one or more feature aggregation layers to generate an output feature; and fusing the output feature with an octree feature to generate a fused feature for the next passing through the iterative process, wherein the second feature is used as the fused feature of the first passing through the iterative process, and wherein the output feature becomes the aggregated feature of the last passing through the iterative process.

[0015] A first example apparatus according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the apparatus to perform any of the methods listed above.

[0016] A second example method according to some embodiments may include: obtaining octree features from the octree encoding process; obtaining input features; determining new input features by fusing the octree features with the input features; and aggregating features based on the new input features.

[0017] In some embodiments of the second example method, the octree feature is used on at least one of the encoder and decoder processes.

[0018] In some embodiments of the second example method, the octree features are obtained from the octree encoding process based on the block graph corresponding to the first feature.

[0019] For some embodiments of the second example method, obtaining octree features from the octree encoding process includes: obtaining a bitstream comprising a block graph previously passed through the encoder's octree encoding process; using the bitstream as input; and passing the bitstream through the octree encoding process to generate octree features.

[0020] For some embodiments of the second example method, aggregating features based on the second feature includes: passing the second feature through one or more convolutional neural network (CNN) blocks; and passing the output of one or more CNN blocks through a transformer process to generate aggregated features.

[0021] In some embodiments of the second example method, the converter process includes one or more multilayer perceptron (MLP) blocks.

[0022] For some embodiments of the second example method, aggregating features based on the second feature includes: passing the second feature through one or more convolutional neural network (CNN) blocks; and passing the output of one or more CNN blocks through an Inception-ResNet (IRN) process to generate aggregated features.

[0023] In some embodiments of the second example method, the IRN process includes one or more multilayer perceptron (MLP) blocks.

[0024] For some embodiments of the second example method, aggregating features based on the second feature includes: performing one or more iterative processes, wherein the iterative process includes: passing a fused feature through a multilayer perceptron (MLP) block; passing the output of the MLP through one or more feature aggregation layers to generate an output feature; and fusing the output feature with an octree feature to generate a fused feature for the next passing through the iterative process, wherein the second feature is used as the fused feature of the first passing through the iterative process, and wherein the output feature becomes the aggregated feature of the last passing through the iterative process.

[0025] Some embodiments of the second example method may further include passing octree features and aggregated features through a point synthesis process to generate an output point cloud.

[0026] A second example apparatus according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the apparatus to perform any of the methods listed above.

[0027] A third example method according to some embodiments may include: obtaining a first bitstream including an encoded block map; using the bitstream as input to an octree decoding process; obtaining octree features and the decoded block map from the octree decoding process; obtaining input features from a second bitstream; determining new input features by fusing the decoded block map with the input features; aggregating features based on the new input features; and passing the octree features and aggregated features through a point synthesis process to generate an output point cloud.

[0028] For some embodiments of the third example method, determining new input features further includes: determining new input features by fusing octree features with the decoded block graph and input features.

[0029] In some embodiments of the third example method, the bit stream includes a first bit stream and a second bit stream.

[0030] A third example apparatus according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the apparatus to perform any of the methods listed above.

[0031] A fourth example method according to some embodiments may include: obtaining information including a point cloud; passing the point cloud through a feature extraction process to generate a first feature; obtaining an octree feature from an octree encoding process; generating a second feature by fusing the octree feature with the first feature; and passing the second feature through a feature aggregation process to generate a third feature.

[0032] A fourth example apparatus according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the apparatus to perform any of the methods listed above.

[0033] A fifth example method according to some embodiments may include: obtaining octree features from an octree encoding process; obtaining a first feature as input; generating a second feature by fusing the octree features with the first feature; and passing the second feature through a feature aggregation process to generate a third feature.

[0034] A fifth example apparatus according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the apparatus to perform any of the methods listed above.

[0035] A sixth example apparatus according to some embodiments may include at least one processor configured to perform any of the methods listed above.

[0036] A seventh example apparatus according to some embodiments may include a computer-readable medium storing instructions for causing one or more processors to perform any of the methods listed above.

[0037] An eighth example apparatus according to some embodiments may include at least one processor and at least one non-transitory computer-readable medium storing instructions for causing at least one processor to perform any of the methods listed above.

[0038] Example signals according to some embodiments may include bit streams generated according to any of the methods listed above. Attached Figure Description

[0039] Figure 1A This is a system diagram illustrating an example communication system according to some embodiments.

[0040] Figure 1B The illustration shows a method according to some embodiments. Figure 1A The diagram shows a system diagram of an example wireless transmit / receive unit (WTRU) used in a communication system.

[0041] Figure 1C This is a system diagram illustrating a set of example interfaces of a system according to some embodiments.

[0042] Figure 2A This is a functional block diagram of a block-based video encoder (such as an encoder for general video coding (VVC)) according to some embodiments.

[0043] Figure 2B This is a functional block diagram of a block-based video decoder (such as a decoder for VVC) according to some embodiments.

[0044] Figure 3A This is a schematic side view of an example waveguide display that can be used with extended reality (XR) applications according to some embodiments.

[0045] Figure 3B This is a schematic side view illustrating an example alternative display type that can be used with extended reality applications according to some embodiments.

[0046] Figure 3C This is a schematic side view illustrating an example alternative display type that can be used with extended reality applications according to some embodiments.

[0047] Figure 4 This is a flowchart illustrating an example of a learning-based point cloud codec according to some embodiments.

[0048] Figure 5 This is a flowchart illustrating an example of a learning-based point cloud codec according to some embodiments.

[0049] Figure 6 This is a flowchart illustrating an example of a learning-based point cloud codec according to some embodiments.

[0050] Figure 7 This is a process diagram illustrating an example process of increasing the processing dimension according to some embodiments and calculating fused features based on octree features.

[0051] Figure 8 It is a diagram illustrating the process of example feature extraction during encoding and feature aggregation during decoding according to some embodiments.

[0052] Figure 9 This is a diagram illustrating the process of example feature aggregation / point synthesis during decoding according to some embodiments.

[0053] Figure 10 This is a diagram illustrating an example of deep octree-based encoding according to some embodiments.

[0054] Figure 11 This is a diagram illustrating an example of a deep octree-based decoding process according to some embodiments.

[0055] Figure 12 This is a flowchart illustrating an example Inception-ResNet block for feature aggregation according to some embodiments.

[0056] Figure 13 This is a process diagram illustrating an example transformer block for feature aggregation according to some embodiments.

[0057] Figure 14 This is a flowchart illustrating an example self-attention block according to some embodiments.

[0058] Figure 15 This is a flowchart illustrating an example encoding process according to some embodiments.

[0059] Figure 16 This is a flowchart illustrating an example decoding process according to some embodiments.

[0060] Entities, connections, arrangements, and such depictions in various figures, as well as those described in conjunction with various figures, are presented by way of example rather than by way of limitation. Therefore, any and all statements or other indications regarding what a particular figure “depicts,” what a particular element or entity in a particular figure “is” or “has,” and any and all similar statements (which may be understood in isolation and out of context as absolute and therefore limiting) may only be correctly understood as being preceded by a clause such as “In at least one embodiment, …”. For the sake of brevity and clarity, this implicit introductory clause is not tiresomely repeated in the detailed description. Detailed Implementation

[0061] Figure 1AThis diagram illustrates an example communication system 100 in which one or more of the disclosed embodiments may be implemented. The communication system 100 may be a multiple access system providing content such as voice, data, video, messaging, and broadcasting to multiple wireless users. The communication system 100 enables multiple wireless users to access such content through the sharing of system resources, including wireless bandwidth. For example, the communication system 100 may employ one or more channel access methods, such as Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal FDMA (OFDMA), Single Carrier FDMA (SC-FDMA), Zero-Tail Unique Word DFT Extended OFDM (ZT UW DTS-s OFDM), Unique Word OFDM (UW-OFDM), Resource Block Filtered OFDM, Filter Bank Multicarrier (FBMC), and the like.

[0062] like Figure 1A As shown, the communication system 100 may include wireless transmit / receive units (WTRUs) 102a, 102b, 102c, 102d, RAN 104 / 113, CN 106, Public Switched Telephone Network (PSTN) 108, Internet 110, and other networks 112. However, it will be understood that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and / or network elements. Each of the WTRUs 102a, 102b, 102c, and 102d can be any type of device configured to operate and / or communicate in a wireless environment. As an example, WTRUs 102a, 102b, 102c, and 102d (any of which may be referred to as a “station” and / or “STA”) may be configured to transmit and / or receive wireless signals and may include user equipment (UE), mobile stations, fixed or mobile subscriber units, subscription-based units, pagers, cellular phones, personal digital assistants (PDAs), smartphones, laptops, netbooks, personal computers, wireless sensors, hotspots or Mi-Fi devices, Internet of Things (IoT) devices, watches or other wearable devices, head-mounted displays (HMDs), vehicles, drones, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices operating in the context of industrial and / or automated processing chains), consumer electronics devices, devices operating on commercial and / or industrial wireless networks, and the like. Any of WTRUs 102a, 102b, 102c, and 102d may be interchangeably referred to as a UE.

[0063] The communication system 100 may also include base station 114a and / or base station 114b. Each of base stations 114a and 114b can be any type of device configured to wirelessly interface with at least one of WTRUs 102a, 102b, 102c, and 102d to facilitate access to one or more communication networks, such as CN 106, Internet 110, and / or other networks 112. As an example, base stations 114a and 114b may be base transceiver stations (BTS), Node-B, eNode B, home node B, home eNode B, gNB, NR NodeB, site controllers, access points (APs), wireless routers, and the like. Although base stations 114a and 114b are each depicted as a single element, it will be understood that base stations 114a and 114b may include any number of interconnected base stations and / or network elements.

[0064] Base station 114a may be part of RAN 104 / 113, which may also include other base stations and / or network elements (not shown), such as base station controllers (BSCs), radio network controllers (RNCs), relay nodes, etc. Base station 114a and / or base station 114b may be configured to transmit and / or receive radio signals on one or more carrier frequencies (which may be referred to as cells (not shown)). These frequencies may be in licensed spectrum, unlicensed spectrum, or a combination of licensed and unlicensed spectrum. A cell may provide coverage of a specific geographic area, which may be relatively fixed or may change over time. A cell may be further divided into cell sectors. For example, the cell associated with base station 114a may be divided into three sectors. Thus, in one embodiment, base station 114a may include three transceivers, i.e., one transceiver per sector of the cell. In embodiments, base station 114a may employ multiple-input multiple-output (MIMO) technology and may utilize multiple transceivers for each sector of the cell. For example, beamforming may be used to transmit and / or receive signals in a desired spatial direction.

[0065] Base stations 114a and 114b can communicate with one or more of WTRUs 102a, 102b, 102c, and 102d via air interface 116, which can be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). Any suitable radio access technology (RAT) can be used to establish air interface 116.

[0066] More specifically, as noted above, communication system 100 can be a multiple access system and can employ one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, and the like. For example, base stations 114a and WTRUs 102a, 102b, and 102c in RAN104 / 113 can implement radio technologies such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which can use Wideband CDMA (WCDMA) to establish the air interface 116. WCDMA may include communication protocols such as High-Speed ​​Packet Access (HSPA) and / or evolved HSPA (HSPA+). HSPA may include High-Speed ​​Downlink (DL) Packet Access (HSDPA) and / or High-Speed ​​UL Packet Access (HSUPA).

[0067] In the embodiment, base station 114a and WTRUs 102a, 102b, 102c can implement radio technologies such as evolved UMTS terrestrial radio access (E-UTRA), which can use Long Term Evolution (LTE) and / or Advanced LTE (LTE-A) and / or Advanced LTE Pro (LTE-A Pro) to establish air interface 116.

[0068] In the embodiment, base station 114a and WTRUs 102a, 102b, 102c can implement radio technology (such as NR radio access) that can use New Radio (NR) to establish air interface 116.

[0069] In the embodiments, base station 114a and WTRUs 102a, 102b, and 102c can implement various radio access technologies. For example, base station 114a and WTRUs 102a, 102b, and 102c can, for example, use a dual connectivity (DC) principle to implement both LTE and NR radio access together. Therefore, the air interface utilized by WTRUs 102a, 102b, and 102c can be characterized by various types of radio access technologies and / or transmissions sent to / from various types of base stations (e.g., eNBs and gNBs).

[0070] In other embodiments, base station 114a and WTRUs 102a, 102b, 102c can implement radio technologies such as IEEE 802.11 (i.e., Wi-Fi), IEEE 802.16 (i.e., Global Microwave Access Interoperability (WiMAX)), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Provisional Standard 2000 (IS-2000), Provisional Standard 95 (IS-95), Provisional Standard 856 (IS-856), Global System for Mobile Communications (GSM), Enhanced Data Rate GSM Evolution (EDGE), GSM EDGE (GERAN), and the like.

[0071] Figure 1A Base station 114b can be, for example, a wireless router, home node B, home eNode B, or access point, and can utilize any suitable RAT to facilitate wireless connectivity in local areas such as commercial locations, homes, vehicles, campuses, industrial facilities, air corridors (e.g., for drone use), roads, and the like. In one embodiment, base station 114b and WTRUs 102c, 102d can implement radio technologies such as IEEE 802.11 to establish a wireless local area network (WLAN). In another embodiment, base station 114b and WTRUs 102c, 102d can implement radio technologies such as IEEE 802.15 to establish a wireless personal area network (WPAN). In yet another embodiment, base station 114b and WTRUs 102c, 102d can utilize cellular-based RATs (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR, etc.) to establish picocells or femtocells. Figure 1A As shown, base station 114b can have a direct connection to Internet 110. Therefore, base station 114b does not need to access Internet 110 via CN 106.

[0072] RAN 104 / 113 can communicate with CN 106, which can be any type of network configured to provide voice, data, application, and / or Voice over Internet Protocol (VoIP) services to one or more of WTRUs 102a, 102b, 102c, and 102d. Data can have different Quality of Service (QoS) requirements, such as different throughput requirements, latency requirements, error tolerance requirements, reliability requirements, data throughput requirements, mobility requirements, and the like. CN 106 can provide call control, billing services, location-based services, prepaid calling, internet connectivity, video distribution, and / or perform advanced security functions such as user authentication. Although not explicitly stated... Figure 1AAs shown, but to be understood, RAN 104 / 113 and / or CN 106 can communicate directly or indirectly with other RANs that use the same RAT as or a different RAT than RAN 104 / 113. For example, in addition to being connected to RAN 104 / 113, which can utilize NR radio technology, CN 106 can also communicate with another RAN (not shown) that uses GSM, UMTS, CDMA 2000, WiMAX, E-UTRA, or WiFi radio technology.

[0073] CN 106 can also serve as a gateway for WTRUs 102a, 102b, 102c, and 102d to access PSTN 108, the Internet 110, and / or other networks 112. PSTN 108 may include a circuit-switched telephone network providing Common Old-Style Telephone Service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices using common communication protocols such as Transmission Control Protocol (TCP), User Datagram Protocol (UDP), and / or Internet Protocol (IP) from the TCP / IP Internet Protocol suite. Network 112 may include wired and / or wireless communication networks owned and / or operated by other service providers. For example, network 112 may include another CN connected to one or more RANs, which may use the same RAT as RAN104 / 113 or a different RAT.

[0074] Some or all of the WTRUs 102a, 102b, 102c, and 102d in communication system 100 may include multi-mode capabilities (e.g., WTRUs 102a, 102b, 102c, and 102d may include multiple transceivers for communicating with different wireless networks via different wireless links). For example, Figure 1A The WTRU 102c shown can be configured to communicate with base station 114a, which can employ cellular-based radio technology, and with base station 114b, which can employ IEEE 802 radio technology.

[0075] Figure 1B This is a system diagram illustrating example WTRU 102. (Example:) Figure 1B As shown, among other things, WTRU 102 may also include a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a keypad 126, a display / touchpad 128, non-removable memory 130, removable memory 132, a power supply 134, a Global Positioning System (GPS) chipset 136, and / or other peripheral devices 138. It will be understood that, while remaining consistent with the embodiments, WTRU 102 may include any sub-combination of the foregoing elements.

[0076] Processor 118 may be a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, and the like. Processor 118 may perform signal encoding, data processing, power control, input / output processing, and / or any other functions that enable WTRU 102 to operate in a wireless environment. Processor 118 may be coupled to transceiver 120, and transceiver 120 may be coupled to transmit / receive element 122. Although Figure 1B The processor 118 and transceiver 120 are depicted as separate components, but it will be understood that the processor 118 and transceiver 120 may be integrated together in an electronic package or chip.

[0077] Transmitting / receiving element 122 can be configured to transmit signals to or receive signals from a base station (e.g., base station 114a) via air interface 116. For example, in one embodiment, transmitting / receiving element 122 can be an antenna configured to transmit and / or receive RF signals. In another embodiment, transmitting / receiving element 122 can be a transmitter / detector configured to transmit and / or receive, for example, IR, UV, or visible light signals. In yet another embodiment, transmitting / receiving element 122 can be configured to transmit and / or receive both RF signals and optical signals. It will be understood that transmitting / receiving element 122 can be configured to transmit and / or receive any combination of wireless signals.

[0078] Although the transmitting / receiving element 122 is in Figure 1B While depicted as a single element, WTRU 102 may include any number of transmit / receive elements 122. More specifically, WTRU 102 may employ MIMO technology. Thus, in one embodiment, WTRU 102 may include two or more transmit / receive elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals via air interface 116.

[0079] Transceiver 120 can be configured to modulate signals to be transmitted by transmitting / receiving element 122 and demodulate signals received by transmitting / receiving element 122. As noted above, WTRU 102 can have multi-mode capability. Therefore, transceiver 120 can include multiple transceivers to enable WTRU 102 to communicate via various RATs, such as NR and IEEE 802.11.

[0080] The processor 118 of WTRU 102 can be coupled to a speaker / microphone 124, a keypad 126, and / or a display / touchpad 128 (e.g., a liquid crystal display (LCD) unit or an organic light-emitting diode (OLED) display unit) and can receive user input data from the speaker / microphone 124, keypad 126, and / or display / touchpad 128. The processor 118 can also output user data to the speaker / microphone 124, keypad 126, and / or display / touchpad 128. Furthermore, the processor 118 can access information from any type of suitable memory (such as non-removable memory 130 and / or removable memory 132) and store data in any type of suitable memory. Non-removable memory 130 may include random access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. Removable memory 132 may include a subscriber identity module (SIM) card, memory stick, secure digital storage (SD) card, and the like. In other embodiments, processor 118 may access memory information that is never physically located on WTRU 102 (such as on a server or home computer (not shown)) and store the data in that memory.

[0081] The processor 118 may receive power from the power supply 134 and may be configured to distribute and / or control power to other components in the WTRU 102. The power supply 134 may be any suitable device for powering the WTRU 102. For example, the power supply 134 may include one or more dry cell battery packs (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, and the like.

[0082] The processor 118 may also be coupled to a GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) about the current location of the WTRU 102. In addition to or instead of the information from the GPS chipset 136, the WTRU 102 may receive location information from base stations (e.g., base stations 114a, 114b) via air interface 116 and / or determine its location based on the timing of signals received from two or more nearby base stations. It will be understood that, while remaining consistent with the embodiments, the WTRU 102 may acquire location information using any suitable location determination method.

[0083] The processor 118 may be further coupled to other peripheral devices 138, which may include one or more software and / or hardware modules providing additional features, functions, and / or wired or wireless connectivity. For example, peripheral devices 138 may include accelerometers, electronic compasses, satellite transceivers, digital cameras (for photos and / or video), Universal Serial Bus (USB) ports, vibration devices, television transceivers, hands-free headsets, Bluetooth® modules, FM radio units, digital music players, media players, video game player modules, internet browsers, virtual reality and / or augmented reality (VR / AR) devices, activity trackers, and the like. Peripheral devices 138 may include one or more sensors, which may be one or more of the following: gyroscopes, accelerometers, Hall effect sensors, magnetometers, orientation sensors, proximity sensors, temperature sensors, time sensors; geolocation sensors; altimeters, light sensors, touch sensors, magnetometers, barometers, gesture sensors, biometric sensors, and / or humidity sensors.

[0084] WTRU 102 may include a full-duplex radio for which the transmission and reception of some or all signals (e.g., associated with specific subframes for both UL (e.g., for transmission) and downlink (e.g., for reception)) may be concurrent and / or simultaneous. The full-duplex radio may include an interference management unit for reducing and / or substantially eliminating self-interference through signal processing via hardware (e.g., a choke) or via a processor (e.g., a separate processor (not shown) or via processor 118). In an embodiment, WTRU 102 may include a half-duplex radio for which the transmission and reception of some or all signals (e.g., associated with specific subframes for UL (e.g., for transmission) or downlink (e.g., for reception)) may be concurrent and / or simultaneous.

[0085] Despite WTRU in Figure 1A-1B While described as a wireless terminal, in some representative embodiments such a terminal may (e.g., temporarily or permanently) use a wired communication interface with a communication network.

[0086] In a representative embodiment, the other network 112 may be a WLAN.

[0087] Given Figure 1A-1BAs described herein, one or more, or all, of the functions described herein may be performed by one or more emulation devices (not shown). An emulation device may be one or more devices configured to emulate one or more, or all, of the functions described herein. For example, an emulation device may be used to test other devices and / or simulate network and / or WTRU functions.

[0088] Simulation devices can be designed to perform one or more tests on other devices in a laboratory environment and / or a carrier network environment. For example, one or more simulation devices can perform one or more or all functions while being fully or partially implemented and / or deployed as part of a wired and / or wireless communication network to test other devices within the communication network. One or more simulation devices can perform one or more or all functions while being temporarily implemented / deployed as part of a wired and / or wireless communication network. Simulation devices can be directly coupled to another device for testing purposes and / or can be used to perform tests via over-the-air wireless communication.

[0089] One or more emulation devices can perform one or more (including all) functions without being implemented / deployed as part of a wired and / or wireless communication network. For example, emulation devices can be used in test scenarios within test laboratories and / or non-deployed (e.g., testing) wired and / or wireless communication networks to perform testing of one or more components. One or more emulation devices can be test rigs. Direct RF coupling and / or wireless communication via RF circuitry (e.g., which may include one or more antennas) can be used by the emulation devices to transmit and / or receive data.

[0090] Figure 1CThis is a system diagram illustrating a set of example interfaces of a system according to some embodiments. In some embodiments, an extended reality display device and its control electronics can be implemented. System 150 can be implemented as a device including the various components described below and configured to perform one or more aspects of the aspects described in this document. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Elements of system 150 can be implemented individually or in combination in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 150 are distributed across multiple ICs and / or discrete components. In various embodiments, system 150 is communicatively coupled to one or more other systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports. In various embodiments, system 150 is configured to implement one or more aspects of the aspects described in this document.

[0091] System 150 includes at least one processor 152 configured to execute instructions loaded therein for implementing aspects such as those described in this document. Processor 152 may include embedded memory, input / output interfaces, and various other circuitry as known in the art. System 150 includes at least one memory 154 (e.g., a volatile memory device and / or a non-volatile memory device). System 150 may include a storage device 158, which may include non-volatile memory and / or volatile memory, including but not limited to electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, disk drives, and / or optical disk drives. As a non-limiting example, storage device 158 may include internal storage devices, attached storage devices (including removable and non-removable storage devices), and / or network-accessible storage devices.

[0092] System 150 includes an encoder / decoder module 156 configured to, for example, process data to provide encoded or decoded video, and the encoder / decoder module 156 may include its own processor and memory. Encoder / decoder module 156 represents one or more modules that can be included in a device to perform encoding and / or decoding functions. It is well known that a device may include one or both encoding and decoding modules. Additionally, encoder / decoder module 156 may be implemented as a separate element of system 150, or may be incorporated into processor 152 as a combination of hardware and software as known to those skilled in the art.

[0093] Program code to be loaded onto processor 152 or encoder / decoder 156 to execute the aspects described in this document may be stored in storage device 158 and subsequently loaded onto memory 154 for execution by processor 152. According to various embodiments, one or more of processor 152, memory 154, storage device 158, and encoder / decoder module 156 may store one or more items during the execution of the processes described in this document. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from processing equations, formulas, operations, and operational logic.

[0094] In some embodiments, the memory within processor 152 and / or encoder / decoder module 156 is used to store instructions and provide working memory for processing during encoding or decoding. However, in other embodiments, external memory (e.g., processor 152 or encoder / decoder module 152) is used for one or more of these functions. External memory may be memory 154 and / or storage device 158, such as volatile memory and / or non-volatile flash memory. In several embodiments, external non-volatile flash memory is used to store, for example, the operating system of a television. In at least one embodiment, a fast external dynamic volatile memory (such as RAM) is used as working memory for video encoding and decoding operations, such as for MPEG-2 (MPEG stands for Moving Picture Experts Group, MPEG-2 is also known as ISO / IEC 13818, and 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC stands for High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Various Video Coding, a new standard developed by the Joint Video Experts Group JVET).

[0095] Inputs to the components of system 150 can be provided through various input devices as indicated in block 172. Such input devices include, but are not limited to: (i) a radio frequency (RF) section that receives, for example, RF signals transmitted over the air by a broadcaster; (ii) component (COMP) input terminals (or a set of COMP input terminals); (iii) universal serial bus (USB) input terminals; and / or (iv) high-definition multimedia interface (HDMI) input terminals. Figure 1C Other examples not shown include composite video.

[0096] In various embodiments, the input device of block 172 has associated corresponding input processing elements as known in the art. For example, the RF section may be associated with elements suitable for: (i) selecting a desired frequency (also known as selecting a signal, or limiting a signal band to a band), (ii) down-converting the selected signal, (iii) re-band-limiting the signal to a narrower band to select (e.g.,) a signal band that may be referred to as a channel in some embodiments), (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired data packet stream. The RF section in various embodiments includes one or more elements for performing these functions, such as frequency selectors, signal selectors, band limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF section may include tuners that perform various functions among these functions, including, for example, down-converting a received signal to a lower frequency (e.g., intermediate frequency or near-baseband frequency) or down-converting it to baseband. In one set-top box embodiment, the RF section and its associated input processing elements receive RF signals transmitted via a wired (e.g., cable) medium and perform frequency selection by filtering, down-converting, and re-filtering to a desired frequency band. Various embodiments rearrange the order of the components described above (and others), remove some of these components, and / or add other components that perform similar or different functions. Adding components may include inserting components between existing components, such as, for example, inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0097] Additionally, the USB and / or HDMI terminals may include corresponding interface processors for connecting system 150 to other electronic devices across USB and / or HDMI connections. It should be understood that various aspects of input processing (e.g., Reed-Solomon error correction) may be implemented as needed, for example, within a separate input processing IC or within processor 152. Similarly, various aspects of USB or HDMI interface processing may be implemented as needed, either within a separate interface IC or within processor 152. Demodulation, error correction, and demultiplexing streams are provided to various processing elements, including, for example, processor 152 and encoder / decoder 156, which operate in conjunction with memory and storage elements to process the data streams as needed for presentation on an output device.

[0098] Various components of system 150 can be provided within an integrated housing in which various components can be interconnected and transmit data therebetween using a suitable connection arrangement 174 (e.g., internal buses as known in the art, including inter-IC (I2C) buses, wiring and printed circuit boards).

[0099] System 150 includes a communication interface 160 that enables communication with other devices via a communication channel 162. The communication interface 160 may include, but is not limited to, a transceiver configured to transmit and receive data via the communication channel 162. The communication interface 160 may include, but is not limited to, a modem or network interface card (NIC), and the communication channel 162 may be implemented, for example, within a wired and / or wireless medium.

[0100] In various embodiments, a wireless network such as Wi-Fi (e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers)) is used to stream or otherwise provide data to system 150. In these embodiments, Wi-Fi signals are received via a communication channel 162 and a communication interface 160 suitable for Wi-Fi communication. The communication channel 162 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other over-the-top communications. Other embodiments use a set-top box to provide streaming data to system 150, delivering data via an HDMI connection to input block 172. Still other embodiments use an RF connection to input block 172 to provide streaming data to system 150. As indicated above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, such as cellular networks or Bluetooth networks.

[0101] System 150 can provide output signals to various output devices, including display 176, speaker 178, and other peripheral devices 180. Display 176 in various embodiments includes one or more of, for example, a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. Display 176 can be used in televisions, tablet computers, laptop computers, cellular phones (mobile phones), or other devices. Display 176 can also be integrated with other components (e.g., as in a smartphone) or separate (e.g., an external monitor for a laptop computer). In various examples of embodiments, other peripheral devices 180 include one or more of a stand-alone digital video disc (or digital universal disc) (DVR, for both terms), a disk player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 180 that provide functionality based on the output of system 150. For example, a disk player performs the function of playing the output of system 150.

[0102] In various embodiments, signaling (such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols enabling device-to-device control with or without user intervention) is used to transmit control signals between system 150 and display 176, speaker 178, or other peripheral devices 180. Output devices can be communicatively coupled to system 150 via dedicated connections through corresponding interfaces 164, 166, and 168. Alternatively, output devices can be connected to system 150 via communication interface 160 using communication channel 162. In electronic devices (such as, for example, televisions), display 176 and speaker 178 can be integrated into a single unit with other components of system 150. In various embodiments, display interface 164 includes a display driver, such as, for example, a timing controller (TCon) chip.

[0103] For example, if the RF portion of input 172 is part of a separate set-top box, then display 176 and speaker 178 can alternatively be separated from one or more other components. In various embodiments where display 176 and speaker 178 are external components, output signals can be provided via dedicated output connections, including, for example, HDMI ports, USB ports, or COMP outputs.

[0104] System 150 may include one or more sensor devices 168. Examples of sensor devices that may be used include one or more GPS sensors, gyroscope sensors, accelerometers, light sensors, cameras, depth cameras, microphones, and / or magnetometers. Such sensors can be used to determine information such as the user's position and orientation. Where system 150 is used as a control module (such as control modules 124, 132) for an extended reality display, the user's position and orientation can be used to determine how image data is rendered so that the user perceives the correct portion of a virtual object or scene from the correct perspective. In the case of a head-mounted display device, the device's own position and orientation can be used to determine the user's position and orientation for the purpose of rendering virtual content. In the case of other display devices such as telephones, tablets, computer monitors, or televisions, other inputs can be used to determine the user's position and orientation for the purpose of rendering content. For example, the user can use a touchscreen, keypad or keyboard, trackball, joystick, or other inputs to select and / or adjust the desired viewpoint and / or direction of observation. When the display device has sensors such as accelerometers and / or gyroscopes, the viewpoint and orientation used for rendering content can be selected and / or adjusted based on the movement of the display device.

[0105] The embodiments may be executed by computer software implemented by processor 152, or by hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. As a non-limiting example, memory 154 may be of any type suitable for the technical environment and may be implemented using any suitable data storage technology, such as optical storage devices, magnetic storage devices, semiconductor-based memory devices, fixed memory, and removable memory. As a non-limiting example, processor 152 may be of any type suitable for the technical environment and may encompass one or more of microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures.

[0106] Block-based video coding Similar to HEVC, VVC is built on a block-based hybrid video coding framework. Figure 2A A block diagram of a block-based hybrid video coding system 200 is given. Variations of this encoder 200 are considered, but for clarity, the encoder 200 is described below without describing all anticipated variations.

[0107] Before being encoded, the video sequence may undergo pre-coding (204), for example, applying a color transform to the input color image (e.g., converting from RGB 4:4:4 to YCbCr 4:2:0), or performing remapping on the input image components to obtain a more compression-resistant signal distribution (e.g., using histogram equalization of one of the color components). Metadata may be associated with the pre-processing and attached to the bitstream.

[0108] The input video signal 202, including the image to be encoded, is segmented (206) and processed block by block, for example, in units of CUs. Different CUs may have different sizes. In VTM-1.0, a CU may be as large as 128x128 pixels. However, unlike HEVC, which is based solely on quadtree-based segmentation blocks, in VTM-1.0, the coding tree unit (CTU) is split into CUs to accommodate the local characteristics of variations based on quadtrees / binaries / tritrees. Furthermore, the concept of multiple segmentation unit types in HEVC is removed, so the separation of CUs, prediction units (PUs), and transform units (TUs) no longer exists in VVC-1.0; instead, each CU is always used as the basic unit for prediction and transform without further segmentation. In the multi-type tree structure, the CTU is first segmented using a quadtree structure. Then, each quadtree leaf node can be further segmented using binary and ternary tree structures. Different segmentation types can be used, such as quadtree segmentation, vertical binary segmentation, horizontal binary segmentation, vertical ternary segmentation, and horizontal ternary segmentation.

[0109] exist Figure 2A In the encoder, spatial prediction (208) and / or temporal prediction (210) can be performed. Spatial prediction (or “intra-frame prediction”) uses pixels from samples (referred to as reference samples) of adjacent blocks already encoded in the same video picture / slice to predict the current video block. Spatial prediction reduces the spatial redundancy inherent in the video signal. Temporal prediction (also referred to as “inter-frame prediction” or “motion-compensated prediction”) uses reconstructed pixels from already encoded video pictures to predict the current video block. Temporal prediction reduces the temporal redundancy inherent in the video signal. The temporal prediction signal for a given CU can be signaled by one or more motion vectors (MVs) indicating the amount and direction of motion between the current CU and its temporal reference. In addition, if multiple reference pictures are supported, a reference picture index can be additionally sent to identify which reference picture in the reference picture storage device (212) the temporal prediction signal comes from.

[0110] The mode decision block (214) in the encoder selects the optimal prediction mode, such as a rate-distortion optimization method. This selection can be made after performing spatial and / or temporal predictions. Intra-frame / inter-frame decisions can be indicated by, for example, a prediction mode flag. The prediction block (216) is subtracted from the current video block to generate the prediction residual. The prediction residual is decorrelated using transform (218) and quantization (220). (For some blocks, the encoder can bypass both transform and quantization, in which case the residual can be encoded directly without applying the transform or quantization process.) The quantized residual coefficients are inversely quantized (222) and inversely transformed (224) to form the reconstructed residual, which is then added back to the prediction block (226) to form the reconstructed signal of the CU. Before the reconstructed CU is placed in the reference image storage device (212) and used for encoding future video blocks, further loop filtering (228) can be applied to the reconstructed CU, such as deblocking / SAO (sample adaptive offset) filtering, to reduce coding artifacts. In order to form the output video bitstream 230, the coding mode (inter-frame or intra-frame), prediction mode information, motion information and quantization residual coefficients are all sent to the entropy coding unit (108) for further compression and packing to form the bitstream.

[0111] Figure 2B A block diagram of a block-based video decoder 250 is given. In decoder 250, the bitstream is decoded by decoder elements, as described below. Video decoder 250 typically performs operations similar to... Figure 2A The encoding process described herein is the reverse of the decoding process. Encoder 200 typically also performs video decoding as part of the encoding of video data.

[0112] Specifically, the decoder's input includes a video bitstream 252, which can be generated by the video encoder 200. The video bitstream 252 is first unpacked and entropy decoded at the entropy decoding unit 254 to obtain transform coefficients, motion vectors, and other encoded information. Image segmentation information indicates how the image is segmented. Therefore, the decoder can segment (256) the image based on the decoded image segmentation information. The encoding mode and prediction information are sent to the spatial prediction unit 258 (if intra-frame coding) or the temporal prediction unit 260 (if inter-frame coding) to form a prediction block. The residual transform coefficients are sent to the inverse quantization unit 262 and the inverse transform unit 264 to reconstruct the residual block. The prediction block and the residual block are then added together at 266 to generate a reconstructed block. The reconstructed block can further undergo loop filtering 268 before being stored in the reference image storage device 270 for use in predicting future video blocks.

[0113] The decoded image 272 can undergo further post-decoding processing (274), such as inverse color transformation (e.g., from YCbCr 4:2:0 to RGB 4:4:4) or inverse remapping of the remapping process performed in pre-encoding processing (204). The post-decoding processing can utilize metadata derived in the pre-encoding process and signaled in the bitstream. The decoded, processed video can be sent to display device 276. Display device 276 can be a device separate from decoder 250, or decoder 250 and display device 276 can be components of the same device.

[0114] The various methods and other aspects described in this application can be used to modify modules of the video encoder 200 or decoder 250. Furthermore, the systems and methods disclosed herein are not limited to VVC or HEVC and can be applied to, for example, other standards and recommendations, whether pre-existing or future-developed, and any extensions of such standards and recommendations (including VVC and HEVC). Unless otherwise indicated or technically excluded, the aspects described in this application can be used individually or in combination.

[0115] Figure 3A This illustration shows a schematic side view of an example waveguide display that can be used with extended reality (XR) applications according to some embodiments. The image is projected by an image generator 302. The image generator 302 can project the image using one or more of a variety of technologies. For example, the image generator 302 can be a laser beam scanning (LBS) projector, a liquid crystal display (LCD), a light-emitting diode (LED) display (including organic LED (OLED) or micro LED (µLED) displays), a digital light processor (DLP), a liquid crystal on silicon (LCoS) display, or other types of image generators or light engines.

[0116] The light representing image 312 generated by image generator 302 is coupled into waveguide 304 via in-coupler 306. In-coupler 306 diffracts the light representing image 312 into one or more diffraction orders. For example, ray 308, representing a portion of the bottom of the image, is diffracted by in-coupler 306, and one of the diffraction orders 310 (e.g., the second order) is at an angle capable of propagating through waveguide 304 via total internal reflection. Image generator 302 displays the image according to instructions from control module 324, which operates to render image data, video data, point cloud data, or other displayable data.

[0117] At least a portion of the light 310, already coupled into waveguide 304 by diffraction-in coupler 306, is coupled out of the waveguide by diffraction-out coupler 314. At least some of the light coupled out of waveguide 304 replicates the incident angle of the light coupled into the waveguide. For example, in the illustration, out-coupled rays 316a, 316b, and 316c replicate the angle of the input coupled ray 308. Because the light leaving the out-coupler replicates the direction of the light entering the in-coupler, the waveguide essentially replicates the original image 312. The user's eye 318 can focus on the replicated image.

[0118] exist Figure 3A In the example, the outgoing coupler 314 outputs only a portion of the coupled light, where each reflection allows a single input beam (such as beam 308) to generate multiple parallel output beams (such as beams 316a, 316b, and 316c). Thus, even if the user's eye is not perfectly aligned with the center of the outgoing coupler, at least some of the light originating from each part of the image may reach the user's eye. For example, if the eye 318 moves downwards, beam 316c may enter the eye even if beams 316a and 316b do not, so the user can still perceive the bottom of image 312 despite the positional shift. Therefore, the outgoing coupler 314 partially functions as an exit pupil expander in the vertical direction. The waveguide may also include one or more additional exit pupil expanders (…). Figure 3A (not shown in the image) to expand the exit pupil in the horizontal direction.

[0119] In some embodiments, waveguide 304 is at least partially transparent to light originating outside the waveguide display. For example, at least some of the light 320 from a real-world object (such as object 322) passes through waveguide 304, allowing the user to see the real-world object when using the waveguide display. Since the light 320 from the real-world object also passes through diffraction grating 314, there will be multiple diffraction orders, and therefore multiple images. To minimize the visibility of multiple images, it is desirable that the zeroth-order diffraction (without the bias of 314) has high diffraction efficiency for both light 320 and the zeroth order, while higher diffraction orders are lower in energy. Therefore, in addition to extending and coupling virtual images, the out-coupler 314 is preferably configured to allow the zeroth order of the real image to pass through. In such embodiments, the image displayed by the waveguide display may appear to be superimposed on the real world.

[0120] Figure 3BThis is a schematic side view illustrating an example alternative display type that can be used with extended reality applications according to some embodiments. In the XR head-mounted display device 330, a control module 332 controls a display 334 (which may be an LCD) to display images. The head-mounted display includes a partially reflective surface 336 that reflects (and in some embodiments, both reflects and focuses) the image displayed on the LCD to make the image visible to the user. The partially reflective surface 336 also allows at least some external light to pass through, thereby allowing the user to see their surroundings.

[0121] Figure 3C This is a schematic side view illustrating example alternative display types that can be used with extended reality applications according to some embodiments. In an XR head-mounted display device 340, a control module 342 controls a display 344 (which may be an LCD) to display an image. The image is focused by one or more lenses of a display optics 346 to make the image visible to the user. Figure 3C In this example, the external light does not reach the user's eyes directly. However, in some such embodiments, an external camera 348 can be used to capture images of the external environment and display such images on a display 344 along with any virtual content that may also be displayed.

[0122] The embodiments described herein are not limited to any particular type or structure of XR display device.

[0123] This application belongs to the field of point cloud compression and processing. This field aims to develop tools for the compression, analysis, interpolation, representation, and understanding of point cloud signals.

[0124] Point clouds are a common data format across several commercial sectors, from autonomous driving, robotics, AR / VR, civil engineering, computer graphics to animation / film. 3D LiDAR sensors have already been deployed in self-driving cars, and affordable LiDAR sensors have been released from Velodyne Velabit, Apple iPad Pro 2020, and Intel RealSense LiDAR camera L515. With advancements in sensing technology, 3D point cloud data is becoming more practical than ever before and is expected to be the ultimate driving force in the aforementioned applications.

[0125] Point cloud data is also considered to consume a significant portion of network traffic, for example, in cars connected via 5G networks and in immersive communications (VR / AR). An efficient representation format is essential for point cloud understanding and communication. In particular, raw point cloud data needs to be properly organized and processed for world modeling and sensing purposes. Compression of the raw point cloud is necessary when data needs to be stored and transmitted in relevant scenarios.

[0126] Furthermore, point clouds can represent continuous scans of the same scene containing multiple moving objects. These are called dynamic point clouds compared to static point clouds captured from static scenes or objects. Dynamic point clouds are typically organized into frames, with different frames captured at different times. Dynamic point clouds may require real-time or low-latency processing and compression.

[0127] The automotive industry and autonomous vehicles are among the sectors where point clouds can be used. Autonomous vehicles should be able to "detect" their environment to make sound driving decisions based on the realities of their immediate surroundings. For example, typical LiDAR sensors generate (dynamic) point clouds used by perception engines. These point clouds are not intended to be seen by the human eye, and they are typically sparse, not necessarily colored, and dynamic, with a high capture frequency. They can possess other properties, such as reflectivity provided by LiDAR, as this property indicates the material of the sensed object and can aid in decision-making.

[0128] Virtual reality (VR) and immersive worlds have become hot topics and are foreseen by many as the future of 2D flat video. The basic idea is to immerse the viewer in the environment around them, rather than as with standard TV where they can only see a virtual world in front of them. Depending on the viewer's degree of freedom within the environment, there are several levels of immersion. Point clouds are a good candidate format for distributing VR worlds. They can be static or dynamic and typically have an average size, say, no more than a few million points at a time.

[0129] Point clouds can also be used for various purposes, such as cultural heritage / buildings, where objects like statues or buildings are 3D scanned to share their spatial configurations without needing to send or access the object. Furthermore, this is a way to preserve knowledge of an object in the event of potential damage (e.g., a temple destroyed by an earthquake). Such point clouds are typically static, colored, and massive.

[0130] Another use case is topography and cartography, where 3D representation is used, maps are not limited to a flat plane, and can include terrain. Google Maps is now a good example of 3D maps, but uses a grid instead of point clouds. However, point clouds may be a suitable data format for 3D maps, and such point clouds are typically static, colored, and huge.

[0131] World modeling and sensing via point clouds may be a necessary technology to allow machines to acquire knowledge about the 3D world around them, which is crucial for the applications discussed above.

[0132] The present invention was designed with the above considerations in mind.

[0133] 3D point cloud data is essentially discrete samples of the surface of an object or scene. To fully represent the real world using these point samples, a massive number of points are actually required. For example, a typical VR immersive scene contains millions of points, while a point cloud typically contains hundreds of millions. Therefore, processing such large-scale point clouds is computationally expensive, especially for consumer devices with limited computing power, such as smartphones, tablets, and car navigation systems.

[0134] The first step in any processing or inference of point clouds is to have an efficient storage method. To store and process the input point cloud at an affordable computational cost, one solution is to first downsample it, where the downsampled point cloud summarizes the geometry of the input point cloud while having far fewer points. The downsampled point cloud is then fed into subsequent machine tasks for further use. However, further reductions in storage space can be achieved by converting the raw point cloud data (raw or downsampled) into a bitstream using entropy coding techniques for lossless compression. A better entropy model results in a smaller bitstream, and thus more efficient compression. Furthermore, the entropy model can be paired with downstream tasks, allowing the entropy encoder to preserve task-specific information during compression.

[0135] In addition to lossless coding, many scenarios seek lossy coding to significantly improve the compression ratio while keeping the resulting distortion at a certain quality level.

[0136] The breakdown of the occupied voxels continues until the final octree depth level. The leaves of the octree ultimately represent the point cloud.

[0137] On the encoder side, octree nodes (node ​​values) are typically sent to the entropy encoder to generate a bitstream. The decoder then uses the decoded octree node values ​​to reconstruct the octree structure and ultimately reconstructs the point cloud based on the leaf nodes of the octree structure.

[0138] To efficiently entropy encode octree nodes, a probability distribution model is typically used to assign shorter symbols to octree node values ​​that appear with a higher probability.

[0139] For typical learning-based point cloud encoding, the encoder transforms the input point cloud into a feature map to be entropy encoded. The learning-based decoder performs the inverse transformation and reconstructs the point cloud. Each feature in the feature map can be associated with a block of point cloud. Such blocks of point cloud are typically constructed via hierarchical octree decomposition. In the case where the octree (decomposition) is encoded via a learning-based method, the encoder and decoder generate feature descriptors for each octree block. Such octree feature descriptors are simply dumped and are not used in the feature aggregation pipeline of the encoder or decoder. This application provides a method for utilizing octree features to benefit feature aggregation.

[0140] In this application, octree features are used not only for octree coding but also to assist in deep feature coding. This application discusses an improvement to learning-based point cloud coding. The idea is to use octree features generated during deep octree coding for improved deep feature coding. Such octree features are available during learning-based octree coding and can be further used in deep feature-based coding with a limited increase in complexity. Compared to existing bottom-up features in deep feature-based pipelines, octree features can be viewed as top-down features that include global information from point cloud frames. Therefore, octree features are beneficial for feature aggregation for compression.

[0141] Deep feature encoding Figure 4 This is a flowchart illustrating an example of a learning-based point cloud codec according to some embodiments. Figure 4 A typical point cloud compression system 400 based on advanced learning is shown. Figure 4 On the upper left is the point cloud encoder 402. The input point cloud PC, at a higher resolution, is fed to the point cloud encoder. The input point cloud undergoes a series of feature extractions 404 and feature aggregations 406 with possible downsampling of the point cloud "↓". Each step of such feature extraction / aggregation produces a feature map associated with a set of blocks (also called octree voxels), if aligned with octree decomposition. During this process, the resulting features are referred to in this application as bottom-up features because the processing begins at the finest detail in the original resolution. The feature maps are encoded into a bitstream BS1. This branch is referred to in this application as deep feature encoding.

[0142] Deep octree coding Parallel to the deep feature encoding is a second branch for encoding the block positions. In this application, the second branch is referred to as Deep Octree Encoding 408. The intention is to encode the positions of blocks associated with the feature map. Figure 4 In the "octree coding" process 408, a set of blocks is taken as input, and this set of blocks is... Figure 4The point cloud is labeled as a block map BM and is associated with feature map FM1. In some embodiments, octree encoding can encode the block at the end of the deep feature encoding, which is the block used for feature map FM2. This can be viewed as merging the "feature aggregation" block into the "feature extraction" block. The reason for performing octree encoding is that the basic shape of the point cloud needs to be transmitted accurately or losslessly. Octree encoding 408 generates a bitstream BS2, which sets the (basic) quality of the reconstructed point cloud.

[0143] In some embodiments, deep learning methods may be used for octree coding block 408. Octree coding uses a strategy to encode the octree decomposition structure starting from the root node. Deep learning methods are used to aggregate features and then predict the occupancy probability of octet nodes. See applications '790 and '609 for more details on the octree feature set. In some embodiments, all such learning-based octree codes are generated as feature descriptors before predicting the occupancy probability. Such feature descriptors are referred to herein as octree features.

[0144] decoding exist Figure 4 On the upper right is the point cloud decoder 410. Decoder 410 has two branches corresponding to the depth feature encoding and octree encoding at the encoder: a depth feature-based decoding branch and an octree decoding branch. The depth feature decoding branch consists of a series of feature aggregations 412 and / or point synthesis 416. Each step in feature aggregation / point synthesis may involve an upsampling operation "↑". The depth feature encoding branch takes the bitstream BS1 as its input.

[0145] exist Figure 4 In the decoding section, octree coding block 414 takes BS2 as input and outputs a block graph BM. The block graph serves as additional input to the "feature aggregation" process. For some embodiments, "octree decoding" block 416 is essentially the same as "octree coding" block 408. Both blocks include a series of steps to construct the octree structure, except that "octree decoding" block 416 acquires the octree bitstream and performs adaptive arithmetic decoding using the predicted context, while "octree coding" (or "octree coding" in some embodiments) block 408 performs arithmetic coding.

[0146] Deep feature encoding using octree features Figure 5This diagram illustrates a process flow of an example learning-based point cloud codec according to some embodiments. In a typical framework of learning-based point cloud coding, octree features are limited to encoding octree voxel occupancy. In this application, octree features can be used to enhance deep feature coding branches. Compared to existing bottom-up features in deep feature-based pipelines, octree features can be viewed as top-down features that incorporate global information about the point cloud frame. Therefore, octree features can be beneficial for feature aggregation for compression.

[0147] Figure 5 An example design 500 demonstrates how to use octree features in deep feature encoding. Figure 4 In comparison, Figure 5 In this paper, a new connection 502 is introduced from the octree branch (or block 504) to the feature branch (or block 506).

[0148] For some embodiments, such as Figure 5 As shown on the upper right, the octree feature (OF) 510 from the octree decoding block 514 is output to the point synthesis block 508. In some embodiments, in addition to the feature map... In addition to (512), the octree feature OF (510) is used as the input to the point synthesis block 508. The two features (OF (510) and (512) are all associated with the same block graph BM (516). Therefore, the octree feature OF 510 (in the d0 dimension) is concatenated with the original feature descriptor. At the top of 512 (in d1 dimension), and the (d0+d1) dimension features are used as new inputs for the first layer of point synthesis block 508. Direct concatenation provides a way to fuse information from octree features and original features.

[0149] In some embodiments, the point cloud encoder side can be designed as follows: Figure 5 As shown on the left side. The octree feature OF (502) is output from the octree coding block 504 to the feature aggregation block 506. In some embodiments, the octree feature OF (502) is also used as input to the feature aggregation block 506 in addition to the feature map FM1 (518). Both features (OF (502) and FM1 (518)) are associated with the same block map BM. Similarly, in some embodiments, the octree feature OF 502 (in the d0 dimension) is concatenated on top of the original feature descriptor FM1518 (in the d1 dimension), and the (d0+d1) dimension feature is used as a new input to the first layer of the feature aggregation block 506. Direct concatenation provides a way to fuse information from the octree feature and the original feature. For example, see Figure 7 For some embodiments, an alternative way of fusing the octree feature OF and the feature map FM1 is described below.

[0150] In some embodiments, a new OF connection is made simultaneously on both the encoder and decoder sides. Although both cascaded connections are... Figure 5 As shown in the diagram, however, OF connections can exist on the decoder side but not on the encoder side, and vice versa.

[0151] Deep feature encoding using octree features Figure 6 This diagram illustrates the process of an example learning-based point cloud encoder / decoder according to some embodiments. Compared to the framework for deep feature encoding using octree features discussed in previous sections, Figure 6 This shows a further increase.

[0152] In some embodiments, with Figure 5 Compared to the framework, the only difference is that Figure 6 On the decoder side, octree features OF 602 and 604 are output to both point synthesis block 608 and feature aggregation block 606. Furthermore, in some embodiments, block graph 612 can be provided to feature aggregation block 606. Octree decoding generates hierarchical features and octree structures up to the feature graph. The statement implies that the octree decoding block 610 can target resolutions from [resolution]. arrive Each feature map provides octree features. In some embodiments, in feature aggregation block 606, appropriate octree features are concatenated to the original input features at each corresponding layer.

[0153] On the encoder side, the block map (BM) is different from the feature map (FM1). The feature map FM1 is a sparse tensor with coordinates equal to BM and a set of feature vectors that exist / are defined at these coordinates. The feature map FM1 includes the features at the top of the block map plus the coordinates of the block map.

[0154] Will Figure 6 Design 600 and Figure 5 Comparing the designs, Figure 6 The octree feature may have a greater impact on the final reconstruction quality because Figure 6 The octree features begin to influence feature aggregation at a much earlier stage of decoding. In other words, for some embodiments, due to the... Figure 5 In comparison, OF was injected earlier. Figure 6 In the decoder system, the effect due to OF injection is therefore more pronounced. Although Figure 6The diagram shows three OF concatenations 602, 604, and 614, two of which are on the decoder side, but any combination of OF concatenations can be used. For example, an OF concatenation from an octree decoder block can be provided to a feature aggregation block instead of a point synthesis block.

[0155] Using octree features to augment deep feature encoding branches requires a limited amount of additional complexity. The main reason is that octree features are generated during octree encoding branches, and they are a free resource for deep feature encoding branches. For some implementations, the only additional complexity is the increase in dimensionality of the input features for deep feature encoding from a concatenated design. Such an increase in dimensionality can be reduced when they raise concerns.

[0156] Figure 7 This is a flowchart illustrating an example process of increasing the processing dimensionality according to some embodiments and computing fused features based on octree features. One approach is to introduce a fully connected layer or MLP to fuse the original input features encoded by octree features and deep features. This is in Figure 7 Example feature aggregation block 700 is shown. The “⊕” symbol 702 signifies concatenation, and MLP 704 is a new proposal prior to the original feature aggregation module for fusion purposes. Several FA layers 706, 708, and 710 represent the structure of the earlier feature aggregation module. The new input feature f2 (712) is set to the same dimensions as the earlier input feature f1 (714). This will at least avoid any changes to the earlier feature set design. In some embodiments, information from the input feature f1 (714) and the octree feature OF (716) is fused into the new feature f2 (712).

[0157] In some embodiments, the "⊕" operator 702 can be replaced by the addition operator "+", which adds f1 (714) and OF (716) instead of concatenating them. To enable the addition operation, the octree feature 716 can be processed / transformed using an additional MLP module.

[0158] Figure 8 It is a diagram illustrating the process of example feature extraction during encoding and feature aggregation during decoding according to some embodiments. Figure 8An example implementation of the feature extraction / aggregation block is shown. In some embodiments, the feature extraction / aggregation block 800 takes feature map f1 (802) as input and starts with convolutional layer 804 to upsample and downsample by 2x in each of the x, y, and z directions. This is followed by two convolutional layers 806 and 808 and a transformer block 810 to enhance the representability of the features and output a new feature map f2 (812). See Zhao, Hengshuang et al., “Point Transformer, ICCV 16239-16248 (2021)” (“Zhao”), Mao, Jiageng et al., “Voxel transformer for 3d object detection”, In Proceedings of the IEEE / CVF International Conference on Computer Vision 3164-3173 (2021) (“Mao”), and Zhang, Cheng et al., “PVT: Point-VoxelTransformer for Point Cloud Learning”, arXiv preprint arXiv: 2108.06076 (2021) (“Zhang”).

[0159] Figure 9 This is a diagram illustrating the process of example feature aggregation / point synthesis during decoding according to some embodiments. Figure 9 An example implementation of feature aggregation / point synthesis is shown. In some embodiments, the feature extraction / aggregation block 900 takes feature map f2 (902) as input and starts with convolutional layer 904 to upsample by 2x in each of the x, y, and z directions. This is followed by two convolutional layers 906 and 908 and a transformer block 910 to enhance the representativeness of the features. Finally, a new feature map f1 (912) is output. In the case of point synthesis, the new feature map f1 (912) is 3D and has (x, y, z) coordinates of the reconstructed point cloud.

[0160] Figure 10 This is a diagram illustrating an example of deep octree-based encoding according to some embodiments. Figure 10Example Figure 1000 of deep octree encoding is shown. PC1 (1002) and PC2 (1004) represent point clouds at the previous and current octree levels. The serialization process 1006 traverses the voxels to be encoded sequentially and outputs a series of occupancy bits O2 (1008). Using O2 (1008), the point cloud at the current octree level, i.e., PC2 (1004), can be fully represented. Next, each bit / voxel in O2 (1008) is processed by the context building module 1010 and then encoded by the adaptive arithmetic encoder block 1012. For the current voxel, the context building module 1010 first estimates its occupancy probability (denoted by p) based on the context of neighboring voxels that have already been encoded. In particular, the context can be n binary occupancy bits of neighboring voxels in the current octree level and / or the previous octree level. The arithmetic encoder encodes the current occupancy bit based on the estimated probability p. By repeating this process for each bit in O2, the current bit stream BS2 (1014) is generated.

[0161] Figure 11 This is a diagram illustrating an example of a deep octree-based decoding process according to some embodiments. Figure 11 An example diagram 1100 shows the decoding based on a deep octree. An adaptive arithmetic decoder block 1102 decodes each occupancy bit from the received bitstream BS2 to obtain O2. To decode the occupancy bits, the arithmetic decoder takes not only BS2 (1104) as input but also the occupancy probability p 1106 estimated by the context building module 1108. Similar to the encoder, the context building module 1108 also takes the context of the already decoded voxels (from the current octree level and / or previous octree levels) as input and outputs the probability p 1108. The occupancy bits O2 (1110) are assigned to the voxel grid at the current octree level by the deserialization block 1112, which performs this assignment by checking the already decoded octree level PC1 (1114). The deserialization block outputs the decoded point cloud at the current octree level PC2 (1116).

[0162] Feature aggregation In applications, MLPs and convolutional layers can be replaced or added with high-level feature aggregation microarchitectures such as Inception ResNets (IRNs) or transformer blocks. See Szegedy, Christian et al., “Inception-v4, inception-resnet and the impact of residual connections on learning”, 31:1 INPROCEEDINGS OF THE AAAI CONFERENCE ON ARTIFICIAL INTELLIGENCE (2017) (“Szegedy”); Zhao; Mao; and Zhang.

[0163] Inception ResNet (IRN) Figure 12 This is a flowchart illustrating an example Inception-ResNet block for feature aggregation according to some embodiments. Figure 12 The example provides an IRN architecture 1200. This example illustrates the architecture of an IRN block 1200 for aggregating features with D channels. Here, "CONV N" means a convolutional layer that accepts input with N channels. In this example, each CONV D / 4 block 1202, 1204, 1208 is followed by a Recti-Linear Unit (ReLU) block 1214, 1216, 1218. The outputs of each in the CONV D / 2 modules 1206, 1210 are concatenated by a concatenation module 1220. The plus sign ("⊕") 1222 indicates summation.

[0164] converter Figure 13 This is a process diagram illustrating an example transformer block for feature aggregation according to some embodiments. Figure 13 The diagram shows transformer block 1300, where “⊕” (1304, 1308) again represents summation. Figure 13 A basic diagram of converter block 1300 is shown, which consists of self-focused block 1302 with residual connections and MLP block 1306 (composed of several MLP layers) with residual connections.

[0165] Figure 14 This is a process diagram illustrating an example self-focusing block according to some embodiments. The block diagram of self-focusing block 1400 is shown in... Figure 14 It is shown in the image. Details are as follows.

[0166] Given the current feature vector f associated with voxel position A A (1402) and its relationship with voxel position Ai Associated k neighboring features A i (0≤i≤k-1) are the k nearest neighbors of A in the input sparse tensor. Self-focused block 1400 is dedicated to based on all neighbor features. (1410) to update feature f A (1402). First, by making feature f A (1402) The point A is obtained by performing a k-nearest neighbor (kNN) search based on the coordinates of A through kNN block 1426. i (1404). Then, the query embedding Q of A is calculated using the following formula. A : , Where the current feature vector f is made A (1402) Pass through MLP block 1406 to generate Q A (1408).

[0167] After that, compute the key embeddings of all nearest neighbors of A. (1420) sum value embedding (1422): Among them, MLP Q (·)(1406)MLP K (·)(1412)and MLP V (·) (1414) are the MLP layers used to obtain query, key, and value entities, respectively, while (1418) are voxels A and A i The positional encoding between them is calculated using the following formula: MLP P (·) (1416) is the MLP layer used to obtain position encoding, P A and P Ai These are 3D coordinates, and they are voxels A and A'. i The center of the self-focused block. The output feature of the position A of the self-focused block is: Where σ(·) is the Softmax normalization function 1424, and d is the eigenvector f A The length of , where c is a predefined constant.

[0168] The transformer block updates the feature maps at all locations in the same manner and then outputs the updated feature maps. Note that in a simplified embodiment, the MLP... Q (·), MLPK (·) and MLP V (·) and MLP P (·) can contain only one fully connected layer, which corresponds to a linear projection.

[0169] Figure 15 This is a flowchart illustrating an example encoding process according to some embodiments. In some embodiments, example process 1500 may include obtaining 1502 information including a point cloud. In some embodiments, example process 1500 may further include passing the point cloud 1504 through a feature extraction process to generate a first feature. In some embodiments, example process 1500 may further include obtaining 1506 octree features from an octree encoding process. In some embodiments, example process 1500 may further include generating 1508 a second feature by fusing the octree features with the first feature. In some embodiments, example process 1500 may further include aggregating features 1510 based on the second feature.

[0170] Figure 16 This is a flowchart illustrating an example decoding process according to some embodiments. In some embodiments, example process 1600 may include obtaining 1602 octree features from an octree encoding process. In some embodiments, example process 1600 may further include obtaining 1604 input features. In some embodiments, example process 1600 may further include determining 1606 new input features by fusing the octree features with the input features. In some embodiments, example process 1600 may further include aggregating 1608 features based on the new input features.

[0171] While the methods and systems according to some embodiments are generally discussed in the context of extended reality (XR), some embodiments can be applied to any XR context, such as, for example, virtual reality (VR) / mixed reality (MR) / augmented reality (AR) contexts. Furthermore, although the term "head-mounted display (HMD)" is used herein according to some embodiments, for some embodiments, some embodiments can be applied to, for example, wearable devices with XR, VR, AR, and / or MR capabilities (which may or may not be attached to the head).

[0172] A first example method according to some embodiments may include: obtaining information including a point cloud; passing the point cloud through a feature extraction process to generate a first feature; obtaining an octree feature from an octree encoding process; generating a second feature by fusing the octree feature with the first feature; and aggregating the feature based on the second feature.

[0173] In some embodiments of the first example method, the octree feature is used only on one of the encoder and decoder processes.

[0174] In some embodiments of the first example method, octree features are obtained from the octree encoding process based on block graphs corresponding to the first features.

[0175] For some embodiments of the first example method, obtaining octree features from the octree encoding process includes: using a block graph corresponding to the first feature as input; and passing the block graph through the octree encoding process to generate octree features.

[0176] In some embodiments of the first example method, the octree feature corresponds to the first feature.

[0177] For some embodiments of the first example method, aggregating features based on a second feature includes: passing the second feature through one or more convolutional neural network (CNN) blocks; and passing the output of one or more CNN blocks through a transformer process to generate aggregated features.

[0178] In some embodiments of the first example method, the converter process includes one or more multilayer perceptron (MLP) blocks.

[0179] For some embodiments of the first example method, aggregating features based on the second feature includes: performing one or more iterative processes, wherein the iterative process includes: passing the fused feature through a multilayer perceptron (MLP) block; passing the output of the MLP through one or more feature aggregation layers to generate an output feature; and fusing the output feature with an octree feature to generate a fused feature for the next passing through the iterative process, wherein the second feature is used as the fused feature of the first passing through the iterative process, and wherein the output feature becomes the aggregated feature of the last passing through the iterative process.

[0180] For some embodiments of the first example method, the MLP block includes the Inception-ResNet block.

[0181] A first example apparatus according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the apparatus to perform any of the methods listed above.

[0182] A second example method according to some embodiments may include: obtaining octree features from the octree encoding process; obtaining input features; determining new input features by fusing the octree features with the input features; and aggregating features based on the new input features.

[0183] In some embodiments of the second example method, the octree feature is used only in one of the encoder and decoder processes.

[0184] In some embodiments of the second example method, the octree features are obtained from the octree encoding process based on the block graph corresponding to the first feature.

[0185] For some embodiments of the second example method, obtaining octree features from the octree encoding process includes: obtaining a bitstream comprising a block graph previously passed through the encoder's octree encoding process; using the bitstream as input; and passing the bitstream through the octree encoding process to generate octree features.

[0186] In some embodiments of the second example method, the octree feature corresponds to the first feature.

[0187] For some embodiments of the second example method, aggregating features based on the second feature includes: passing the second feature through one or more convolutional neural network (CNN) blocks; and passing the output of one or more CNN blocks through a transformer process to generate aggregated features.

[0188] In some embodiments of the second example method, the converter process includes one or more multilayer perceptron (MLP) blocks.

[0189] For some embodiments of the second example method, aggregating features based on the second feature includes: performing one or more iterative processes, wherein the iterative process includes: passing the fused feature through a multilayer perceptron (MLP) block; passing the output of the MLP through one or more feature aggregation layers to generate an output feature; and fusing the output feature with an octree feature to generate a fused feature for the next passing through the iterative process, wherein the second feature is used as the fused feature of the first passing through the iterative process, and wherein the output feature becomes the aggregated feature of the last passing through the iterative process.

[0190] For some embodiments of the second example method, the MLP block includes the Inception-ResNet block.

[0191] Some embodiments of the second example method may further include passing octree features and aggregated features through a point synthesis process to generate an output point cloud.

[0192] A second example apparatus according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the apparatus to perform any of the methods listed above.

[0193] A third example method according to some embodiments may include: obtaining information including a point cloud; passing the point cloud through a feature extraction process to generate a first feature; obtaining an octree feature from an octree encoding process; generating a second feature by fusing the octree feature with the first feature; and passing the second feature through a feature aggregation process to generate a third feature.

[0194] A third example apparatus according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the apparatus to perform any of the methods listed above.

[0195] A fourth example method according to some embodiments may include: obtaining octree features from an octree encoding process; obtaining a first feature as input; generating a second feature by fusing the octree features with the first feature; and passing the second feature through a feature aggregation process to generate a third feature.

[0196] A fourth example apparatus according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the apparatus to perform any of the methods listed above.

[0197] A fifth example apparatus according to some embodiments may include at least one processor configured to perform any of the methods listed above.

[0198] A sixth example apparatus according to some embodiments may include a computer-readable medium storing instructions for causing one or more processors to perform any of the methods listed above.

[0199] A seventh example apparatus according to some embodiments may include at least one processor and at least one non-transitory computer-readable medium storing instructions for causing at least one processor to perform any of the methods listed above.

[0200] Example signals according to some embodiments may include: a bit stream generated according to any of the methods listed above.

[0201] This application describes various aspects, including tools, features, embodiments, models, methods, etc. Many of these aspects are described in detail, and are generally described in a manner that may sound limiting, at least for the purpose of illustrating the various characteristics. However, this is for the purpose of clarity and does not limit the application or scope of those aspects. In fact, all the different aspects can be combined and interchanged to provide further aspects. Furthermore, the aspects described can also be combined and interchanged with those described in earlier filings.

[0202] The aspects described and contemplated in this application can be implemented in many different forms. While specific embodiments are illustrated, other embodiments are contemplated, and the discussion of particular embodiments does not limit the breadth of implementation. At least one of the aspects generally relates to point cloud encoding and decoding, and at least one other aspect generally relates to the transmission of generated or encoded bitstreams. These and other aspects can be implemented as methods, apparatus, computer-readable storage media having instructions stored thereon for encoding or decoding video data according to any of the methods, and / or computer-readable storage media having a bitstream generated according to any of the methods stored thereon.

[0203] In this application, the terms "reconstruction" and "decoding" are used interchangeably, as are the terms "point," "voxel," and "sample," and the terms "image," "picture," and "frame." Generally, but not necessarily, the term "reconstruction" is used on the encoder side, while "decoding" is used on the decoder side.

[0204] Various methods are described herein, and each method includes one or more steps or actions for implementing the method. Unless the correct operation of the method requires a specific order of steps or actions, the order and / or use of specific steps and / or actions can be modified or combined. Furthermore, terms such as "first," "second," etc., may be used in various embodiments to modify elements, components, steps, operations, etc., such as, for example, "first decoding" and "second decoding." Unless specifically required, the use of such terms does not imply a sequence of modified operations. Therefore, in this example, the first decoding does not need to be performed before the second decoding and can occur, for example, before, during, or within a time period overlapping with the second decoding.

[0205] For example, various numerical values ​​may be used in this application. Specific values ​​are for illustrative purposes only, and the aspects described are not limited to these specific values.

[0206] The embodiments described herein can be executed by computer software implemented by a processor or other hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments can be implemented by one or more integrated circuits. As a non-limiting example, the processor can be of any type suitable for the technical environment and can encompass one or more of microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures.

[0207] Various implementations involve decoding. As used herein, “decoding” can encompass all or part of a process, such as performing on a received encoded sequence to produce a final output suitable for display. In various embodiments, such a process includes one or more processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such a process also includes, or alternatively includes, processes performed by a decoder of the various implementations described herein, such as extracting an image from a tiled (packed) image, determining the upsampling filter to use, and then upsampling the image, and flipping the image back to its intended orientation.

[0208] As a further example, in one embodiment, "decoding" refers only to entropy decoding; in another embodiment, "decoding" refers only to differential decoding; and in yet another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" is intended to specifically refer to a subset of operations or generally to a broader decoding process will be clear based on the specific context of the description.

[0209] Various implementations involve encoding. In a manner similar to the discussion above regarding “decoding,” the term “encoding,” as used herein, can encompass all or part of a process performed, for example, on an input video sequence to produce an encoded bitstream. In various embodiments, such a process includes one or more processes typically performed by an encoder, such as segmentation, differential coding, transform, quantization, and entropy coding. In various embodiments, such a process also includes, or alternatively includes, processes performed by encoders of the various implementations described herein.

[0210] As a further example, in one embodiment, "encoding" refers only to entropy encoding; in another embodiment, "encoding" refers only to differential encoding; and in yet another embodiment, "encoding" refers to a combination of differential and entropy encoding. Whether the phrase "encoding process" is intended to specifically refer to a subset of operations or generally to a broader encoding process will be clear based on the specific context of the description.

[0211] When the accompanying drawings are presented as flowcharts, it should be understood that block diagrams of the corresponding devices are also provided. Similarly, when the accompanying drawings are presented as block diagrams, it should be understood that flowcharts of the corresponding methods / processes are also provided.

[0212] Various embodiments involve rate-distortion optimization. Specifically, during the encoding process, a balance or trade-off between rate and distortion is typically considered, usually with constraints on computational complexity. Rate-distortion optimization is generally expressed as minimizing a rate-distortion function, which is a weighted sum of rate and distortion. Different approaches exist to address the rate-distortion optimization problem. For example, these approaches can be based on extensive testing of all encoding options, including all considered modes or encoding parameter values, and a complete evaluation of their encoding costs and the associated distortion of the reconstructed signal after encoding and decoding. Faster methods can also be used to save encoding complexity, particularly by calculating approximate distortion based on predicting or predicting the residual signal rather than the reconstructed signal. A hybrid of these two approaches can also be used, such as by using approximate distortion only for some of the possible encoding options and full distortion for others. Other methods evaluate only a subset of the possible encoding options. More generally, many methods employ any of a variety of techniques to perform optimization, but optimization is not necessarily a complete evaluation of both encoding costs and associated distortion.

[0213] The implementations and aspects described herein can be implemented, for example, in a method or process, apparatus, software program, data stream, or signal. Even if discussed only in the context of a single form of implementation (e.g., discussed only as a method), the implementation of the discussed features can also be implemented in other forms (e.g., apparatus or program). Apparatus can be implemented, for example, in suitable hardware, software, and firmware. Methods can be implemented, for example, in a processor, which generally refers to a processing device, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices, such as, for example, computers, cellular phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate the transfer of information between end users.

[0214] References to "an embodiment," "an embodiment," "an implementation," or "an implementation," and other variations thereof, mean that a particular feature, structure, characteristic, etc., described in connection with the embodiment is included in at least one embodiment. Therefore, the phrases "in an embodiment," "in an embodiment," "in an implementation," or "in an implementation," and any other variations appearing throughout this application, do not necessarily refer to the same embodiment.

[0215] Additionally, this application may relate to "determining" fragments of various information. Determining information may include one or more of, for example, estimation information, calculation information, prediction information, or information retrieved from memory.

[0216] Furthermore, this application may relate to “accessing” fragments of various information. Accessing information may include one or more of the following, such as receiving information, retrieving information (e.g., retrieving information from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0217] Additionally, this application may relate to "receiving" fragments of various information. As with "access," receiving is intended to be a broad term. Receiving information may include one or more of, for example, accessing information or retrieving information (e.g., retrieving information from memory). Furthermore, "receiving" is generally referred to in one or more ways during operations such as, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0218] To understand this, for example, in the cases of “A / B,” “A and / or B,” and “at least one of A and B,” the use of any of the following “ / ,” “and / or,” and “at least one of…” is intended to cover selecting only the first listed option (A), or only the second listed option (B), or both options (A and B). As a further example, in the cases of “A, B, and / or C” and “at least one of A, B, and C,” such wording is intended to cover selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A, B, and C). This can be extended to as many items as possible listed.

[0219] Furthermore, among other things, as used herein, the term "signaling" also refers to instructing the corresponding decoder to do something. For example, in some embodiments, the encoder signals a specific one of several parameters for selecting region-based filter parameters for artifact removal filtering. Thus, in embodiments, the same parameter is used on both the encoder and decoder sides. Therefore, for example, the encoder can transmit (explicitly signal) the specific parameter to the decoder so that the decoder can use the same specific parameter. Conversely, if the decoder already has the specific parameter as well as other parameters, signaling can be used without transmission (implicitly signaling) to allow only the decoder to know and select the specific parameter. Bit savings are achieved in various embodiments by avoiding the transmission of any actual functionality. It should be understood that signaling can be implemented in a variety of ways. For example, in various embodiments, information is signaled to the corresponding decoder using one or more syntax elements, flags, etc. Although the verb form of the term "signaling" has been referred to above, the word "signal" can also be used as a noun herein.

[0220] The implementation can generate various signals, which are formatted to carry information, such as information that can be stored or transmitted. The information may include, for example, instructions for performing a method or data generated by one of the described implementations. For example, the signal may be formatted to carry a bitstream of the described embodiment. Such a signal may be formatted as, for example, electromagnetic waves (e.g., using the radio frequency portion of the spectrum) or as a baseband signal. Formatting may include, for example, encoding the data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. It is well known that signals can be transmitted via a variety of different wired or wireless links. The signal may be stored on a processor-readable medium.

[0221] We have described several embodiments. Features of these embodiments may be provided individually or in any combination across various claim classes and types. Furthermore, embodiments may include one or more of the following features, devices, or aspects, individually or in any combination across various claim classes and types: • Adapt the residual at the encoder according to any of the embodiments discussed.

[0222] • Includes a bitstream or signal of one or more of the described syntax elements or their variants.

[0223] • Includes bitstreams or signals that convey the syntax of information generated according to any embodiment of the described embodiments.

[0224] • Inserting into the signaling syntax enables the decoder to adapt the residual elements in a manner corresponding to that used by the encoder.

[0225] • Creating and / or transmitting and / or receiving and / or decoding bit streams or signals including one or more of the described syntax elements or their variants.

[0226] • Create and / or transmit and / or receive and / or decode according to any embodiment of the described embodiments.

[0227] • Methods, processes, apparatus, media for storing instructions, media for storing data, or signals according to any of the embodiments described.

[0228] • TVs, set-top boxes, cellular phones, tablet computers, or other electronic devices that perform filter parameter adaptation according to any embodiment of the described embodiments.

[0229] • A TV, set-top box, cellular phone, tablet computer or other electronic device that performs filter parameter adaptation according to any embodiment of the described embodiments, and displays (e.g., using a monitor, screen or other type of display) the resulting image.

[0230] • TVs, set-top boxes, cellular phones, tablet computers, or other electronic devices that select (e.g., using a tuner) a channel to receive signals including encoded images and perform filter parameter adaptation according to any embodiment of the described embodiments.

[0231] • TVs, set-top boxes, cellular phones, tablet computers, or other electronic devices that receive signals including encoded images over the air (e.g., using an antenna) and perform filter parameter adaptation according to any embodiment of the described embodiments.

[0232] Note that the various hardware elements in one or more of the described embodiments are referred to as “modules” that perform (i.e., execute, implement, and so on) the various functions described herein in connection with the respective modules. As used herein, a module includes hardware (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), one or more memory devices) that a person skilled in the art would consider suitable for a given implementation. Each described module may also include executable instructions for performing one or more functions described as being performed by the respective module, and note that those instructions may take the form of hardware (i.e., hardwired) instructions, firmware instructions, software instructions, and / or such instructions, or include hardware (i.e., hardwired) instructions, firmware instructions, software instructions, and / or such instructions, and may be stored in any suitable one or more non-transitory computer-readable media (such as commonly referred to as RAM, ROM, etc.).

[0233] Although the features and elements have been described above in specific combinations, those skilled in the art will understand that each feature or element can be used alone or in any combination with other features and elements. Furthermore, the methods described herein can be implemented as computer programs, software, or firmware incorporated in a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media (such as internal hard disks and removable disks), magneto-optical media, and optical media (such as CD-ROMs and digital universal discs (DVDs)). A processor associated with the software can be used to implement a radio frequency transceiver for use in a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

1. A method comprising: Obtain information including point clouds; The point cloud is passed through a feature extraction process to generate the first feature; Obtain octree features from the octree encoding process; The second feature is generated by fusing the octree feature with the first feature; as well as Features are aggregated based on the second feature.

2. The method according to claim 1, wherein, Octree features are used on at least one of the encoder and decoder processes.

3. The method according to claim 1, further comprising: The first bit stream of features aggregated by the transmission encapsulation; as well as The transmission encapsulates the second bit stream from the block graph of the octree encoding process.

4. The method according to any one of claims 1-3, wherein, Octree features are obtained from the octree encoding process based on the block graph corresponding to the first feature.

5. The method according to any one of claims 1-4, wherein, Octree features are obtained from the octree encoding process, including: Use the block map corresponding to the first feature as input; and The block graph is passed through an octree encoding process to generate octree features.

6. The method according to any one of claims 1-5, wherein, Feature aggregation based on the second feature includes: The second feature is passed through one or more convolutional neural network (CNN) blocks; and The output of one or more CNN blocks is passed through a transformer process to generate aggregated features.

7. The method according to claim 6, wherein, The converter process includes one or more multilayer perceptron (MLP) blocks.

8. The method according to any one of claims 1-5, wherein, Feature aggregation based on the second feature includes: The second feature is passed through one or more convolutional neural network (CNN) blocks; and The output of one or more CNN blocks is passed through the Inception-ResNet (IRN) process to generate aggregated features.

9. The method according to claim 8, wherein, The IRN process includes one or more multilayer perceptron (MLP) blocks.

10. The method according to any one of claims 1-5, wherein, Feature aggregation based on the second feature includes: Perform one or more iterations. The iterative process includes: Enables the transfer of fused features through a multilayer perceptron (MLP) block; The output of the MLP is passed through one or more feature aggregation layers to generate output features; and The output features are fused with the octree features to generate fused features for the next pass through the iterative process. The second feature is used as the fusion feature of the first transmission through the iterative process, and The output feature becomes the aggregated feature that is finally passed through the iterative process.

11. An apparatus comprising: processor; as well as A non-transitory computer-readable medium storing instructions that, when executed by a processor, are operable to cause the apparatus to perform the method of any one of claims 1 to 10.

12. A method comprising: Obtain octree features from the octree encoding process; Obtain input features; New input features are determined by fusing octree features with input features; as well as Aggregate features based on new input features.

13. The method according to claim 12, wherein, Octree features are used on at least one of the encoder and decoder processes.

14. The method according to any one of claims 12-13, wherein, Octree features are obtained from the octree encoding process based on the block graph corresponding to the first feature.

15. The method according to any one of claims 12-13, wherein, Octree features are obtained from the octree encoding process, including: Obtain a bitstream that includes the block diagram previously passed through the encoder's octree encoding process; Using bitstreams as input; and The bitstream is transmitted through an octree encoding process to generate octree features.

16. The method according to any one of claims 12-15, wherein, Feature aggregation based on the second feature includes: The second feature is passed through one or more convolutional neural network (CNN) blocks; and The output of one or more CNN blocks is passed through a transformer process to generate aggregated features.

17. The method according to claim 16, wherein, The converter process includes one or more multilayer perceptron (MLP) blocks.

18. The method according to any one of claims 12-15, wherein, Feature aggregation based on the second feature includes: The second feature is passed through one or more convolutional neural network (CNN) blocks; and The output of one or more CNN blocks is passed through the Inception-ResNet (IRN) process to generate aggregated features.

19. The method according to claim 18, wherein, The IRN process includes one or more multilayer perceptron (MLP) blocks.

20. The method according to any one of claims 12-15, wherein, Feature aggregation based on the second feature includes: Perform one or more iterations. The iterative process includes: Enables the transfer of fused features through a multilayer perceptron (MLP) block; The output of the MLP is passed through one or more feature aggregation layers to generate output features; and The output features are fused with the octree features to generate fused features for the next pass through the iterative process. The second feature is used as the fusion feature of the first transmission through the iterative process, and The output feature becomes the aggregated feature that is finally passed through the iterative process.

21. The method according to any one of claims 12-20, further comprising passing octree features and aggregated features through a point synthesis process to generate an output point cloud.

22. An apparatus comprising: processor; as well as A non-transitory computer-readable medium storing instructions that, when executed by a processor, are operable to cause the apparatus to perform the method of any one of claims 12 to 21.

23. A method comprising: Obtain the first bitstream, including the encoded block diagram; Use bitstreams as input to the octree decoding process; Obtain octree features and the decoded block diagram from the octree decoding process; Obtain input features from the second bitstream; New input features are determined by fusing the decoded block map with the input features; as well as Aggregate features based on new input features; as well as Octree features and aggregated features are passed through a point synthesis process to generate an output point cloud.

24. The method according to claim 23, wherein, Determining new input features further includes: New input features are determined by fusing octree features with the decoded block graph and input features.

25. The method according to claim 23, wherein, The bit stream includes a first bit stream and a second bit stream.

26. An apparatus comprising: processor; as well as A non-transitory computer-readable medium storing instructions that, when executed by a processor, are operable to cause the apparatus to perform the method of any one of claims 23 to 25.

27. A method comprising: Obtain information including point clouds; The point cloud is passed through a feature extraction process to generate the first feature; Obtain octree features from the octree encoding process; The second feature is generated by fusing the octree feature with the first feature; as well as The second feature is passed through the feature aggregation process to generate the third feature.

28. An apparatus comprising: processor; as well as A non-transitory computer-readable medium storing instructions that, when executed by a processor, are operable to cause the apparatus to perform the method of claim 27.

29. A method comprising: Obtain octree features from the octree encoding process; Obtain the first feature as input; The second feature is generated by fusing the octree feature with the first feature; as well as The second feature is passed through the feature aggregation process to generate the third feature.

30. An apparatus comprising: processor; as well as A non-transitory computer-readable medium storing instructions that, when executed by a processor, are operable to cause the apparatus to perform the method of claim 29.

31. An apparatus comprising at least one processor configured to perform the method of any one of claims 1-10, 12-21, 23-25, 27 and 29.

32. An apparatus comprising a computer-readable medium storing instructions for causing one or more processors to perform the method of any one of claims 1-10, 12-21, 23-25, 27 and 29.

33. An apparatus comprising at least one processor and at least one non-transitory computer-readable medium storing instructions for causing at least one processor to perform the method of any one of claims 1-10, 12-21, 23-25, 27 and 29.

34. A signal comprising a bitstream generated according to any one of claims 1-10, 12-21, 23-25, 27 and 29.

Images