Point cloud compression and enhanced feature processing based on feature distribution learning

Abstract

Some embodiments of point cloud geometry methods may include a learning-based point cloud geometry method, wherein the method may include: obtaining a first feature map, wherein the first feature map includes C channels, and wherein the first feature map is generated by one or more previous neural network layers; generating a first set of distribution parameters based on the first feature map using a first set of neural network layers; transforming the first feature map into a second feature map based on the first set of distribution parameters; and outputting the second feature map to a subsequent neural network layer. For some embodiments of the point cloud geometry method, the first feature map may also include n feature vectors corresponding to n distinct points in 3D space.

Description

[0001] Cross-references to related applications This application claims the benefit of U.S. Provisional Patent Application No. 63 / 536,321, filed September 1, 2023, entitled “Point Cloud Compression Enhancement Feature Processing Based on Feature Distribution Learning,” the contents of which are hereby incorporated by reference.

[0002] Merge by reference This application incorporates in its entirety the following applications by reference: International Application No. PCT / US2023 / 027424, entitled "Depth Distribution-Aware Point Feature Extractor for AI-Based Point Cloud Compression," filed July 11, 2023 ("'424 Application"); U.S. Provisional Patent Application Serial No. 63 / 388,600, entitled "Depth Distribution-Aware Point Feature Extractor for AI-Based Point Cloud Compression," filed July 12, 2022 ("'600 Application"); International Application No. PCT / US2022 / 052861, entitled "Scalable Framework for Point Cloud Compression," filed December 14, 2022 ("'861 Application"); U.S. Provisional Patent Application Serial No. 63 / 460,798, entitled "Generative Predictive Encoding / Decoding for Point Cloud Compression," filed April 20, 2023 ("'798 Application"); and U.S. Provisional Patent Application Serial No. Application 63 / 536,340, entitled “Image compression enhancement feature processing based on feature distribution learning”, was filed on September 1, 2023 (“340 application”). Background Technology

[0003] Point cloud data is applicable to numerous business sectors, such as autonomous driving, robotics, AR / VR, civil engineering, computer graphics, and the animation / film industry. 3D LiDAR sensors have been deployed in autonomous vehicles, and affordable LiDAR sensors include the Velodyne Velabit, the Apple iPad Pro 2020, and the Intel RealSense LiDAR camera L515. With advancements in sensing technology, 3D point cloud data is becoming increasingly prevalent in the applications and industries mentioned above. Summary of the Invention

[0004] The embodiments described herein include methods for video encoding and decoding (collectively, "encoding and decoding").

[0005] Example methods according to some embodiments may include: obtaining a first feature map, wherein the first feature map includes C channels, and wherein the first feature map is generated by one or more previous neural network layers; generating a first set of distribution parameters based on the first feature map using a first set of neural network layers; transforming the first feature map into a second feature map based on the first set of distribution parameters; and encoding a bitstream based on the second feature map.

[0006] A first example method according to some embodiments may include: obtaining a first feature map, wherein the first feature map includes C channels, and wherein the first feature map is generated by one or more previous neural network layers; generating a first set of distribution parameters based on the first feature map using a first set of neural network layers; transforming the first feature map into a second feature map based on the first set of distribution parameters; and outputting the second feature map to a subsequent neural network layer.

[0007] In some embodiments of the first example method, the first feature map further includes n feature vectors corresponding to n different points in 3D space.

[0008] In some embodiments of the first example method, each feature vector represents a feature of the corresponding point in 3D space; and the first feature map has a dimension of n×C.

[0009] Some embodiments of the first example method may further include: generating a second set of distribution parameters using a second set of neural network layers; transforming a first feature map into a third feature map based on the second set of distribution parameters; and updating the first feature map into the third feature map.

[0010] For some embodiments of the first example method, generating the first set of distribution parameters may include: updating a given feature map via a neural network layer; and simplifying the updated feature map to obtain the first set of per-channel distribution parameters.

[0011] For some embodiments of the first example method, transforming the first feature map may include: generating a corresponding reshaping vector for each of the C channels in the first feature map, wherein each reshaping vector is generated by reshaping the corresponding channel vector in the first feature map; obtaining a distribution parameter for each of the C channels, the distribution parameter corresponding to one of the corresponding reshaping vectors; expanding each distribution parameter to the corresponding feature channel; and transforming each element in the reshaping vector by the expanded distribution parameter for each of the C channels.

[0012] Some embodiments of the first example method may also include updating the first feature map by normalizing each feature element of the first feature map.

[0013] For some embodiments of the first example method, normalizing each feature element of the first feature map may include: generating a third set of distribution parameters using a third set of neural network layers; centering the first feature map to a fourth feature map based on the third set of distribution parameters; and updating the first feature map to the fourth feature map.

[0014] Some embodiments of the first example method may further include: generating a fourth set of distribution parameters from the fourth feature map; and normalizing the fourth feature map into a fifth feature map based on the fourth set of distribution parameters.

[0015] Some embodiments of the first example method may also include: generating a fifth set of distribution parameters using a fifth set of neural network layers; and normalizing a fourth feature map to a sixth feature map based on the fifth set of distribution parameters.

[0016] Some embodiments of the first example method may further include: downsampling a second feature map to generate a first feature vector, wherein the downsampling of the second feature map uses a pooling function.

[0017] For some embodiments of the first example method, the pooling function is selected from the group including average pooling and max pooling.

[0018] Some embodiments of the first example method may further include: generating an extended feature map by extending the first feature vector toward the channel dimension; concatenating the second feature map with the extended feature map to generate a concatenated feature map; and passing the concatenated feature map through a filtering neural network layer to generate a filtered feature map.

[0019] Some embodiments of the first example method may also include using an additional neural network to aggregate a second feature map.

[0020] For some embodiments of the first example method, the additional neural network includes at least one of a sparse convolutional neural network (CNN) and a multilayer perceptron (MLP).

[0021] Some embodiments of the first example method may also include using a residual network to aggregate a second feature map.

[0022] In some embodiments of the first example method, the residual network is a ResNet block.

[0023] Some embodiments of the first example method may also include using transformer blocks to aggregate the second feature map.

[0024] For some embodiments of the first example method, the transformer block is selected from the group including point transformers and voxel transformers.

[0025] Some embodiments of the first example method may further include: generating a seventh feature map by aggregating the second feature map using a neural network, transforming the first feature map into the second feature map in parallel; and concatenating the seventh feature map to the second feature map.

[0026] Some embodiments of the first example method may further include: repeating the learning-based point cloud geometry process once or multiple times to generate an eighth feature map; and adding the eighth feature map to the first feature map to generate a ninth feature map.

[0027] Example apparatus according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, cause the apparatus to: obtain a first feature map, wherein the first feature map includes C channels, and wherein the first feature map is generated by one or more prior neural network layers; generate a first set of distribution parameters based on the first feature map using a first set of neural network layers; transform the first feature map into a second feature map based on the first set of distribution parameters; and encode a bitstream based on the second feature map.

[0028] A first example apparatus according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, cause the apparatus to: obtain a first feature map, wherein the first feature map includes C channels, and wherein the first feature map is generated by one or more previous neural network layers; generate a first set of distribution parameters based on the first feature map using a first set of neural network layers; transform the first feature map into a second feature map based on the first set of distribution parameters; and output the second feature map to a subsequent neural network layer.

[0029] In some embodiments of the first example device, the first feature map further includes n feature vectors corresponding to n different points in 3D space.

[0030] In some embodiments of the first example device, each feature vector represents a feature of a corresponding point in 3D space; and the first feature map has a dimension of n×C.

[0031] Example methods according to some embodiments may include: obtaining a first feature map, wherein the first feature map includes C channels, and wherein the first feature map is generated by one or more previous neural network layers; generating a first set of distribution parameters based on the first feature map using a first set of neural network layers; expanding the first set of distribution parameters to match the first feature map in dimensions; subtracting the expanded first set of distribution parameters from the first feature map to generate a centered feature map; generating a second set of distribution parameters based on the first feature map using a second set of neural network layers; expanding the second set of distribution parameters to match the centered feature map in dimensions; dividing the centered feature map by the expanded second set of distribution parameters to generate a normalized feature map; and encoding a bitstream based on the normalized feature map.

[0032] A second example method according to some embodiments may include: obtaining a first feature map, wherein the first feature map includes C channels, and wherein the first feature map is generated by one or more previous neural network layers; generating a first set of distribution parameters based on the first feature map using a first set of neural network layers; expanding the first set of distribution parameters to match the first feature map in dimensions; subtracting the expanded first set of distribution parameters from the first feature map to generate a centered feature map; generating a second set of distribution parameters based on the first feature map using a second set of neural network layers; expanding the second set of distribution parameters to match the centered feature map in dimensions; dividing the centered feature map by the expanded second set of distribution parameters to generate a normalized feature map; and outputting the normalized feature map to a subsequent neural network layer.

[0033] In some embodiments of the second example method, the first feature map further includes n feature vectors corresponding to n different points in 3D space.

[0034] In some embodiments of the second example method, each feature vector represents a feature of the corresponding point in 3D space; and the first feature map has a dimension of n×C.

[0035] For some embodiments of the second example method, expanding the first set of distribution parameters to match the first feature map in a dimension includes: copying the first distribution parameter vector once or multiple times to match the corresponding dimension of the first feature map in a dimension, the first distribution parameter vector including the first set of distribution parameters; expanding the second set of distribution parameters to match the centered feature map in a dimension includes: copying the second distribution parameter vector once or multiple times to match the corresponding dimension of the centered feature map in a dimension, and the second distribution parameter vector including the second set of distribution parameters.

[0036] Example apparatuses according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, cause the apparatus to: obtain a first feature map, wherein the first feature map includes C channels, and wherein the first feature map is generated by one or more prior neural network layers; generate a first set of distribution parameters based on the first feature map using a first set of neural network layers; expand the first set of distribution parameters to match the first feature map in dimensions; subtract the expanded first set of distribution parameters from the first feature map to generate a centered feature map; generate a second set of distribution parameters based on the first feature map using a second set of neural network layers; expand the second set of distribution parameters to match the centered feature map in dimensions; divide the centered feature map by the expanded second set of distribution parameters to generate a normalized feature map; and encode a bitstream based on the normalized feature map.

[0037] A second example apparatus according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, cause the apparatus to: obtain a first feature map, wherein the first feature map includes C channels, and wherein the first feature map is generated by one or more previous neural network layers; generate a first set of distribution parameters based on the first feature map using a first set of neural network layers; expand the first set of distribution parameters to match the first feature map in dimensions; subtract the expanded first set of distribution parameters from the first feature map to generate a centered feature map; generate a second set of distribution parameters based on the first feature map using a second set of neural network layers; expand the second set of distribution parameters to match the centered feature map in dimensions; divide the centered feature map by the expanded second set of distribution parameters to generate a normalized feature map; and output the normalized feature map to a subsequent neural network layer.

[0038] Example methods according to some embodiments may include: obtaining a normalized feature map, wherein the normalized feature map includes C channels corresponding to an original feature map, and wherein the original feature map and the normalized feature map are generated by one or more previous neural network layers respectively; generating a first set of distribution parameters based on the normalized feature map using a first set of neural network layers; expanding the first set of distribution parameters to match the normalized feature map in dimensions; scaling the normalized feature map by the expanded first set of distribution parameters to generate a scaled feature map; generating a second set of distribution parameters based on the normalized feature map using a second set of neural network layers; expanding the second set of distribution parameters to match the scaled feature map in dimensions; adding the scaled feature map to the expanded second set of distribution parameters to generate a transformed feature map; and encoding a bitstream based on the transformed feature map.

[0039] A third example method according to some embodiments may include: obtaining a normalized feature map, wherein the normalized feature map includes C channels corresponding to an original feature map, and wherein the original feature map and the normalized feature map are generated by one or more previous neural network layers respectively; generating a first set of distribution parameters based on the normalized feature map using a first set of neural network layers; expanding the first set of distribution parameters to match the normalized feature map in dimensions; scaling the normalized feature map by the expanded first set of distribution parameters to generate a scaled feature map; generating a second set of distribution parameters based on the normalized feature map using a second set of neural network layers; expanding the second set of distribution parameters to match the scaled feature map in dimensions; adding the scaled feature map to the expanded second set of distribution parameters to generate a transformed feature map; and outputting the transformed feature map to a subsequent neural network layer.

[0040] In some embodiments of the third example method, the first feature map also includes n feature vectors corresponding to n different points in 3D space.

[0041] In some embodiments of the third example method, each feature vector represents a feature of the corresponding point in 3D space; and the first feature map has a dimension of n×C.

[0042] In some embodiments of the third example method, the input to the second set of neural network layers is a scaled feature map.

[0043] In some embodiments of the third example method, the input to the second set of neural network layers is a normalized feature map.

[0044] Some embodiments of the third example method may further include: downsampling the transformed feature map to generate a first feature vector, wherein the downsampling of the transformed feature map uses a pooling function.

[0045] For some embodiments of the third example method, the pooling function is selected from the group including average pooling and max pooling.

[0046] Some embodiments of the third example method may further include: generating an extended feature map by extending the first feature vector toward the channel dimension; concatenating the first feature map with the extended feature map to generate a concatenated feature map; and passing the concatenated feature map through a filtering neural network layer to generate a filtered feature map.

[0047] Some embodiments of the third example method may also include: using an additional neural network to aggregate the first feature map.

[0048] For some embodiments of the third example method, the additional neural network includes at least one of a sparse convolutional neural network (CNN) and a multilayer perceptron (MLP).

[0049] Some embodiments of the third example method may also include using a residual network to aggregate the first feature map.

[0050] In some embodiments of the third example method, the residual network is a ResNet block.

[0051] Some embodiments of the third example method may further include: repeating the learning-based point cloud geometry method once or multiple times to generate an enhanced feature map; and adding the enhanced feature map to the first feature map to generate an enhanced output feature map.

[0052] Some embodiments of the third example method may also include repeating the learning-based point cloud geometry method once or multiple times within the point cloud encoder.

[0053] Some embodiments of the third example method may also include repeating the learning-based point cloud geometry method once or multiple times within the point cloud decoder.

[0054] In some embodiments of the third example method, the point cloud encoder processes voxels.

[0055] For some embodiments of the third example method, the point cloud encoder performs point and feature analysis.

[0056] Example apparatus according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, cause the apparatus to: obtain a normalized feature map, wherein the normalized feature map includes C channels corresponding to an original feature map, and wherein the original feature map and the normalized feature map are generated by one or more prior neural network layers, respectively; generate a first set of distribution parameters based on the normalized feature map using a first set of neural network layers; expand the first set of distribution parameters to match the normalized feature map in dimensions; scale the normalized feature map by the expanded first set of distribution parameters to generate a scaled feature map; generate a second set of distribution parameters based on the normalized feature map using a second set of neural network layers; expand the second set of distribution parameters to match the scaled feature map in dimensions; add the scaled feature map to the expanded second set of distribution parameters to generate a transformed feature map; and encode a bitstream based on the transformed feature map.

[0057] A third example apparatus according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, cause the apparatus to: obtain a normalized feature map, wherein the normalized feature map includes C channels corresponding to an original feature map, and wherein the original feature map and the normalized feature map are generated by one or more previous neural network layers, respectively; generate a first set of distribution parameters based on the normalized feature map using a first set of neural network layers; expand the first set of distribution parameters to match the normalized feature map in dimensions; scale the normalized feature map using the expanded first set of distribution parameters to generate a scaled feature map; generate a second set of distribution parameters based on the normalized feature map using a second set of neural network layers; expand the second set of distribution parameters to match the scaled feature map in dimensions; add the scaled feature map to the expanded second set of distribution parameters to generate a transformed feature map; and output the transformed feature map to a subsequent neural network layer.

[0058] In additional embodiments, encoder and decoder devices are provided to perform the methods described herein. The encoder or decoder device may include a processor configured to perform the methods described herein. The device may include a computer-readable medium (e.g., a non-transitory medium) storing instructions for performing the methods described herein. In some embodiments, the computer-readable medium (e.g., a non-transitory medium) stores video encoded using any of the methods described herein.

[0059] One or more embodiments of this disclosure also provide a computer-readable storage medium storing instructions for performing bidirectional optical flow, encoding or decoding video data according to any of the methods described above. Embodiments of this disclosure also provide a computer-readable storage medium storing a bitstream generated according to the methods described above. Embodiments of this disclosure also provide a method and apparatus for transmitting a bitstream generated according to the methods described above. Embodiments of this disclosure also provide a computer program product including instructions for performing any of the methods described above. Attached Figure Description

[0060] Figure 1A This is a system diagram illustrating an example communication system according to some embodiments.

[0061] Figure 1B The figure illustrates that, according to some embodiments, it is possible to Figure 1A The diagram shows a system diagram of an example wireless transmit / receive unit (WTRU) used in the communication system.

[0062] Figure 1C This is a system diagram illustrating an example group of interfaces of a system according to some embodiments.

[0063] Figure 2A This is a functional block diagram of a block-based video encoder according to some embodiments, such as an encoder for general video codec (VVC).

[0064] Figure 2B This is a functional block diagram of a block-based video decoder according to some embodiments, such as a decoder for VVC.

[0065] Figure 3A This is a schematic side view of an example waveguide display that can be used with extended reality (XR) applications according to some embodiments.

[0066] Figure 3B This is a schematic side view illustrating an example alternative display type that can be used with extended reality applications according to some embodiments.

[0067] Figure 3C This is a schematic side view illustrating an example alternative display type that can be used with extended reality applications according to some embodiments.

[0068] Figure 4 These are schematic illustrations of example point-based feature vectors and feature maps based on some embodiments.

[0069] Figure 5 This is a schematic illustration of example distribution parameters corresponding to the reshaping vector, based on some embodiments.

[0070] Figure 6This is a functional block diagram of an example neural network layer for learning feature map distribution parameters, based on some embodiments.

[0071] Figure 7 This is a schematic illustration of an example of three shared MLP layers according to some embodiments.

[0072] Figure 8 This is a functional block diagram illustrating the first example feature map normalization process according to some embodiments.

[0073] Figure 9 This is a schematic illustration showing an example of extending to the feature map dimension according to some embodiments.

[0074] Figure 10 This is a functional block diagram illustrating the second example feature map normalization process according to some embodiments.

[0075] Figure 11 This is a functional block diagram illustrating the first example feature map transformation process according to some embodiments.

[0076] Figure 12 This is a functional block diagram illustrating the second example feature map transformation process according to some embodiments.

[0077] Figure 13 This is a functional block diagram illustrating an example of feature map downsampling according to some embodiments.

[0078] Figure 14 This is a functional block diagram illustrating an example of feature map filtering based on some embodiments.

[0079] Figure 15 This is a functional block diagram illustrating an example parallel architecture for enhancing distributed learning networks, based on some embodiments.

[0080] Figure 16 This is a functional block diagram illustrating an example serial architecture for enhancing distributed learning networks, based on some embodiments.

[0081] Figure 17 This is a functional block diagram illustrating an example deep augmented distributed learning network based on some embodiments.

[0082] Figure 18 This is a functional block diagram illustrating an example deep augmented distributed learning network in a voxel-based point cloud compression framework, based on some embodiments.

[0083] Figure 19 This is a functional block diagram illustrating an example deep augmented distributed learning network in a point cloud compression framework based on some embodiments.

[0084] Figure 20 This is a flowchart illustrating an example of a learning-based point cloud geometry process, based on some embodiments.

[0085] Figure 21 This is a flowchart illustrating an example of a learning-based point cloud geometry process, based on some embodiments.

[0086] Figure 22 This is a flowchart illustrating an example of a learning-based point cloud geometry process, based on some embodiments.

[0087] Entities, connections, arrangements, etc., depicted in the various figures and described in the associated descriptions are presented in an exemplary rather than restrictive manner. Therefore, any statement or other indication regarding what a particular figure “depicts,” what a particular element or entity in a particular figure “is” or “has,” and any and all such statements—which, when read in isolation and out of context, might be interpreted as absolute and therefore restrictive—can only be correctly read as being implicitly preceded by a clause such as “In at least one embodiment, ...". For the sake of brevity and clarity, this implicit preceding clause will not be repeated in the detailed description. Detailed Implementation

[0088] Figure 1A This diagram illustrates an example communication system 100 in which one or more of the disclosed embodiments may be implemented. The communication system 100 may be a multiple access system providing content such as voice, data, video, messaging, and broadcasting to multiple wireless users. The communication system 100 enables multiple wireless users to access such content through the sharing of system resources (including wireless bandwidth). For example, the communication system 100 may employ one or more channel access methods, such as Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal FDMA (OFDMA), Single Carrier FDMA (SC-FDMA), Zero-Tail Unique Word DFT Extended OFDM (ZT UW DTS-s OFDM), Unique Word OFDM (UW-OFDM), Resource Block Filtered OFDM, Filter Bank Multicarrier (FBMC), etc.

[0089] like Figure 1AAs shown, the communication system 100 may include wireless transmit / receive units (WTRUs) 102a, 102b, 102c, 102d, RAN 104 / 113, CN 106, public switched telephone network (PSTN) 108, Internet 110, and other networks 112. However, it should be understood that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and / or network elements. Each of the WTRUs 102a, 102b, 102c, 102d can be any type of device configured to operate and / or communicate in a wireless environment. For example, any of the WTRUs 102a, 102b, 102c, and 102d, which can be referred to as a "station" and / or "STA," can be configured to transmit and / or receive wireless signals and can include user equipment (UE), mobile stations, fixed or mobile subscriber units, subscription-based units, pagers, cellular phones, personal digital assistants (PDAs), smartphones, laptops, netbooks, personal computers, wireless sensors, hotspots or Mi-Fi devices, Internet of Things (IoT) devices, watches or other wearable devices, head-mounted displays (HMDs), vehicles, drones, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices operating in industrial and / or automated processing chain scenarios), consumer electronics devices, devices operating on commercial and / or industrial wireless networks, etc. Any of the WTRUs 102a, 102b, 102c, and 102d can be interchangeably referred to as a UE.

[0090] The communication system 100 may also include base station 114a and / or base station 114b. Each of base stations 114a and 114b can be any type of device configured to wirelessly interface with at least one of WTRUs 102a, 102b, 102c, and 102d to facilitate access to one or more communication networks, such as CN 106, Internet 110, and / or Network 112. For example, base stations 114a and 114b can be base transceiver stations (BTS), Node-B, eNode B, home Node B, home eNode B, gNB, NR Node B, site controller, access point (AP), wireless router, etc. Although each of base stations 114a and 114b is depicted as a single element, it should be understood that base stations 114a and 114b can include any number of interconnected base stations and / or network elements.

[0091] Base station 114a may be part of RAN 104 / 113, which may also include other base stations and / or network elements (not shown), such as base station controllers (BSCs), radio network controllers (RNCs), relay nodes, etc. Base station 114a and / or base station 114b may be configured to transmit and / or receive radio signals on one or more carrier frequencies (which may be referred to as cells (not shown)). These frequencies may be in licensed spectrum, unlicensed spectrum, or a combination of licensed and unlicensed spectrum. A cell may provide coverage of a specific geographic area, which may be relatively fixed or may vary over time. A cell may be further divided into cell sectors. For example, the cell associated with base station 114a may be divided into three sectors. Thus, in one embodiment, base station 114a may include three transceivers, i.e., one transceiver per sector of the cell. In one embodiment, base station 114a may employ multiple-input multiple-output (MIMO) technology and may utilize multiple transceivers for each sector of the cell. For example, beamforming may be used to transmit and / or receive signals in a desired spatial direction.

[0092] Base stations 114a and 114b can communicate with one or more of WTRUs 102a, 102b, 102c, and 102d via air interface 116, which can be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). Any suitable radio access technology (RAT) can be used to establish air interface 116.

[0093] More specifically, as noted above, communication system 100 can be a multiple access system and can employ one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, etc. For example, base stations 114a and WTRUs 102a, 102b, and 102c in RAN 104 / 113 can implement radio technologies such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which can use Wideband CDMA (WCDMA) to establish the air interface 116. WCDMA may include communication protocols such as High-Speed ​​Packet Access (HSPA) and / or Evolved HSPA (HSPA+). HSPA may include High-Speed ​​Downlink (DL) Packet Access (HSDPA) and / or High-Speed ​​UL Packet Access (HSUPA).

[0094] In one embodiment, base station 114a and WTRUs 102a, 102b, 102c can implement radio technologies such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which can use Long Term Evolution (LTE) and / or Advanced LTE (LTE-A) and / or Advanced LTE Pro (LTE-A Pro) to establish air interface 116.

[0095] In one embodiment, base station 114a and WTRUs 102a, 102b, 102c can implement radio technologies such as NR radio access, which can use New Radio (NR) to establish air interface 116.

[0096] In one embodiment, base station 114a and WTRUs 102a, 102b, and 102c can implement multiple radio access technologies. For example, base station 114a and WTRUs 102a, 102b, and 102c can, for instance, use a dual connectivity (DC) principle to implement both LTE and NR radio access. Therefore, the air interface utilized by WTRUs 102a, 102b, and 102c can be characterized by multiple types of radio access technologies and / or the transmissions sent to / from multiple types of base stations (e.g., eNBs and gNBs).

[0097] In other embodiments, base station 114a and WTRUs 102a, 102b, 102c can implement radio technologies such as IEEE 802.11 (i.e., Wi-Fi), IEEE 802.16 (i.e., Global Microwave Access Interoperability (WiMAX)), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Provisional Standard 2000 (IS-2000), Provisional Standard 95 (IS-95), Provisional Standard 856 (IS-856), Global System for Mobile Communications (GSM), Enhanced Data Rate GSM Evolution (EDGE), GSM EDGE (GERAN), etc.

[0098] Figure 1ABase station 114b can be, for example, a wireless router, a home Node B, a home eNode-B, or an access point, and can utilize any suitable RAT to facilitate wireless connectivity in local areas such as commercial locations, homes, vehicles, campuses, industrial facilities, air corridors (e.g., for drone use), roads, etc. In one embodiment, base station 114b and WTRUs 102c, 102d can implement radio technologies such as IEEE 802.11 to establish a wireless local area network (WLAN). In one embodiment, base station 114b and WTRUs 102c, 102d can implement radio technologies such as IEEE 802.15 to establish a wireless personal area network (WPAN). In yet another embodiment, base station 114b and WTRUs 102c, 102d can utilize cellular-based RATs (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR, etc.) to establish picocells or femtocells. Figure 1A As shown, base station 114b can have a direct connection to Internet 110. Therefore, base station 114b does not need to access Internet 110 via CN 106.

[0099] RAN 104 / 113 can communicate with CN 106, which can be any type of network configured to provide voice, data, application, and / or Voice over Internet Protocol (VoIP) services to one or more of WTRUs 102a, 102b, 102c, and 102d. Data can have different Quality of Service (QoS) requirements, such as different throughput requirements, latency requirements, fault tolerance requirements, reliability requirements, data throughput requirements, mobility requirements, etc. CN 106 can provide call control, billing services, location-based services, prepaid calling, internet connectivity, video distribution, etc., and / or perform advanced security functions such as user authentication. Although not explicitly stated... Figure 1A As shown, but it should be understood that RAN 104 / 113 and / or CN 106 can communicate directly or indirectly with other RANs that use the same RAT as RAN 104 / 113 or a different RAT. For example, in addition to connecting to RAN 104 / 113, which may be using NR radio technology, CN 106 can also communicate with another RAN (not shown) using GSM, UMTS, CDMA2000, WiMAX, E-UTRA, or WiFi radio technology.

[0100] CN 106 can also act as a gateway for WTRUs 102a, 102b, 102c, and 102d to access PSTN 108, the Internet 110, and / or other networks 112. PSTN 108 may include a circuit-switched telephone network providing Common Old-Style Telephone Service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices using common communication protocols such as Transmission Control Protocol (TCP), User Datagram Protocol (UDP), and / or Internet Protocol (IP) from the TCP / IP Internet Protocol suite. Network 112 may include wired and / or wireless communication networks owned and / or operated by other service providers. For example, network 112 may include another CN connected to one or more RANs, which may use the same RAT as RAN 104 / 113 or a different RAT.

[0101] Some or all of the WTRUs 102a, 102b, 102c, and 102d in communication system 100 may include multi-mode capabilities (e.g., WTRUs 102a, 102b, 102c, and 102d may include multiple transceivers for communicating with different wireless networks via different wireless links). For example, Figure 1A The WTRU 102c shown can be configured to communicate with base station 114a, which can employ cellular-based radio technology, and with base station 114b, which can employ IEEE 802 radio technology.

[0102] Figure 1B This is a system diagram illustrating example WTRU 102. (See diagram below.) Figure 1B As shown, among other things, WTRU 102 may include a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a keypad 126, a display / touchpad 128, non-removable memory 130, removable memory 132, a power supply 134, a global positioning system (GPS) chipset 136, and / or other peripheral devices 138. It should be understood that, while remaining consistent with the embodiments, WTRU 102 may include any sub-combination of the foregoing elements.

[0103] Processor 118 may be a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc. Processor 118 may perform signal encoding / decoding, data processing, power control, input / output processing, and / or any other functions that enable WTRU 102 to operate in a wireless environment. Processor 118 may be coupled to transceiver 120, which may be coupled to transmitting / receiving element 122. Although Figure 1B The processor 118 and transceiver 120 are depicted as separate components, but it should be understood that the processor 118 and transceiver 120 may be integrated together into an electronic package or chip.

[0104] Transmitting / receiving element 122 can be configured to transmit signals to or receive signals from a base station (e.g., base station 114a) via air interface 116. For example, in one embodiment, transmitting / receiving element 122 can be an antenna configured to transmit and / or receive RF signals. In one embodiment, for example, transmitting / receiving element 122 can be a transmitter / detector configured to transmit and / or receive IR, UV, or visible light signals. In yet another embodiment, transmitting / receiving element 122 can be configured to transmit and / or receive both RF signals and optical signals. It should be understood that transmitting / receiving element 122 can be configured to transmit and / or receive any combination of wireless signals.

[0105] Although the transmitting / receiving element 122 is in Figure 1B While depicted as a single element, WTRU 102 may include any number of transmit / receive elements 122. More specifically, WTRU 102 may employ MIMO technology. Thus, in one embodiment, WTRU 102 may include two or more transmit / receive elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals via air interface 116.

[0106] Transceiver 120 can be configured to modulate signals transmitted by transmitting / receiving element 122 and demodulate signals received by transmitting / receiving element 122. As noted above, WTRU 102 can have multi-mode capability. Thus, for example, transceiver 120 may include multiple transceivers to enable WTRU 102 to communicate via multiple RATs (e.g., NR and IEEE 802.11).

[0107] The processor 118 of WTRU 102 can be coupled to a speaker / microphone 124, a keypad 126, and / or a display / touchpad 128 (e.g., a liquid crystal display (LCD) unit or an organic light-emitting diode (OLED) display unit) and can receive user input data therefrom. The processor 118 can also output user data to the speaker / microphone 124, keypad 126, and / or display / touchpad 128. Furthermore, the processor 118 can access information from any type of suitable memory (such as non-removable memory 130 and / or removable memory 132) and store the data in that memory. Non-removable memory 130 may include random access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. Removable memory 132 may include a subscriber identity module (SIM) card, memory stick, secure digital storage (SD) card, etc. In other embodiments, the processor 118 can access information from memory not physically located on WTRU 102 (such as on a server or home computer (not shown)) and store the data in that memory.

[0108] The processor 118 can receive power from the power supply 134 and can be configured to distribute and / or control power to other components in the WTRU 102. The power supply 134 can be any suitable device for powering the WTRU 102. For example, the power supply 134 may include one or more dry cell batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, etc.

[0109] The processor 118 may also be coupled to a GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) regarding the current location of the WTRU 102. In addition to, or alternatively to, the information from the GPS chipset 136, the WTRU 102 may receive location information from base stations (e.g., base stations 114a, 114b) via air interface 116 and / or determine its location based on the timing of signals received from two or more nearby base stations. It should be understood that, while remaining consistent with the embodiments, the WTRU 102 may acquire location information using any suitable location determination method.

[0110] The processor 118 may be further coupled to other peripheral devices 138, which may include one or more software and / or hardware modules providing additional features, functions, and / or wired or wireless connectivity. For example, peripheral devices 138 may include accelerometers, electronic compasses, satellite transceivers, digital cameras (for photos and / or video), Universal Serial Bus (USB) ports, vibration devices, television transceivers, hands-free headsets, Bluetooth® modules, FM radio units, digital music players, media players, video game player modules, internet browsers, virtual reality and / or augmented reality (VR / AR) devices, activity trackers, etc. Peripheral devices 138 may include one or more sensors. Sensors may be one or more of the following: gyroscopes, accelerometers, Hall effect sensors, magnetometers, orientation sensors, proximity sensors, temperature sensors, time sensors; geolocation sensors, altimeters, light sensors, touch sensors, magnetometers, barometers, gesture sensors, biometric sensors, and / or humidity sensors.

[0111] WTRU 102 may include a full-duplex radio transceiver, for which the transmission and reception of some or all signals (e.g., associated with specific subframes for both UL (e.g., for transmission) and downlink (e.g., for reception)) may be concurrent and / or simultaneous. The full-duplex radio transceiver may include an interference management unit for reducing and / or substantially eliminating self-interference via hardware (e.g., chokes) or via signal processing (e.g., a separate processor (not shown) or via processor 118). In one embodiment, WTRU 102 may include a half-duplex radio transceiver, for which the transmission and reception of some or all signals (e.g., associated with specific subframes for UL (e.g., for transmission) or downlink (e.g., for reception)) may be concurrent and / or simultaneous.

[0112] Despite WTRU in Figure 1A-1B While described as a wireless terminal, it is conceivable that in some representative embodiments, such a terminal may use (e.g., temporarily or permanently) a wired communication interface with a communication network.

[0113] In a representative embodiment, another network 112 may be a WLAN.

[0114] Given Figure 1A-1B as well as Figure 1A-1BAs described herein, one or more, or all, of the functions described herein may be performed by one or more emulation devices (not shown). An emulation device may be one or more devices configured to mimic one or more, or all, of the functions described herein. For example, an emulation device may be used to test other devices and / or simulate network and / or WTRU functions.

[0115] Simulation devices can be designed to perform one or more tests on other devices in laboratory and / or carrier network environments. For example, one or more simulation devices can perform one or more or all functions while being fully or partially implemented and / or deployed as part of a wired and / or wireless communication network to test other devices within the communication network. One or more simulation devices can perform one or more or all functions while being temporarily implemented / deployed as part of a wired and / or wireless communication network. Simulation devices can be directly coupled to another device for testing purposes and / or can use over-the-air wireless communication to perform tests.

[0116] One or more emulation devices can perform one or more (including all) functions without being implemented / deployed as part of a wired and / or wireless communication network. For example, emulation devices can be utilized in test scenarios within test laboratories and / or non-deployed (e.g., testing) wired and / or wireless communication networks to perform testing of one or more components. One or more emulation devices can be test rigs. Direct RF coupling and / or wireless communication via RF circuitry (which may include one or more antennas) can be used by the emulation devices to transmit and / or receive data.

[0117] Figure 1C This is a system diagram of an example group of interfaces of a system according to some embodiments. Extended reality display devices and their control electronics can use, for example... Figure 3BSystem 150 may be implemented as a system such as an example system of 3C. System 150 can be implemented as a device including various different components described below and configured to perform one or more of the aspects described in this document. Examples of such devices include, but are not limited to, various different electronic devices such as personal computers, laptops, smartphones, tablets, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. The elements of system 150 may be implemented individually or in combination in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 150 are distributed across multiple ICs and / or discrete components. In various different embodiments, system 150 is communicatively coupled to one or more other systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports. In various different embodiments, system 1000 is configured to implement one or more of the aspects described in this document.

[0118] System 150 includes at least one processor 152 configured to execute instructions loaded therein to implement various aspects, such as those described in this document. Processor 152 may include embedded memory, input / output interfaces, and various other circuit systems known in the art. System 150 includes at least one memory 154 (e.g., a volatile memory device and / or a non-volatile memory device). System 150 may include a storage device 158, which may include non-volatile memory and / or volatile memory, including but not limited to electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, disk drives, and / or optical disk drives. As a non-limiting example, storage device 158 may include internal storage devices, attached storage devices (including removable and non-removable storage devices), and / or network-accessible storage devices.

[0119] System 150 includes an encoder / decoder module 156 configured to, for example, process data to provide encoded or decoded video, and the encoder / decoder module 156 may include its own processor and memory. The encoder / decoder module 156 represents a module that can be included in a device to perform encoding and / or decoding functions. It is well known that a device can include one or both encoding and decoding modules. Furthermore, as those skilled in the art will appreciate, the encoder / decoder module 156 can be implemented as a separate element of system 150, or it can be incorporated into processor 152 as a combination of hardware and software.

[0120] Program code to be loaded onto processor 152 or encoder / decoder 156 to execute the various aspects described in this document may be stored in storage device 158 and subsequently loaded onto memory 154 for execution by processor 152. According to various embodiments, one or more of processor 152, memory 154, storage device 158, and encoder / decoder module 156 may store one or more various items during the execution of the processes described in this document. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from processing equations, formulas, operations, and operational logic.

[0121] In some embodiments, memory within processor 152 and / or encoder / decoder module 156 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, external memory (e.g., processor 152 or encoder / decoder module 152) is used for one or more of these functions. External memory may be memory 154 and / or storage device 158, such as volatile memory and / or non-volatile flash memory. In several embodiments, external non-volatile flash memory is used to store, for example, the operating system of a television. In at least one embodiment, fast external volatile memory such as RAM is used as working memory for video encoding and decoding operations, such as for MPEG-2 (MPEG stands for Moving Picture Experts Group; MPEG-2 is also known as ISO / IEC 13818, 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC stands for High Efficiency Video Codec, also known as H.265 and MPEG-H Part 2), or VVC (Various Video Codec, a new standard developed by the Joint Video Experts Group JVET).

[0122] Inputs to the components of system 150 can be provided by a variety of different input devices as shown in box 172. Such input devices include, but are not limited to: (i) a radio frequency (RF) section that receives, for example, RF signals transmitted over the air by a broadcaster; (ii) component (COMP) input terminals (or a set of COMP input terminals); (iii) universal serial bus (USB) input terminals; and / or (iv) high-definition multimedia interface (HDMI) input terminals. Figure 1C Other examples not shown include composite video.

[0123] In various embodiments, the input device of block 172 has associated corresponding input processing elements known in the art. For example, the RF section may be associated with elements suitable for: (i) selecting a desired frequency (also known as selecting a signal, or limiting a signal band to a certain band); (ii) down-converting the selected signal; (iii) further limiting the band to a narrower band to select, for example, a signal band that may be referred to as a channel in some embodiments; (iv) demodulating the down-converted and band-limited signal; (v) performing error correction; and (vi) demultiplexing to select a desired data packet stream. The RF section in various embodiments includes one or more elements performing these functions, such as frequency selectors, signal selectors, band limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF section may include tuners that perform various of these functions, including, for example, down-converting received signals to a lower frequency (e.g., intermediate frequency or near-baseband frequency) or baseband. In one set-top box embodiment, the RF section and its associated input processing elements receive RF signals transmitted via a wired (e.g., cable) medium and perform frequency selection by filtering, down-converting, and re-filtering to a desired frequency band. Various embodiments rearrange the order of the above (and other) components, remove some of them, and / or add other components that perform similar or different functions. Adding components may include inserting components between existing components, such as, for example, inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0124] Furthermore, the USB and / or HDMI terminals may include corresponding interface processors for connecting system 150 to other electronic devices across USB and / or HDMI connections. It should be understood that various aspects of input processing, such as Reed-Solomon error correction, may be implemented as needed, for example, within a separate input processing IC or within processor 152. Similarly, aspects of USB or HDMI interface processing may be implemented as needed within a separate interface IC or within processor 152. The demodulated, error-corrected, and demultiplexed streams are provided to various processing elements, including, for example, an encoder / decoder 156 operating in conjunction with memory and storage elements, and processor 152, to process the data streams as needed for presentation on the output device.

[0125] Various components of system 150 can be housed within an integrated housing. Within the integrated housing, various components can be interconnected and transmit data therebetween using appropriate connection arrangements 174, such as internal buses known in the art, including inter-IC (I2C) buses, wiring, and printed circuit boards.

[0126] System 150 includes a communication interface 160 that allows communication with other devices via a communication channel 162. The communication interface 160 may include, but is not limited to, a transceiver configured to transmit and receive data via the communication channel 162. The communication interface 160 may include, but is not limited to, a modem or network interface card (NIC), and the communication channel 162 may be implemented, for example, within a wired and / or wireless medium.

[0127] In various embodiments, wireless networks such as Wi-Fi networks, such as IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers), are used to stream or otherwise provide data to system 150. In these embodiments, Wi-Fi signals are received via a communication channel 162 and a communication interface 160 adapted for Wi-Fi communication. The communication channel 162 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other over-the-top communications. Other embodiments use a set-top box to provide streaming data to system 150, delivering data via an HDMI connection in input box 172. Still other embodiments use an RF connection in input box 172 to provide streaming data to system 150. As shown above, various embodiments provide data in a non-streaming manner. Furthermore, various embodiments use wireless networks other than Wi-Fi, such as cellular networks or Bluetooth® networks.

[0128] System 150 can provide output signals to a variety of different output devices, including a display 176, a speaker 178, and other peripheral devices 180. The display 176 in various embodiments includes one or more of, for example, a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. The display 176 can be used in a television, tablet, laptop, mobile phone, or other device. The display 176 can also be integrated with other components (e.g., as in a smartphone) or stand alone (e.g., as an external monitor for a laptop). In various example embodiments, the other peripheral devices 180 include one or more of a standalone digital video disc (or digital versatile disc) (DVR, for both terms), a disc player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 180 that provide functionality based on the output of system 150. For example, a disc player performs the function of playing the output of system 150.

[0129] In various embodiments, signaling such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols that allow device-to-device control with or without user intervention is used to transmit control signals between system 150 and display 176, speaker 178, or other peripheral devices 180. Output devices can be communicatively coupled to system 150 via dedicated connections through their respective interfaces 164, 166, and 168. Alternatively, output devices can be connected to system 150 via communication interface 160 using communication channel 162. Display 176 and speaker 178 can be integrated into a single unit with other components of system 150 in an electronic device such as a television. In various embodiments, display interface 164 includes a display driver, such as, for example, a timing controller (TCon) chip.

[0130] Display 176 and speaker 178 may be separate from, for example, one or more other components if the RF section of input 172 is part of a separate set-top box. In various embodiments where display 176 and speaker 178 are external components, output signals may be provided via dedicated output connections, including, for example, HDMI ports, USB ports, or COMP outputs.

[0131] System 150 may include one or more sensor devices 168. Examples of sensor devices that may be used include one or more GPS sensors, gyroscope sensors, accelerometers, light sensors, cameras, depth cameras, microphones, and / or magnetometers. Such sensors can be used to determine information such as the user's position and orientation. Where system 150 is used as a control module for an extended reality display (such as control modules 124, 132), the user's position and orientation can be used to determine how image data is rendered so that the user perceives the correct portion of a virtual object or scene from the correct viewpoint. In the case of a head-mounted display device, the position and orientation of the device itself can be used to determine the position and orientation of the user used to render virtual content. In the case of other display devices, such as telephones, tablets, computer monitors, or televisions, other inputs can be used to determine the position and orientation of the user used to render content. For example, the user can use a touchscreen, keypad or keyboard, trackball, joystick, or other inputs to select and / or adjust the desired viewpoint and / or viewing direction. Where the display device has sensors such as accelerometers and / or gyroscopes, the viewpoint and orientation used to render content can be selected and / or adjusted based on the movement of the display device.

[0132] The embodiments can be implemented by computer software implemented by processor 152, or by hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments can be implemented by one or more integrated circuits. Memory 154 can be of any type suitable for the technical environment and can be implemented using any suitable data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory, as non-limiting examples. Processor 154 can be of any type suitable for the technical environment and, as a non-limiting example, can encompass one or more of microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures.

[0133] Block-based video encoding and decoding Like HEVC, VVC is built on a block-based hybrid video codec framework. Figure 2A A block diagram of a block-based hybrid video coding system 200 is given. Variations of this encoder 200 are envisioned, but for clarity, the encoder 200 is described below without describing all anticipated variations.

[0134] Before being encoded, the video sequence can undergo pre-coding (204), for example, applying color transformations to the input color image (e.g., converting from RGB 4:4:4 to YCbCr 4:2:0), or performing remapping on the input image components to obtain a signal distribution that is more robust to compression (e.g., using histogram equalization of one of the color components). Metadata can be associated with the pre-processing and attached to the bitstream.

[0135] The input video signal 202, including the image to be encoded, is partitioned (206) and processed block by block, for example, in units of coding units (CUs). Different CUs can have different sizes. In VTM-1.0, CUs can be as large as 128x128 pixels. However, unlike HEVC, which partitions blocks based solely on quadtrees, in VTM-1.0, coding tree units (CTUs) are split into CUs based on quadtrees / binaries / tritrees to accommodate varying local characteristics. Furthermore, the concept of multi-partition unit types in HEVC has been removed, meaning that the separation of CUs, prediction units (PUs), and transform units (TUs) no longer exists in VVC-1.0; instead, each CU is always used as the basic unit for prediction and transform without further partitioning. In the multi-type tree structure, CTUs are first partitioned using a quadtree structure. Then, each quadtree leaf node can be further partitioned using binary and ternary tree structures. Different partitioning types can be used, such as quadtree partitioning, vertical binary partitioning, horizontal binary partitioning, vertical ternary partitioning, and horizontal ternary partitioning.

[0136] exist Figure 2AIn the encoder, spatial prediction (208) and / or temporal prediction (210) can be performed. Spatial prediction (or “intra-frame prediction”) uses samples from neighboring blocks (called reference samples) already encoded in the same video picture / slice to predict the current video block. Spatial prediction reduces the spatial redundancy inherent in the video signal. Temporal prediction (also called “inter-frame prediction” or “motion-compensated prediction”) uses reconstructed pixels from already encoded video pictures to predict the current video block. Temporal prediction reduces the temporal redundancy inherent in the video signal. The temporal prediction signal for a given CU can be signaled by one or more motion vectors (MVs) indicating the amount and direction of motion between the current CU and its temporal reference. In addition, if multiple reference pictures are supported, a reference picture index can be sent to identify which reference picture in the reference picture repository (212) generated the temporal prediction signal.

[0137] The mode decision block (214) in the encoder selects the optimal prediction mode, for example, based on a rate-distortion optimization method. This selection can be made after performing spatial and / or temporal predictions. Intra-frame / inter-frame decisions can be indicated, for example, by a prediction mode flag. The prediction block is subtracted (216) from the current video block to generate the prediction residual. The prediction residual is decorrelated and quantized (220) using a transform (218). (For some blocks, the encoder can bypass the transform and quantization, in which case the residual can be directly encoded or decoded without applying the transform or quantization process.) The quantized residual coefficients are inversely quantized (222) and inversely transformed (224) to form the reconstructed residual, which is then added back to the prediction block (226) to form the reconstructed signal of the CU. Intra-loop filtering, such as deblocking / SAO (sample adaptive offset) filtering (228), can be further applied to the reconstructed CU before the reconstructed signal is put into the reference image repository (212) and used to encode future video blocks. In order to form the output video bitstream 230, the coding mode (inter-frame or intra-frame), prediction mode information, motion information and quantization residual coefficients are sent to the entropy coding unit (108) for further compression and packing to form the bitstream.

[0138] Figure 2B A block diagram of a block-based video decoder 250 is given. As described below, in decoder 250, the bitstream is decoded by decoder elements. Video decoder 250 typically performs operations similar to... Figure 2A The decoding process is the inverse of the encoding process described herein. Encoder 200 typically also performs video decoding as part of the encoding of video data.

[0139] Specifically, the decoder's input includes a video bitstream 252, which can be generated by the video encoder 200. The video bitstream 252 is first unpacked and entropy decoded at the entropy decoding unit 254 to obtain transform coefficients, motion vectors, and other encoded information. Picture partitioning information indicates how to partition the picture. Therefore, the decoder can partition the picture according to the decoded picture partitioning information (256). The encoding mode and prediction information are sent to the spatial prediction unit 258 (if intra-frame coded) or the temporal prediction unit 260 (if inter-frame coded) to form a prediction block. The residual transform coefficients are sent to the inverse quantization unit 262 and the inverse transform unit 264 to reconstruct the residual block. The prediction block and the residual block are then added at 266 to generate a reconstructed block. The reconstructed block can be further processed by in-loop filtering 268 before being stored in the reference picture repository 270 for predicting future video blocks.

[0140] The decoded image 272 can be further processed by post-decoding (274), such as inverse color transformation (e.g., from YCbCr 4:2:0 to RGB 4:4:4) or inverse remapping, which is the opposite of the remapping process performed in pre-encoding (204). The post-decoding process can use metadata derived in the pre-encoding process and signaled in the bitstream. The decoded, processed video can be sent to a display device 276. The display device 276 can be a device separate from the decoder 250, or the decoder 250 and the display device 276 can be components of the same device.

[0141] The various methods and other aspects described in this disclosure can be used to modify modules of the video encoder 200 or decoder 250. Furthermore, the systems and methods disclosed herein are not limited to VVC or HEVC and can be applied to, for example, other standards and recommendations, whether pre-existing or future-developed, and any extensions of such standards and recommendations, including VVC and HEVC. Unless otherwise stated or technically excluded, the aspects described in this disclosure can be used individually or in combination.

[0142] Figure 3A This is a schematic side view illustrating an example waveguide display that can be used with extended reality (XR) applications according to some embodiments. The image is projected by an image generator 302. The image generator 302 can use one or more different technologies to project the image. For example, the image generator 302 can be a laser beam scanning (LBS) projector, a liquid crystal display (LCD), a light-emitting diode (LED) display (including organic LED (OLED) or micro LED (µLED) displays), a digital light processor (DLP), a liquid crystal on silicon (LCoS) display, or other types of image generators or light engines.

[0143] The light representing image 312 generated by image generator 302 is coupled into waveguide 304 by diffraction coupler 306. Coupler 306 diffracts the light representing image 312 into one or more diffraction orders. For example, ray 308, representing one of the rays at the bottom of the image, is diffracted by coupler 306, and one of the diffraction orders 310 (e.g., the second order) is at an angle capable of propagating through waveguide 304 by total internal reflection. Image generator 302 displays the image, as directed by control module 324, which operates to render image data, video data, point cloud data, or other displayable data.

[0144] At least a portion of the light 310 coupled into waveguide 304 by diffraction-in coupler 306 is coupled out of waveguide 304 by diffraction-out coupler 314. At least some of the light coupled out of waveguide 304 replicates the incident angle of the light coupled into the waveguide. For example, in the illustration, out-coupled rays 316a, 316b, and 316c replicate the angle of the in-coupled ray 308. Since the light emanating from the out-coupler replicates the direction of the light entering the in-coupler, the waveguide essentially replicates the original image 312. The user's eye 318 can focus on the replicated image.

[0145] exist Figure 3A In the example, the outgoing coupler 314 couples out only a portion of the light on each reflection, allowing a single input beam (such as beam 308) to generate multiple parallel output beams (such as beams 316a, 316b, and 316c). In this way, at least some light originating from each part of the image may reach the user's eye, even if the eye is not perfectly aligned with the center of the outgoing coupler. For example, if the eye 318 moves downwards, beam 316c may enter the eye even if beams 316a and 316b do not, so the user can still perceive the bottom of image 312 even if their position shifts. The outgoing coupler 314 thus partially operates as an exit pupil expander in the vertical direction. The waveguide may also include one or more additional exit pupil expanders (…). Figure 3A (not shown in the image) to expand the exit pupil in the horizontal direction.

[0146] In some embodiments, waveguide 304 is at least partially transparent to light originating outside the waveguide display. For example, at least some light 320 from a real-world object (such as object 322) passes through waveguide 304, allowing the user to see the real-world object while using the waveguide display. Since the light 320 from the real-world object also passes through diffraction grating 314, multiple diffraction orders will exist, and thus multiple images will exist. To minimize the visibility of multiple images, it is desirable that diffraction order zero (314 undeflected) has high diffraction efficiency for light 320 and order zero, while higher diffraction orders have lower energy. Therefore, in addition to extending and coupling out virtual images, the out-coupler 314 is preferably configured to allow the zeroth order of the real image to pass through. In such embodiments, the image displayed by the waveguide display may appear to be superimposed on the real world.

[0147] Figure 3B This is a schematic side view illustrating an example alternative display type that can be used with extended reality applications according to some embodiments. In the XR head-mounted display device 330, a control module 332 controls a display 334, which may be an LCD, to display images. The head-mounted display includes a partially reflective surface 336 that reflects (and in some embodiments, both reflects and focuses) the image displayed on the LCD so that the image is visible to the user. The partially reflective surface 336 also allows at least some external light to pass through, allowing the user to see their surroundings.

[0148] Figure 3C This is a schematic side view illustrating example alternative display types that can be used with extended reality applications according to some embodiments. In the XR head-mounted display device 340, a control module 342 controls a display 344, which may be an LCD, to display images. The images are focused by one or more lenses of display optics 346 to make them visible to the user. Figure 3C In some examples, external light does not reach the user's eyes directly. However, in some such embodiments, an external camera 348 may be used to capture images of the external environment and display such images on a display 344 along with any virtual content that may also be displayed.

[0149] The embodiments described herein are not limited to any particular type or structure of XR display device.

[0150] This application relates to learning-based point cloud compression and processing systems, as well as other aspects. The field of point cloud compression and processing aims to develop tools for compressing, analyzing, interpolating, representing, and understanding input signals such as point clouds.

[0151] Point cloud data is a common data format across multiple business sectors, such as autonomous driving, robotics, extended reality (XR), gaming, civil engineering, video streaming, teleconferencing, computer graphics, and the animation / film industry. This common data is considered to consume a significant portion of network traffic, such as between connected cars on 5G networks and in immersive communications (VR / AR). Efficient representation formats are required for point cloud understanding and communication. Raw data needs to be properly organized and processed for purposes such as world modeling, sensing, and rendering. Compression of these raw input signals is essential when data storage and transmission are required in relevant scenarios.

[0152] 3D point cloud data consists of discrete samples from visible objects and the background of a captured scene. In practice, a large number of points are needed to fully represent the real world using point samples.

[0153] For example, a typical fully detailed object in an immersive XR scene contains millions of points, while the point cloud of the entire scene may have hundreds of millions of points. Processing such a large-scale input signal is computationally expensive, especially for consumer devices such as smartphones, tablets, and car navigation systems, which have limited computing power.

[0154] Processing or inferring such input signals tends to require efficient storage methods. To store and process input signals at an affordable computational cost, in some embodiments, the input signal may first be downsampled, where the downsampled signal summarizes the geometry of the input signal while having fewer points. The downsampled signal is then fed into subsequent machine tasks for further processing. Efficient feature extractors can be used to improve the quality of packaged data within a given resource budget.

[0155] Due to recent advancements in computing hardware, the article "Representation Learning: A Review and New Perspectives" by Bengio, Yoshua et al., arXiv Preprint, arXiv: 1206.5538 (2012), discusses how autoencoders can be integrated with traditional image compression frameworks, such as the linear transform coding system described in JPEG ITU-R Rec. T.81 & ISO / IEC 10918-1: "Digital Compression and Coding of Continuous Tone Still Images", ITU-R (1992). This framework consists of transform analysis, entropy coding, and transform synthesis modules. Regarding transform analysis, the Discrete Cosine Transform (DCT) function transforms the image data from the signal domain to the transform domain. Following the DCT is a quantizer, which discretizes the transformed coefficients before inputting them into the entropy encoder. The set of coefficients after entropy decoding is resynthesized via the inverse DCT function to reconstruct the compressed image back into the signal domain.

[0156] Building upon the traditional image compression framework described above, the latest challenge in learning-based image compression is to effectively replace the DCT and inverse DCT blocks with artificial neural networks, such as convolutional neural networks (CNNs or CNNs). Such neural networks approximate functions that transfer the input signal from the data domain to the latent domain or vice versa, as was proposed in earlier autoencoder proposals. Another challenge in learning-based compression is making the entropy encoder differentiable, thus making the entire compression system end-to-end learnable.

[0157] Unlike DCT, approximation functions derived from artificial neural networks are often embedded in the input signal as feature vectors in a latent space. The latent space is a mathematical space where features extracted from the input signal are condensed into vector form. This space typically has a lower dimension than the extracted feature space. Therefore, this transformation can be performed for compression purposes in some embodiments. Elements in the latent space vectors can be inferred via a trained neural network-based encoder. In the downsampling case, blocks of the input signal are condensed into single feature vectors, where the set of feature vectors representing the entire downsampled input is called a feature map. This compressed feature map can be passed through a set of neural network layers and an entropy bottleneck, and then decompressed via a decoding architecture. The entropy bottleneck can include an entropy encoder, a bitstream, and an entropy decoder. In some embodiments, the entropy bottleneck is an entropy encoder. Considering this pipeline, one of the main challenges of learning-based compression is generating highly abstract feature maps that contain the finest details of a given input signal and can be decoded by a carefully designed neural network architecture.

[0158] Inspired by learning-based image compression methods, a similar architectural design has been introduced into the field of point cloud research. Compared to the regularly arranged 2D pixel samples of an image, an additional challenge is to effectively interpret the sparsity properties of 3D point clouds. To address this problem, various sparse convolution methods have been introduced, as detailed in Graham and Ben's "Sparse 3D Convolutional Neural Networks," arXiv Preprint, arXiv:1505.02890 (2015); Liu, Baoyuan et al.'s "Sparse Convolutional Neural Networks," Proc. of IEEE Conf. on Comp. Vision and Pattern Recognition, 806-814 (2015); Graham, Benjamin et al.'s "3D Semantic Segmentation with Submanifold Sparse Convolutional Networks," Proc. of IEEE Conf. on Comp. Vision and Pattern Recognition, 9224-9232 (2018); and Choy, Christopher et al.'s "4D Spatio-Temporal Convnets: Minkowski Convolutional Neural Networks," Proc. Of IEEE / CVF Conf. on Comp. Vision and Pattern Described in Recognition, 3075-3084 (2019). Based on these so-called sparse CNNs, learning-based point cloud compression (PCC) has become an interesting topic in the computer vision and machine learning communities. The processing of feature maps from point cloud input signals and their efficient entropy encoding remain challenging, just like image compression.

[0159] With the growing interest in applying sparse convolution to point cloud compression (PCC), point feature extraction from point cloud data can be used in a wide range of learning-based point cloud analysis applications, such as classification, segmentation, registration, and compression. In these diverse applications, PCC benefits from a trade-off between complexity (e.g., computational cost or storage consumption) and performance (e.g., accuracy) for reconstruction. PCC also utilizes a paired architecture between encoders and decoders.

[0160] The importance of generating efficient feature maps in learning-based compression systems across many applications has been described above. One possible way to meet this criterion is to design neural network layers that preserve the detailed properties of the compressed point samples. However, varying rates of change and distortion optimization objectives, or complex architectures designed for multiple purposes, make it difficult to retain all the details within the computed feature maps. It may be beneficial to directly process or correct the feature maps to preserve or even enhance the feature representation for better compression.

[0161] State-of-the-art work in end-to-end learning-based image compression has demonstrated significant coding gains by analyzing probability distributions in the entropy coding process. For example, the paper "Variational Image Compression with a Scale Hyperprior" by Ballé, Johannes, et al., Intern'l Conf. on Learning Representations (2018) ("Ballé") describes a change in the entropy model of an autoencoder, where auxiliary information is sent in an additional bitstream. This hyperprior network performs scale capture on the quantized output of the encoded feature map. The estimated scale is then applied to both the arithmetic encoder and decoder to improve the performance of the entropy model.

[0162] In the PCC framework, point-based feature map distribution analysis and tuning can be used to improve the coding gain of point cloud compression. In the '424 application, a predetermined set of distribution parameters is applied to each local point in the PCC encoder. The distribution of feature elements is transformed to decorrelate the feature map representation for better point cloud reconstruction.

[0163] Based on observations of related work on learning-based compression, adjusting the feature distribution is a challenging process. Since feature maps are passed between most neural network layers, and because encoding and decoding time are critical in the compression framework, distribution adjustment can remain simple, unsupervised, and easily modular for some embodiments. Furthermore, distribution adjustment can emphasize feature elements differently for the input signal. Therefore, a neural network-based distributed parameter learning framework can approximate a more suitable nonlinear function for compression.

[0164] Typical learning-based point cloud compression systems often generate feature maps as representations of the input signal. The compression of the input signal is then converted into entropy coding of the feature maps. The efficiency of entropy coding often heavily depends on the distribution of the feature maps. This trade-off can favor entropy coding when the feature distribution is updated based on a predetermined and fixed set of distribution parameters. The coding gains may be limited because the manipulation of the feature distribution cannot adapt to the input signal. Therefore, learnable methods can adaptively update the feature distribution to improve the encoding and decoding efficiency of point cloud input signals.

[0165] The compression framework employs a good trade-off between reconstruction accuracy and computational cost. Distributed learning networks are flexible and scalable microarchitectures that can be applied to various end-to-end learning-based point cloud compression systems to improve the encoding and decoding gains of existing compression frameworks.

[0166] This architecture learns multiple sets of distribution parameters from the input feature map via neural network layers. These distribution parameters can be applied to further preserve or even enhance the level of detail abstraction of the feature map through the point cloud compression system. Distributed learning networks (DLNs) can be further enhanced by combining DLNs with other feature extractors, such as residual networks or transform-based blocks. Distributed learning networks can also be extended to multi-level architectures for some implementations.

[0167] Distribution analysis and adjustment on feature maps are processes that can be used to improve the encoding and decoding performance of learning-based compression systems. For learning-based point cloud compression (PCC), manipulation of feature distributions is a relatively new area. Similar to learning-based image compression, distribution analysis often focuses on entropy models. This can be demonstrated by the end-to-end PCC framework recently proposed in Wang, Jianqiang et al., “Lossy Point Cloud Geometry Compression via End-to-End Learning,” 12 IEEE Transactions on Circuits and Systems for Video Technology 31 4909-4923 (2021) (“Wang”) and ’861 application. Ballé’s decomposition entropy model is adapted to these end-to-end PCC frameworks.

[0168] The paper "Rethinking Network Design and Local Geometry in Point Cloud: A Simple Residual MLP Framework" by Ma, Xu, et al., arXiv Preprint, arXiv:2202.07123 (2022), introduces the feature distribution adjustment method PointMLP for classification. This PointMLP is designed as a hierarchical, multi-level architecture with an affine local geometry extractor and demonstrates good improvements on classification tasks. However, this example method requires multi-level sampling and grouping processes, which can be expensive for time-critical processes. Furthermore, the optimized loss function formulated for classification problems (such as cross-entropy) differs from the loss function used for end-to-end PCC (rate and distortion). Therefore, such a feature distribution method may not be entirely suitable for learning-based PCC frameworks because the trade-off between accuracy and complexity is not carefully considered.

[0169] Recent advancements in feature map distribution adjustment within the PCC framework are the point-based feature extractors introduced in '424 and '600 applications. In this approach, multiple sets of distribution parameters are pre-trained directly from the end-to-end PCC architecture and subsequently applied to transform elements of the feature map. The method analyzes each local group (which, according to some embodiments, is a block of points to be downsampled) before sending it via a bitstream. For each block, two sets of distribution parameters are learned, which helps to aggregate the feature map using information from local neighbors. In some embodiments, aggregating the feature map can be a process of combining several values ​​into a representative value. In some embodiments, neighbors within a block are aggregated to generate representative features for the block. In this method, the distribution parameters are pre-trained without specific neural network layers dedicated to them. Such a structure is quite adaptable to encoders because the architecture focuses on neighbor analysis and distribution adjustment of multiple local point groups.

[0170] This application describes a feature extractor that extracts features from sample points using an example neural network microarchitecture, according to some embodiments. The example neural network microarchitecture according to some embodiments further enhances the abstract details of the feature map (FM) through a learning-based compression framework. This example architecture according to some embodiments takes any feature as input and manipulates each feature element by applying multiple sets of learned distribution parameters. In some embodiments, the feature elements then become more discriminative for better reconstruction. This example architecture can be inserted at one or more locations between neural network blocks and can be applied to both encoder and decoder architectures.

[0171] According to some embodiments, a distributed learning network is first introduced, and the manipulation of feature elements through normalization and transformation processes is further described in detail. Both the normalization and transformation processes can learn their own set of distributed parameters based on their neural network (NN) layers. In some embodiments, the FM transformation is the primary process. In some embodiments, the FM normalization process can be performed before the transformation to further enhance the feature map.

[0172] According to some embodiments, an extension with downsampling and filtering is introduced, followed by a multi-level architecture. According to some embodiments, an evolved distributed learning architecture is further introduced by combining it with other feature extraction NN layers, such as residual networks, IRNs, point transformers, and voxel transformers. The paper "Deep Residual Learning for Image Recognition" by He, Kaiming et al., Proc. of IEEE / CVF Conf. on Comp. Vision and PatternRecognition, 770-778 (2016) ("He") describes residual networks. The paper "Inception-v4, Inception-Resnet and the Impact of Residual Connections on Learning" by Szegedy, Christian et al., Proc. of AAAI Conf. on Artificial Intelligence, vol. 31, no. 1 (2017) ("Szegedy") describes IRNs. The paper "Point Transformer" by Zhao, Hengshuang et al., ICCV, 16239-16248 (2021) ("... Zhao The paper “VoxelTransformer for 3D Object Detection” by Mao, Jiageng et al., Proc. of IEEE / CVF Intern'l Conf. on Comp. Vision, 3164-3173 (2021) (“Mao”) and “PVT: Point-VoxelTransformer for Point Cloud Learning” by Zhang, Cheng et al., arXiv Preprint, arXiv: 2108.06076 (2021) (“Mao”) describe point transformers. Zhang ” describes the voxel converter.

[0173] Distributed learning networks Figure 4This is a schematic illustration of examples of point-based feature vectors and feature maps according to some embodiments. According to some embodiments, the neural network layer generates feature vectors 402, 406, and 410 for each output point 404, 408, and 412, where the feature vectors exist in a C-dimensional (or C-channel) space. Each channel is a reshaped vector, and each element in the reshaped vector is a feature element. The set of feature vectors is called a feature map. Figure 4 The middle figure shows example feature maps in image space and 3D space 400. For example... Figure 4 As shown in the example, feature map 414 can correspond to n points 418 scattered in 3D. Each of the n points 418 can be associated with C feature elements. Therefore, C(416)×n(418) feature map (FM) 414 can correspond to this example scenario.

[0174] In some applications, one of the main purposes of feature extraction networks is to learn multiple sets of distribution parameters on feature maps and apply them to adjust the elements of the feature maps for point cloud compression. This operation contrasts with supervised classification tasks, which homogenize features within the same semantic category. This application, according to some embodiments, aims to learn different features that are descriptive for the reconstruction of a given signal by inserting a neural network-based adjustment module within a learning-based compression framework. Example feature adjustment blocks according to some embodiments can be trained as part of an end-to-end compression system, based on geometric errors as a loss function (e.g., chamfer distance or cross-entropy). In some embodiments, feature adjustment can be implemented in two steps: element-wise normalization and transformation, where each step is designed with its own neural network layer.

[0175] Feature elements normalized via neural network layers Figure 5 This is a schematic illustration of example distribution parameters corresponding to the reshaping vector, based on some embodiments.

[0176] Normalization is a training technique that can benefit convergence speed and accuracy. Feature maps (FM) can be normalized for more reliable neural network architecture design. In some implementations, each feature vector can be normalized independently so that the norm (length) of the feature vectors is the same (all equal to a predefined value, such as 1).

[0177] For some embodiments, such as Figure 5 As shown, all feature elements from a certain feature channel within FM 502 are placed into a reshaping vector 508. There are a total of "C" reshaping vectors 504. Each reshaping vector 508 is independently normalized by applying distribution parameters estimated for each channel. Such a set of distribution parameters can be in a distribution parameter array 510. In some embodiments, the estimated mean is first removed from the reshaping vectors. (like Figure 5 (as shown), and then based on the estimated standard deviation Perform normalization. For example, suppose... Figure 5 The shaping vector in the vector has n elements. Parameters It will be the mean of n elements 506. For some implementations, Figure 5 It can be viewed as a conceptual diagram for explaining the reshaping vector 508.

[0178] Figure 6 This is a functional block diagram illustrating an example neural network layer for learning feature map distribution parameters, based on some embodiments. A set of distribution parameters 608 (such as the mean) It can be learned through a set of 604 layers of a neural network (NN), an example of which is... Figure 6 Example procedure 600 is shown. For some embodiments, parameters... (Or) can be a set of parameters, where each parameter is normalized to its corresponding vector in FM 602, or a single parameter representing the entire FM. Pooling operation 606 (e.g., average pooling) can define the final form of the estimated parameters. Similarly, for some embodiments, the standard deviation... You can press each channel Learning, or learning per feature map by pooling the entire FM. For some implementations, or It can be computed directly from the initial FM elements without passing the initial FM elements through a set of NN layers. For example... Figure 6 As shown, neural network (NN) layers can be positioned between the original feature map and pooling to enhance the FM, before calculating or estimating distribution parameters (such as...). ).

[0179] Figure 7 It is shown according to some embodiments Figure 6 A schematic illustration of an example of the NN layer proposed in [the document / reference]. Figure 7 An example of a neural network (NN) layer designed using three shared MLP layers 704, 708, and 712 is shown. The outputs of the NN layers 704, 708, and 712 can be pooled to estimate a set of distributed parameters. The pooling operation can be an averaging or maxing operation. The scope of the pooling operation can be, for example, the entire FM, per-channel basis, a subset of the FM, or some other basis. Figure 7 As shown in example procedure 700, the three MLP layers are shared. Original feature map (702) is input to the input MLP layer 704. The original feature map 702 is transformed into a second feature map by the input (first) MLP layer 704. (706). The second feature map 706 is transformed into the third feature map after passing through the hidden (second) MLP layer 708. (710). The third feature map 710 is transformed into the fourth feature map through the output (third) MLP layer 712. (714). This MLP architecture 700 learns how to refine feature maps before pooling to a set of distribution parameters. The type of distribution parameters can be determined by... Figure 8 , 10 The network design is defined as shown in Figures 11 or 12. During the training phase, each "NN layer" is trained by inputting feature maps ( )702 is adjusted to output feature map ( )714, to better estimate distribution parameters such as mean, standard deviation, scale and bias.

[0180] Figure 8 This is a functional block diagram illustrating the first example feature map normalization process according to some embodiments. Figure 8 An example of a learning-based FM normalization process 800 is shown. The initial FM 802 is input into the "NN layer 1" block 804, where a set of mean parameters is estimated. 806. Estimated mean parameter ( 806 is expanded back to the feature map by 808 and subtracted from the original feature map. Expanding back is a process of adding parameter values ​​(e.g., ...) to the original feature map. The process of copying to the c-channel axis. Centered feature map. (810) is passed to block 812 for estimation. The estimated standard deviation parameter ( 814 is expanded to 816 and returned to the feature map. The mean and standard deviation parameters are examples of the distribution parameters 806 and 814. Centered feature map Divide 810 by the estimated standard deviation parameter to generate a normalized feature map. 818. For some embodiments, for the centered feature map matrix For each column of 810, each element in that particular column is divided by the estimated standard deviation parameter array ( The element in the column that corresponds to that specific column.

[0181] Formula 1 shows the normalization formula: (1) like Figure 8 As shown, The operator is obtained by element-wise addition of FM and the extended mean parameter (e.g.) Figure 8 (As shown in the subtraction diagram) the center of each c-th dimension vector element is shifted to zero to generate the estimated recentered feature map. . Operator through The elements are scaled by element-wise multiplication of 810 with the expanded and inverted standard deviation parameter to generate a normalized feature map. 818. In some embodiments, expansion is performed first, followed by inversion, and then element-wise multiplication. In some embodiments, a very small floating-point value can be added to the estimated standard deviation distribution parameter. ϵ As shown in Formula 1, this is to avoid the possibility of division by zero.

[0182] Figure 9 This is a schematic illustration showing examples of extending to the feature map dimension according to some embodiments. For some embodiments, Figure 8 The operation of "extend to FM" blocks 808 and 816 is as follows: Figure 9 As shown. A set of estimated mean parameters. 910 is input to process / block 908, which expands the FM dimension to match the dimension of the original FM. In some embodiments, the distribution parameter array 910 may be repeated for each of the n rows of the FM matrix to increase the dimension of the distribution parameter array to match the FM matrix. In some embodiments, the expanded distribution parameter matrix 902 may have a dimension of C-dimensional 904 multiplied by n pixels 906.

[0183] Figure 10 This is a functional block diagram illustrating the second example feature map normalization process according to some embodiments. For some embodiments, such as... Figure 10 As shown, a set of distribution parameters 1014 can be learned from "NN layer 2" block 1012. For some embodiments ( Figure 10 (Not shown in the image), the input to the "NN layer 2" block can be the initial FM rather than the centralized one. .

[0184] The initial FM1002 is input into block 1004 of “NN layer 1”, where a set of mean distribution parameters is estimated. 1006. Estimated mean distribution parameters ( 1006 is expanded to 1008 and returned to the feature map, and subtracted from the original feature map 1002. Centered feature map 1010 is passed to block 1012 of “NN layer 2” for estimation. The estimated standard deviation distribution parameter ( 1014 is expanded by 1016 back to the feature map. Centralized feature map. Divide 1010 by the estimated standard deviation distribution parameter to generate a normalized feature map. 1018. For some embodiments, for the centered feature map matrix... For each column of 1010, each element in that particular column is divided by the estimated standard deviation distribution parameter array ( The element corresponding to that specific column in the table. In some embodiments, a very small floating-point value may be added to the estimated standard deviation distribution parameter. ϵTo avoid the possibility of division by zero.

[0185] Transforming feature elements via neural network layers Transforming vector elements in a feature map (FM) using distributed parameters can emphasize the properties of reconstructed points embedded in the FM. This channel-level (or point-by-point) independent vector element transformation can effectively tune the expressive power of the abstract FM by optimizing the proposed distributed learning model through an end-to-end training process. Independent parameter estimation for each vector further distinguishes feature representations by breaking the rigidity of deformations within the FM.

[0186] Figure 11 This is a functional block diagram illustrating a first example feature map transformation process according to some embodiments. The transformation can be performed using parameters. γ and β The entire set of elements is scaled and translated. In some embodiments, the parameters can be predetermined. Different scaling and translation value pairs can be applied separately to each vector in the feature map. This separation can be performed on the feature vectors, but... Figure 5 As shown, the feature vector can be reshaped into a per-channel vector. Transformation parameters and (They can be distribution parameters) and The elements are defined and applied independently for each reshaping vector (channel level). These neural network layers, “NN layer 3” 1104 and “NN layer 4” 1112, can be trained by an end-to-end PCC framework. During inference, features can be derived from a given input feature map. 1102 Estimate the optimal set of distribution parameters and .

[0187] For some implementations, such as when the entire FM is transformed at once, γ and β Used to perform linear transformations on feature maps. For example, suppose a 3D object is located in a virtual scene. Parameters γ and β The 3D object is scaled and translated to another location. The point coordinates of the 3D object can be represented as a feature map (FM) after the feature extraction process. Parameters γ and β It can also be applied to FM. To more precisely adjust the distribution of the feature map, the parameters... γ and β It can be learned / estimated on a per-channel basis. In this case, each parameter... γ and β By converting the parameters to vectorized parameters, the transformation becomes a nonlinear transformation with more degrees of freedom to deform FM.

[0188] For some embodiments of example process 1100, given normalization Taking 1102 as input, the distribution parameters per channel for each group (which form a vector in C dimensions) are learned by a set of neural network (NN) layers. The operation of this set of NN layers 1104, 1112 (which can learn the core groups 1106, 1114 of the distribution parameters) can be similar to... Figure 7 The given description.

[0189] Using these NN layer groups 1104, 1112, such as Figure 11 As shown, two sets of distribution parameters γ and β It is learned through its own "NN layer 3" block 1104 and "NN layer 4" block 1112 respectively. Each set of parameters... γ and β After "extending to FM" blocks 1108 and 1116 to match the FM dimension, and for their respective and Preparing for the operation. This transformation learning architecture via NN layers is based on transformation formula 2: (2) like Figure 11 As shown, Operator through With extended distribution parameter set γ Element-wise multiplication is used to scale the elements in order to generate a scaled feature map. 1110. Scaling Feature Maps — By Operation 1110 and 1102 — Used as input to block 1112 of “NN layer 4”. The operator uses a set of distribution parameters corresponding to a specific column. β Extended elements, translation and scaling of feature maps Each element in a specific column of 1110. Operator-generated transformation feature map 1118.

[0190] Figure 12 This is a functional block diagram illustrating the second example feature map transformation process according to some embodiments. In some embodiments, "NN layer 3" block 1204, "NN layer 4" block 1212 and "extend to FM" blocks 1208, 1216 operate as described above.

[0191] For some embodiments, such as Figure 12 As shown, the FM input 1202 of “NN layer 4” is parallel. No scaling of feature maps is used. Instead of using block 1210 as input to "NN layer 4" block 1212, the normalized feature map is used. 1202 is directly used as the input to "NN layer 4" 1212. This variation of the input FM can be achieved by independently optimizing the distribution parameter set. β To improve the training of NN layers.

[0192] For some embodiments of example process 1200 The operator will scale the feature map 1210 is added to the output of the extended FM process 1216 to generate a transformed feature map. 1218.

[0193] In some embodiments, a single transform block can be used without a normalization process. In this case, the initial FM is directly input into the "NN layer 3" and / or "NN layer 4" block, instead of inputting a normalized feature map. .

[0194] For some embodiments, the distributed parameters may be shared by: (1) the process within the encoder; (2) the process within the decoder; and / or (3) the process in the encoder and the process in the decoder, and vice versa.

[0195] Expansion towards a multi-level architecture Multi-level architectures can further refine the feature maps computed by the distributed learning network described above. To enable deep and smooth communication between levels, additional features (such as downsampling or filtering) can be added after feature map transformation.

[0196] Downsampled feature map Figure 13 This is a functional block diagram illustrating examples of downsampled feature maps according to some embodiments. Voxel-based layers (e.g., CNNs) and point-based neural network layers (e.g., shared MLPs in PointNet, as described in Qi, Charles R. et al., "Pointnet: Deep Learning on Point Sets for 3D Classification and Segmentation," Proc. of IEEE Conf. on Comp. Vision and Pattern Recognition, 652-660 (2017)) can both downsample points via subsequent max or average pooling operations. The degree of downsampling can be further adjusted by the size of local groups in the image or 3D space. Figure 13 In the given set, there are m input feature maps. By controlling this number m, the size of the local groups can be adjusted. If m is larger, the size of the local groups becomes larger.

[0197] exist Figure 13 In example process 1300,m Feature map example groups 1302, 1304, and 1306 are processed by shared "Distributed Learning Network" blocks 1308, 1310, and 1312. In some embodiments, the DLN block may be, for example, Figure 7 or Figure 9 The process shown plus Figure 10 or Figure 11 The process is a combination of steps. Each FM 1302, 1304, 1306 of the local group pixels is transformed into transformed feature maps 1314, 1316, 1318 via distributed learning networks 1308, 1310, 1312. Transformed feature maps 1314, 1316, 1318 are then pooled 1320, 1322, 1324 respectively to form feature vectors FV 1326, 1328, 1330. In this way, features are progressively aggregated across multiple layers. For some embodiments, for an n×C transformed feature map... For each of the C columns, the pooling operation can, for example, determine the average value of the elements in a particular column, or, for example, select the maximum value of the elements in a particular column.

[0198] Feature map filtering Figure 14 This is a functional block diagram illustrating example filtering of feature maps according to some embodiments. In some embodiments, the dimensions of points and / or feature dimensions are kept constant across different levels of a multi-level design. This structure improves compatibility and allows the methods described above to be inserted between layers of any end-to-end compression framework. In some embodiments, such as... Figure 14 As shown in example structure 1400, the downsampled feature vector (FV) 1406 is expanded to the original point size by copying the same vector for each expanded point. In some embodiments, downsampling may occur via pooling process 1404. The feature map (FM) 1402 before downsampling is concatenated to the expanded FV 1410. The expansion of the feature vector (FV) to match the size of the original feature map is performed before concatenation. In some embodiments, concatenation generates a concatenated feature map. 1408. In some embodiments, an additional feedforward network 1412 (e.g., an MLP) can be connected after the concatenation to generate a filtered feature map 1414. This additional neural network layer also matches the input-output dimensions of FM. This process is described in Figure 14 In some embodiments, feedforward network block 1412 may be a neural network with unidirectional information flow. Feedforward network 1412 may be, for example, an MLP or a CNN. In some embodiments, the purpose of feedforward network 1412 is to refine the spliced ​​FM to smoothly adapt to the next block in the process.

[0199] According to some embodiments, based on these downsampling and filtering functions, distributed learning networks can be connected multiple times with various options.

[0200] For some embodiments, Figure 13 The first line can be with Figure 14 combination. Figure 13 First feature map Can be with Figure 13 Extended first eigenvector splicing. See also Figure 14 The extended FV in the model. The feature map of the extension and splicing. Can be input to Figure 14 The feedforward network block outputs a filtered feature map. In some embodiments, Figure 13 First transformation feature map Can be with Figure 13 Extended first feature splicing. For some embodiments, Figure 13 and 14 This combination of features can be used to generate filtered feature maps, such as... Figure 10 and 11 of .

[0201] Combined with another feature extractor In some embodiments, to further enrich the abstraction level of the proposed feature extractor, neural networks for different purposes can be placed together with distributed learning networks. He and Szegedy's residual network and Zhao, Mao, and Zhang's transformer-based model are examples of other feature extractors that can be combined with distributed learning networks. Two such combined architectures are discussed in the following subsections.

[0202] Parallel architecture Figure 15 This is a functional block diagram illustrating an example parallel architecture for enhancing distributed learning networks, based on some embodiments. Figure 15 The diagram illustrates an enhanced distributed learning network process 1500, in which a distributed learning network 1504, whose output transform FM, is combined in parallel with another feature extraction network 1506. Feature map FM 1502 is fed in parallel into the two networks 1504 and 1506. These neural networks 1504 and 1506 extract features from another feature map... Extracting feature maps with adjusted distribution 1510. After concatenating the two feature maps 1508, an additional feedforward network 1512 (such as an MLP) is added to further merge and concatenate the FM and output an enhanced feature map. 1514. In some embodiments, to avoid degradation in deeper networks with multiple levels, the architecture may be combined with, for example, a residual network. This residual network block ( Figure 15The "another feature extractor" block 1506 in the deep network works in parallel with the distributed learning block to propagate rich representations through the deep network. In some embodiments, Figure 15 The "Distributed Learning Network" block 1504 in the code can handle downsampling and filtering of FM. In this way, the "filtering" process preserves both global and local features.

[0203] Serial architecture Figure 16 This is a functional block diagram illustrating an example serial architecture for enhancing distributed learning networks, based on some embodiments. Similar to... Figure 15 Parallel architecture, Figure 16 Another enhanced distributed learning network process 1600 is shown. Figure 16 In the middle, the output of the distributed learning network block 1604 (which can be a distributed adjusted feature map) Feature map FM 1606 is sequentially combined with another feature extraction network block 1608. In this architecture, feature map FM 1602 is fed into the distributed learning network block 1604. Distributed learning feature map 1606 becomes another network block ( Figure 16 The input of the "another feature extractor" block 1608 is the output of another feature map. Enhanced feature maps 1612 is generated by feedforward network 1610.

[0204] For some embodiments, Figure 16 The serial-enhanced "distributed learning network" in the model can also handle downsampling and filtering FM. For both parallel and serial architectures, in some embodiments, multiple other-purpose networks (more than one) can be combined with the distributed learning network.

[0205] Deep Reinforcement Distributed Learning Networks Figure 17 This is a functional block diagram illustrating an example deep reinforcement distributed learning network according to some embodiments. The reinforcement distributed learning network can be designed as a multi-level architecture 1700. Such deep neural networks (which may be referred to as deep reinforcement distributed learning networks (deep EDL-Net) 1704, 1706, 1708) in... Figure 17 The details are shown below. At each stage, the augmented distributed learning network (a parallel or serial EDL network combined with residual or point transform layers) outputs augmented feature maps at levels 1704, 1706, and 1708. ,in i Indicates the first i Phase. After the final phase. The operation stabilizes the deep network by estimating the residuals of the input feature map 1702. In some embodiments, the network attempts to learn network parameters by reinforcing the chains of distributed learning network blocks, and this... The operation results in an estimate of the FM residual. Output feature map. 1710 is the final rich feature map generated by a deep EDL network.

[0206] Application in a learning-based compression framework Feature extraction in learning-based compression systems is a relatively new field compared to classic point cloud classification or segmentation processes. As observed in the recent learning-based point cloud compression framework from Wang and '861 applications, feature extractors need to be carefully designed to handle the trade-off between fine abstraction and encoding / decoding time.

[0207] The deep EDL-Net architecture enriches point features with scalable design. In some implementations, this architecture can follow any neural network layer that outputs compatible feature maps. The architecture is also suitable for both the encoder and decoder of any learning-based compression framework.

[0208] Voxel-based PCC framework Figure 18 This is a functional block diagram illustrating an example deep augmentation distributed learning network in a voxel-based point cloud compression framework, based on some embodiments. In the voxel-based PCC architecture, points are located in a regular voxel grid. Downsampling of points can be accomplished by merging neighboring voxels in the regular grid to a coarser resolution. For example, some sparse CNN layers can be placed for two “voxel downsampling NN layers”. Upsampling can be performed using some sparse CNN layers, for example, placed in two “voxel upsampling NN layers”, to reconstruct the merged neighboring voxels. Recent examples of voxel-based PCC are proposed in Wang and '798 applications.

[0209] The voxel-based PCC architecture 1800 can be combined with deep EDL-Net blocks, such as... Figure 18 The architecture 1800 is shown in the diagram. For example, between voxel upsampling layers 1 (1822) and 2 (1818) in the decoder, a deep EDL-Net 1820 can be inserted to enhance the feature maps upsampled from the voxel upsampling NN layers 2 (1818). Figure 18 Other possible locations where depths EDL-Net 1806, 1810, 1816, and 1820 are applied for some embodiments are also shown.

[0210] In some embodiments, the input point cloud 1802 can be fed into a voxel downsampling NN layer 1 block 1804, and the output passes through a depth EDL-Net block 1806. The output of depth EDL-Net block 1806 can pass through a voxel downsampling NN layer 2 block 1808, and the output passes through a second depth EDL-Net block 1810. The output of the second depth EDL-Net block 1810 can pass through an entropy encoder 1812 to generate a bitstream. On the decoder side, the bitstream can pass through an entropy decoder 1814. The output of entropy decoder 1814 can pass through a depth EDL-Net block 1816. The output of depth EDL-Net block 1816 can pass through a voxel upsampling NN layer 2 block 1818, and its output can pass through a second depth EDL-Net block 1820. The output of the second depth EDL-Net block 1820 can pass through a voxel upsampling NN layer 1 block 1822 to generate an output point cloud 1824.

[0211] Point-based FCC framework Figure 19 This is a functional block diagram illustrating an example deep augmented distributed learning network in a point cloud compression framework based on some embodiments. A recent point-based PCC architecture is described in '424 application. The point-based PCC architecture uses an end-to-end PCC framework that handles point and feature analysis in the encoder and performs feature and point synthesis in the decoder. In some embodiments, on the encoder side, residuals of points around a central block are computed for each block. Features are embedded onto each block (via point analysis). These blocks and their corresponding features are downsampled to generate more condensed features (via feature analysis). On the decoder side, the condensed features are upsampled to create finer blocks (via feature synthesis). The surrounding points of each block are reconstructed (via point synthesis). In this framework, an MLP and an aggregation layer are combined in both the point analysis and synthesis layers.

[0212] The point-based PCC architecture can be combined with deep EDL-Net blocks, such as Figure 18 As shown. For example, a deep EDL-Net 1920 can be inserted between the feature synthesis layer 1918 and the point synthesis layer 1922 in the decoder to enhance the feature map generated by the feature synthesis NN layer block. Figure 19 Other locations where depth EDL-Net 1906, 1910, 1916, and 1920 are applied for some embodiments are shown.

[0213] In some embodiments, the input point cloud 1902 can be fed into a point analysis NN layer block 1904, and the output passes through a deep EDL-Net block 1906. The output of the deep EDL-Net block 1906 can pass through a feature analysis NN layer block 1908, and the output passes through a second deep EDL-Net block 1910. The output of the second deep EDL-Net block 1910 can pass through an entropy encoder 1912 to generate a bitstream. On the decoder side, the bitstream can pass through an entropy decoder 1914. The output of the entropy decoder 1914 can pass through a deep EDL-Net block 1916. The output of the deep EDL-Net block 1916 can pass through a feature synthesis NN layer block 1918, and its output can pass through a second deep EDL-Net block 1920. The output of the second deep EDL-Net block 1920 can pass through a point synthesis NN layer block 1922 to generate an output point cloud 1924.

[0214] Figure 20 This is a flowchart illustrating an example of a learning-based point cloud geometry process according to some embodiments. In some embodiments, example process 2000 may include obtaining a first feature map (2002), wherein the first feature map includes C channels, and wherein the first feature map is generated by one or more previous neural network layers. In some embodiments, example process 2000 may also include generating a first set of distribution parameters (2004) based on the first feature map using a first set of neural network layers. In some embodiments, example process 2000 may also include transforming the first feature map (2006) into a second feature map based on the first set of distribution parameters. In some embodiments, example process 2000 may also include outputting the second feature map (2008) to subsequent neural network layers.

[0215] Figure 21This is a flowchart illustrating an example of a learning-based point cloud geometry process according to some embodiments. In some embodiments, example process 2100 may include obtaining 2102 a first feature map, wherein the first feature map includes C channels, and wherein the first feature map is generated by one or more previous neural network layers. In some embodiments, example process 2100 may also include generating 2104 a first set of distribution parameters based on the first feature map using a first set of neural network layers. In some embodiments, example process 2100 may also include expanding 2106 the first set of distribution parameters to match the first feature map in a dimension. In some embodiments, example process 2100 may also include subtracting 2108 the expanded first set of distribution parameters from the first feature map to generate a centered feature map. In some embodiments, example process 2100 may also include generating 2110 a second set of distribution parameters based on the first feature map using a second set of neural network layers. In some embodiments, example process 2100 may also include expanding 2112 the second set of distribution parameters to match the centered feature map in a dimension. In some embodiments, example process 2100 may further include dividing the centered feature map by a second set of distribution parameters expanded by 2114 to generate a normalized feature map. In some embodiments, example process 2100 may further include outputting the normalized feature map 2116 to a subsequent neural network layer.

[0216] Figure 22 This is a flowchart illustrating an example of a learning-based point cloud geometry process according to some embodiments. In some embodiments, example process 2200 may include obtaining 2202 a normalized feature map, wherein the normalized feature map includes C channels corresponding to the original feature map, and wherein the original feature map and the normalized feature map are generated by one or more previous neural network layers, respectively. In some embodiments, example process 2200 may also include generating 2204 a first set of distribution parameters based on the normalized feature map using a first set of neural network layers. In some embodiments, example process 2200 may also include expanding 2206 the first set of distribution parameters to match the normalized feature map in dimensions. In some embodiments, example process 2200 may also include scaling 2208 the normalized feature map by the expanded first set of distribution parameters to generate a scaled feature map. In some embodiments, example process 2200 may also include generating 2210 a second set of distribution parameters based on the normalized feature map using a second set of neural network layers. In some embodiments, example process 2200 may also include expanding 2212 the second set of distribution parameters to match the scaled feature map in dimensions. In some embodiments, example process 2200 may further include adding a scaled feature map to a second set of distribution parameters expanded at 2214 to generate a transformed feature map. In some embodiments, example process 2200 may further include outputting the transformed feature map at 2216 to a subsequent neural network layer.

[0217] While the methods and systems according to some embodiments are generally discussed in the context of extended reality (XR), some embodiments can be applied to any XR context, such as, for example, virtual reality (VR) / mixed reality (MR) / augmented reality (AR) contexts. Furthermore, although the term "head-mounted display (HMD)" is used herein according to some embodiments, some embodiments can be applied to wearable devices capable of, for example, XR, VR, AR, and / or MR (which may or may not be attached to the head) in some embodiments.

[0218] A first example method according to some embodiments may include: obtaining a first feature map, wherein the first feature map includes C channels, and wherein the first feature map is generated by one or more previous neural network layers; generating a first set of distribution parameters based on the first feature map using a first set of neural network layers; transforming the first feature map into a second feature map based on the first set of distribution parameters; and outputting the second feature map to a subsequent neural network layer.

[0219] In some embodiments of the first example method, the first feature map further includes n feature vectors corresponding to n different points in 3D space.

[0220] In some embodiments of the first example method, each feature vector represents a feature of the corresponding point in 3D space; and the first feature map has a dimension of n×C.

[0221] Some embodiments of the first example method may further include: generating a second set of distribution parameters using a second set of neural network layers; transforming a first feature map into a third feature map based on the second set of distribution parameters; and updating the first feature map into the third feature map.

[0222] For some embodiments of the first example method, generating the first set of distribution parameters may include: updating a given feature map via a neural network layer; and simplifying the updated feature map to obtain the first set of per-channel distribution parameters.

[0223] For some embodiments of the first example method, transforming the first feature map may include: generating a corresponding reshaping vector for each of the C channels in the first feature map, wherein each reshaping vector is generated by reshaping the corresponding channel vector in the first feature map; obtaining a distribution parameter for each of the C channels, the distribution parameter corresponding to one of the corresponding reshaping vectors; expanding each distribution parameter to the corresponding feature channel; and transforming each element in the reshaping vector by the expanded distribution parameter for each of the C channels.

[0224] Some embodiments of the first example method may also include updating the first feature map by normalizing each feature element of the first feature map.

[0225] For some embodiments of the first example method, normalizing each feature element of the first feature map may include: generating a third set of distribution parameters using a third set of neural network layers; centering the first feature map to a fourth feature map based on the third set of distribution parameters; and updating the first feature map to the fourth feature map.

[0226] Some embodiments of the first example method may further include: generating a fourth set of distribution parameters from the fourth feature map; and normalizing the fourth feature map into a fifth feature map based on the fourth set of distribution parameters.

[0227] Some embodiments of the first example method may also include: generating a fifth set of distribution parameters using a fifth set of neural network layers; and normalizing a fourth feature map to a sixth feature map based on the fifth set of distribution parameters.

[0228] Some embodiments of the first example method may further include: downsampling a second feature map to generate a first feature vector, wherein the downsampling of the second feature map uses a pooling function.

[0229] For some embodiments of the first example method, the pooling function is selected from the group including average pooling and max pooling.

[0230] Some embodiments of the first example method may further include: generating an extended feature map by extending the first feature vector toward the channel dimension; concatenating the second feature map with the extended feature map to generate a concatenated feature map; and passing the concatenated feature map through a filtering neural network layer to generate a filtered feature map.

[0231] Some embodiments of the first example method may also include using an additional neural network to aggregate a second feature map.

[0232] For some embodiments of the first example method, the additional neural network includes at least one of a sparse convolutional neural network (CNN) and a multilayer perceptron (MLP).

[0233] Some embodiments of the first example method may also include using a residual network to aggregate a second feature map.

[0234] In some embodiments of the first example method, the residual network is a ResNet block.

[0235] Some embodiments of the first example method may also include using transformer blocks to aggregate the second feature map.

[0236] For some embodiments of the first example method, the transformer block is selected from the group including point transformers and voxel transformers.

[0237] Some embodiments of the first example method may further include: generating a seventh feature map by aggregating the second feature map using a neural network, transforming the first feature map into the second feature map in parallel; and concatenating the seventh feature map to the second feature map.

[0238] Some embodiments of the first example method may further include: repeating the learning-based point cloud geometry process once or multiple times to generate an eighth feature map; and adding the eighth feature map to the first feature map to generate a ninth feature map.

[0239] A first example apparatus according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, cause the apparatus to: obtain a first feature map, wherein the first feature map includes C channels, and wherein the first feature map is generated by one or more previous neural network layers; generate a first set of distribution parameters based on the first feature map using a first set of neural network layers; transform the first feature map into a second feature map based on the first set of distribution parameters; and output the second feature map to a subsequent neural network layer.

[0240] In some embodiments of the first example device, the first feature map further includes n feature vectors corresponding to n different points in 3D space.

[0241] In some embodiments of the first example device, each feature vector represents a feature of a corresponding point in 3D space; and the first feature map has a dimension of n×C.

[0242] A second example method according to some embodiments may include: obtaining a first feature map, wherein the first feature map includes C channels, and wherein the first feature map is generated by one or more previous neural network layers; generating a first set of distribution parameters based on the first feature map using a first set of neural network layers; expanding the first set of distribution parameters to match the first feature map in dimensions; subtracting the expanded first set of distribution parameters from the first feature map to generate a centered feature map; generating a second set of distribution parameters based on the first feature map using a second set of neural network layers; expanding the second set of distribution parameters to match the centered feature map in dimensions; dividing the centered feature map by the expanded second set of distribution parameters to generate a normalized feature map; and outputting the normalized feature map to a subsequent neural network layer.

[0243] In some embodiments of the second example method, the first feature map further includes n feature vectors corresponding to n different points in 3D space.

[0244] In some embodiments of the second example method, each feature vector represents a feature of the corresponding point in 3D space; and the first feature map has a dimension of n×C.

[0245] For some embodiments of the second example method, expanding the first set of distribution parameters to match the first feature map in a dimension includes: copying the first distribution parameter vector once or multiple times to match the corresponding dimension of the first feature map in a dimension, the first distribution parameter vector including the first set of distribution parameters; expanding the second set of distribution parameters to match the centered feature map in a dimension includes: copying the second distribution parameter vector once or multiple times to match the corresponding dimension of the centered feature map in a dimension, and the second distribution parameter vector including the second set of distribution parameters.

[0246] A second example apparatus according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, cause the apparatus to: obtain a first feature map, wherein the first feature map includes C channels, and wherein the first feature map is generated by one or more previous neural network layers; generate a first set of distribution parameters based on the first feature map using a first set of neural network layers; expand the first set of distribution parameters to match the first feature map in dimensions; subtract the expanded first set of distribution parameters from the first feature map to generate a centered feature map; generate a second set of distribution parameters based on the first feature map using a second set of neural network layers; expand the second set of distribution parameters to match the centered feature map in dimensions; divide the centered feature map by the expanded second set of distribution parameters to generate a normalized feature map; and output the normalized feature map to a subsequent neural network layer.

[0247] A third example method according to some embodiments may include: obtaining a normalized feature map, wherein the normalized feature map includes C channels corresponding to an original feature map, and wherein the original feature map and the normalized feature map are generated by one or more previous neural network layers respectively; generating a first set of distribution parameters based on the normalized feature map using a first set of neural network layers; expanding the first set of distribution parameters to match the normalized feature map in dimensions; scaling the normalized feature map by the expanded first set of distribution parameters to generate a scaled feature map; generating a second set of distribution parameters based on the normalized feature map using a second set of neural network layers; expanding the second set of distribution parameters to match the scaled feature map in dimensions; adding the scaled feature map to the expanded second set of distribution parameters to generate a transformed feature map; and outputting the transformed feature map to a subsequent neural network layer.

[0248] In some embodiments of the third example method, the first feature map also includes n feature vectors corresponding to n different points in 3D space.

[0249] In some embodiments of the third example method, each feature vector represents a feature of the corresponding point in 3D space; and the first feature map has a dimension of n×C.

[0250] In some embodiments of the third example method, the input to the second set of neural network layers is a scaled feature map.

[0251] In some embodiments of the third example method, the input to the second set of neural network layers is a normalized feature map.

[0252] Some embodiments of the third example method may further include: downsampling the transformed feature map to generate a first feature vector, wherein the downsampling of the transformed feature map uses a pooling function.

[0253] For some embodiments of the third example method, the pooling function is selected from the group including average pooling and max pooling.

[0254] Some embodiments of the third example method may further include: generating an extended feature map by extending the first feature vector toward the channel dimension; concatenating the first feature map with the extended feature map to generate a concatenated feature map; and passing the concatenated feature map through a filtering neural network layer to generate a filtered feature map.

[0255] Some embodiments of the third example method may also include: using an additional neural network to aggregate the first feature map.

[0256] For some embodiments of the third example method, the additional neural network includes at least one of a sparse convolutional neural network (CNN) and a multilayer perceptron (MLP).

[0257] Some embodiments of the third example method may also include using a residual network to aggregate the first feature map.

[0258] In some embodiments of the third example method, the residual network is a ResNet block.

[0259] Some embodiments of the third example method may further include: repeating the learning-based point cloud geometry method once or multiple times to generate an enhanced feature map; and adding the enhanced feature map to the first feature map to generate an enhanced output feature map.

[0260] Some embodiments of the third example method may also include repeating the learning-based point cloud geometry method once or multiple times within the point cloud encoder.

[0261] Some embodiments of the third example method may also include repeating the learning-based point cloud geometry method once or multiple times within the point cloud decoder.

[0262] In some embodiments of the third example method, the point cloud encoder processes voxels.

[0263] For some embodiments of the third example method, the point cloud encoder performs point and feature analysis.

[0264] A third example apparatus according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, cause the apparatus to: obtain a normalized feature map, wherein the normalized feature map includes C channels corresponding to an original feature map, and wherein the original feature map and the normalized feature map are generated by one or more previous neural network layers, respectively; generate a first set of distribution parameters based on the normalized feature map using a first set of neural network layers; expand the first set of distribution parameters to match the normalized feature map in dimensions; scale the normalized feature map using the expanded first set of distribution parameters to generate a scaled feature map; generate a second set of distribution parameters based on the normalized feature map using a second set of neural network layers; expand the second set of distribution parameters to match the scaled feature map in dimensions; add the scaled feature map to the expanded second set of distribution parameters to generate a transformed feature map; and output the transformed feature map to a subsequent neural network layer.

[0265] Example apparatuses according to some embodiments may include at least one processor configured to perform any of the methods listed above.

[0266] Example apparatuses according to some embodiments may include a computer-readable medium storing instructions for causing one or more processors to perform any of the methods listed above.

[0267] Example apparatuses according to some embodiments may include at least one processor and at least one non-transitory computer-readable medium storing instructions for causing at least one processor to perform any of the methods listed above.

[0268] Example apparatuses according to some embodiments may include a computer-readable medium storing a scene description file generated according to any of the methods listed above.

[0269] Example apparatuses according to some embodiments may include signals comprising a scene description file generated according to any of the methods listed above.

[0270] This disclosure describes a variety of aspects, including tools, features, embodiments, models, methods, etc. Many of these aspects are described in detail and are generally described in a manner that may sound restrictive, at least to show individual characteristics. However, this is for the purpose of clarity of description and does not limit the scope of this disclosure or these aspects. In fact, all the different aspects can be combined and interchanged to provide further aspects. Furthermore, these aspects can also be combined and interchanged with aspects described in earlier applications.

[0271] The aspects described and contemplated in this disclosure can be implemented in many different forms. While some embodiments are specifically illustrated, others are contemplated, and the discussion of particular embodiments does not limit the breadth of implementation. At least one of these aspects generally relates to video encoding and decoding, and at least one other aspect generally relates to the transmission of a generated or encoded bitstream. These and other aspects can be implemented as methods, apparatus, computer-readable storage media having instructions thereon stored thereon for encoding or decoding video data according to any of the methods, and / or computer-readable storage media having a bitstream generated according to any of the methods stored thereon.

[0272] In this disclosure, the terms “reconstruction” and “decoding” are used interchangeably, as are the terms “pixel” and “sample”, and the terms “image,” “picture,” and “frame.” Generally, but not necessarily, the term “reconstruction” is used on the encoder side, while “decoding” is used on the decoder side.

[0273] The terms HDR (High Dynamic Range) and SDR (Standard Dynamic Range) generally convey specific dynamic range values ​​to those skilled in the art. However, additional embodiments are contemplated in which references to HDR are understood to mean "higher dynamic range" and references to SDR are understood to mean "lower dynamic range". These additional embodiments are not bound by any specific dynamic range values ​​that may generally be associated with the terms "high dynamic range" and "standard dynamic range".

[0274] This document describes various methods, and each method includes one or more steps or actions for implementing the method. Unless the correct operation of the method requires a specific order of steps or actions, the order and / or use of specific steps and / or actions can be modified or combined. Furthermore, terms such as "first" and "second" can be used in various embodiments to modify elements, components, steps, operations, etc., such as, for example, "first decoding" and "second decoding." The use of these terms does not imply that the modified operations have a specific order unless specifically required. Therefore, in this example, the first decoding does not need to be performed before the second decoding and can occur, for example, before, during, or during overlapping time periods of the second decoding.

[0275] For example, various numerical values ​​may be used in this disclosure. Specific values ​​are used for illustrative purposes, and the aspects described are not limited to these specific values.

[0276] The embodiments described herein can be executed by computer software implemented in a processor or other hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments can be implemented by one or more integrated circuits. The processor can be of any type suitable for the technical environment and can encompass one or more microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures as non-limiting examples.

[0277] Various implementations involve decoding. As used in this disclosure, "decoding" can encompass, for example, all or part of the processes performed on a received decoded sequence to produce a final output suitable for display. In various embodiments, such processes include one or more processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such processes also include, or alternatively include, processes performed by the decoder of the various implementations described in this disclosure, such as extracting images from a collage (packaging) of images, determining an upsampling filter to use and then upsampling the images, and flipping the images back to their intended orientation.

[0278] As a further example, in one embodiment, "decoding" refers only to entropy decoding; in another embodiment, "decoding" refers only to differential decoding; and in yet another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" is intended to specifically refer to a subset of operations or more broadly to the decoding process will be clear based on the specific context of the description.

[0279] Various implementations involve encoding. Similar to the discussion of “decoding” above, the term “encoding” as used in this disclosure can encompass all or part of the processes performed, for example, on an input video sequence to produce an encoded bitstream. In various embodiments, such processes include one or more processes typically performed by an encoder, such as partitioning, differential coding, transform, quantization, and entropy coding. In various embodiments, such processes also include, or alternatively include, the processes performed by the encoders of the various implementations described in this disclosure.

[0280] As a further example, in one embodiment, "encoding" refers only to entropy encoding; in another embodiment, "encoding" refers only to differential encoding; and in yet another embodiment, "encoding" refers to a combination of differential and entropy encoding. Whether the phrase "encoding process" is intended to specifically refer to a subset of operations or more broadly to the encoding process will be clear based on the specific context of the description.

[0281] When a diagram is presented as a flowchart, it should be understood that it also provides a block diagram of the corresponding device. Similarly, when a diagram is presented as a block diagram, it should be understood that it also provides a flowchart of the corresponding method / process.

[0282] Various implementations involve rate-distortion optimization. Specifically, during the encoding process, often given constraints on computational complexity, a balance or trade-off between rate and distortion is typically considered. Rate-distortion optimization is generally formulated as minimizing a rate-distortion function, which is a weighted sum of rate and distortion. Different approaches exist for solving rate-distortion optimization problems. For example, these approaches can be based on extensive testing of all encoding options, including all considered modes or encoding parameter values, and a complete evaluation of their encoding costs and the correlated distortion of the reconstructed signal after encoding and decoding. Faster methods can also be used to save encoding complexity, particularly by calculating approximate distortion based on the predicted or predicted residual signal, rather than the reconstructed signal. A hybrid of these approaches can also be used, such as by using approximate distortion only for some possible encoding options and full distortion for others. Other methods evaluate only a subset of possible encoding options. More generally, many methods employ any of a variety of techniques to perform optimization, but optimization is not necessarily a complete evaluation of both encoding costs and correlated distortion.

[0283] The implementations and aspects described herein can be implemented, for example, in methods or processes, apparatuses, software programs, data streams, or signals. Even if discussed only in the context of one implementation (e.g., only as a method), the implementation of the features in question can be implemented in other forms (e.g., apparatuses or programs). Apparatuses can be implemented, for example, in suitable hardware, software, and firmware. Methods can be implemented, for example, in a processor, which generally refers to a processing device, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices, such as, for example, computers, cellular phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate information communication between end users.

[0284] References to “an embodiment” or “an embodiment” or “an implementation” or “implementation”, as well as other variations, mean that a particular feature, structure, characteristic, etc., described in connection with that embodiment is included in at least one embodiment. Therefore, the appearance of the phrases “in an embodiment” or “in an embodiment” or “in an implementation” or “in an implementation”, and any other variations, in various places in this disclosure does not necessarily refer to the same embodiment.

[0285] Furthermore, this disclosure can refer to "determining" various pieces of information. Determining information may include, for example, one or more of estimated information, calculated information, predicted information, or information retrieved from memory.

[0286] Furthermore, this disclosure can refer to "accessing" various information fragments. Accessing information may include, for example, receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information, or one or more of these.

[0287] Furthermore, this disclosure can refer to "receiving" various pieces of information. "Receiving," like "accessing," is intended to be a broad term. Receiving information can include, for example, accessing information or retrieving information (e.g., from memory). Moreover, "receiving" is generally involved in some way in operations such as, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0288] It should be understood that the use of “ / ”, “and / or”, and “at least one” in the following cases, such as “A / B”, “A and / or B”, and “at least one of A and B”, is intended to cover selecting only the first listed option (A), or only the second listed option (B), or both options (A and B). As a further example, in the cases of “A, B, and / or C” and “at least one of A, B, and C”, this wording is intended to cover selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A, B, and C). This can be extended to any number of items listed.

[0289] Furthermore, as used herein, the term "signal" refers, among other things, to instructing the corresponding decoder to do something. For example, in some embodiments, the encoder signals a specific one of a plurality of parameters for region-based filter parameter selection used for artifact removal filtering. In this way, in one embodiment, the same parameter is used on both the encoder and decoder sides. Thus, for example, the encoder can transmit (explicit signaling) a specific parameter to the decoder so that the decoder can use the same specific parameter. Conversely, if the decoder already has the specific parameter as well as other parameters, signaling can be used without transmission (implicit signaling) to simply allow the decoder to know and select the specific parameter. Bit savings are achieved in various embodiments by avoiding the transmission of any actual function. It should be understood that signaling can be done in a variety of ways. For example, in various embodiments, information is signaled to the corresponding decoder using one or more syntax elements, flags, etc. While the foregoing refers to the verb form of the term "signal," the term "signal" can also be used as a noun herein.

[0290] The implementation can generate various signals formatted to carry information, which can be stored or transmitted, for example. The information may include, for example, instructions for performing a method, or data generated by one of the implementations. For example, the signal may be formatted to carry a bit stream of the embodiments described. Such a signal may be formatted, for example, as an electromagnetic wave (e.g., using the radio frequency portion of a spectrum) or a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. It is known that the signal can be transmitted via a variety of different wired or wireless links. The signal may be stored on a processor-readable medium.

[0291] We have described several embodiments. Spanning various claim classes and types, the features of these embodiments may be provided individually or in any combination. Furthermore, spanning various claim classes and types, embodiments may include one or more of the following features, devices, or aspects, individually or in any combination: • Adapt the residual at the encoder according to any of the embodiments discussed.

[0292] • A bitstream or signal that includes one or more of the syntax elements or variations thereof.

[0293] • Includes a bitstream or signal that conveys the syntax of information generated according to any of the embodiments described.

[0294] • Insert syntax elements into the signaling that enable the decoder to adapt the residuals in a manner corresponding to that used by the encoder.

[0295] • Creating and / or transmitting and / or receiving and / or decoding bitstreams or signals comprising one or more of the stated syntax elements or variations thereof.

[0296] • Create and / or transmit and / or receive and / or decode according to any of the embodiments described.

[0297] • Methods, processes, apparatus, media for storing instructions, media for storing data, or signals according to any of the embodiments described.

[0298] • TVs, set-top boxes, cellular phones, tablets or other electronic devices that perform filter parameter adaptation according to any of the embodiments described.

[0299] • A TV, set-top box, cellular phone, tablet computer, or other electronic device that performs filter parameter adaptation according to any of the embodiments and displays (e.g., using a monitor, screen, or other type of display) the resulting image.

[0300] • TVs, set-top boxes, cellular phones, tablets, or other electronic devices that select (e.g., using a tuner) channels to receive signals including encoded images and perform filter parameter adaptations according to any of the embodiments described.

[0301] • A TV, set-top box, cellular phone, tablet computer, or other electronic device that receives (e.g., using an antenna) signals transmitted over the air, including encoded images, and performs filter parameter adaptation according to any of the embodiments described.

[0302] Note that the various hardware elements in one or more embodiments described herein are referred to as “modules” that perform (i.e., execute, run, etc.) the various functions described herein in conjunction with corresponding modules. As used herein, a module includes hardware (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), one or more memory devices) that are deemed suitable for a given implementation by those skilled in the art. Each module may also include executable instructions for performing the one or more functions described herein, and it is noted that these instructions may take the form of hardware (i.e., hardwired) instructions, firmware instructions, software instructions, and / or similar instructions, and may be stored in any suitable non-transitory computer-readable medium, such as commonly referred to as RAM, ROM, etc.

[0303] While the features and elements have been described above in specific combinations, those skilled in the art will understand that each feature or element can be used alone or in combination with any other feature and element. Furthermore, the methods described herein can be implemented in a computer program, software, or firmware that is contained in a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROMs and digital multifunction discs (DVDs). The processor can be used in conjunction with software to implement a radio frequency transceiver for use in a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

1. A learning-based point cloud geometry method, the method comprising: Obtain the first feature map. The first feature map includes C channels, and The first feature map is generated by one or more previous neural network layers; The first set of distribution parameters is generated based on the first feature map using the first set of neural network layers; The first feature map is transformed into the second feature map based on the first set of distribution parameters; as well as The bitstream is encoded based on the second feature map.

2. A learning-based point cloud geometry method, the method comprising: Obtain the first feature map. The first feature map includes C channels, and The first feature map is generated by one or more previous neural network layers; The first set of distribution parameters is generated based on the first feature map using the first set of neural network layers; The first feature map is transformed into the second feature map based on the first set of distribution parameters; as well as The second feature map is then output to subsequent neural network layers.

3. The method according to claim 2, wherein the first feature map further includes n feature vectors corresponding to n different points in 3D space.

4. The method according to claim 3, Each feature vector represents a feature of the corresponding point in 3D space; and The first feature map has a dimension of n×C.

5. The method according to claim 2, further comprising: A second set of distribution parameters is generated using a second set of neural network layers; The first feature map is transformed into the third feature map based on the second set of distribution parameters; as well as Update the first feature map to the third feature map.

6. The method of claim 2, wherein generating the first set of distribution parameters comprises: Update the given feature map via neural network layers; as well as The updated feature map is simplified to obtain the first set of per-channel distribution parameters.

7. The method according to claim 2, wherein transforming the first feature map comprises: For each of the C channels in the first feature map, a corresponding reshaping vector is generated. Each reshaping vector is generated by reshaping the corresponding channel vector in the first feature map; Obtain the distribution parameters of each of the C channels, which correspond to one of the corresponding reshaping vectors; Expand each distribution parameter into a corresponding feature channel; and For each of the C channels, reshape each element in the vector by transforming the distribution parameters.

8. The method of claim 2, further comprising updating the first feature map by normalizing each feature element of the first feature map.

9. The method of claim 7, wherein each feature element of the normalized first feature map comprises: The third set of distribution parameters is generated using the third set of neural network layers; The first feature map is centered to the fourth feature map based on the third set of distribution parameters; as well as Update the first feature map to the fourth feature map.

10. The method of claim 9, further comprising: Generate the fourth set of distribution parameters from the fourth feature map; as well as The fourth feature map is normalized into the fifth feature map based on the fourth set of distribution parameters.

11. The method of claim 9, further comprising: The fifth set of distribution parameters is generated using the fifth set of neural network layers; as well as The fourth feature map is normalized into the sixth feature map based on the fifth set of distribution parameters.

12. The method according to claim 2, further comprising: The second feature map is downsampled to generate the first feature vector. The pooling function is used to downsample the second feature map.

13. The method of claim 12, wherein the pooling function is selected from the group consisting of average pooling and max pooling.

14. The method of claim 12, further comprising: An extended feature map is generated by extending the first feature vector toward the channel dimension; The second feature map is concatenated with the extended feature map to generate a concatenated feature map; as well as The concatenated feature map is passed through a filtering neural network layer to generate a filtered feature map.

15. The method of claim 2, further comprising using an additional neural network to aggregate a second feature map.

16. The method of claim 15, wherein the additional neural network comprises at least one of a sparse convolutional neural network (CNN) and a multilayer perceptron (MLP).

17. The method of claim 2, further comprising using a residual network to aggregate a second feature map.

18. The method of claim 17, wherein the residual network is a ResNet block.

19. The method of claim 2, further comprising aggregating a second feature map using a converter block.

20. The method of claim 19, wherein the converter block is selected from the group consisting of point converters and voxel converters.

21. The method according to claim 2, further comprising: The seventh feature map is generated by aggregating the second feature map using a neural network, and in parallel, the first feature map is transformed into the second feature map; as well as The seventh feature map is stitched onto the second feature map.

22. The method according to claim 2, further comprising: Repeat the learning-based point cloud geometry process once or multiple times to generate the eighth feature map; as well as The eighth feature map is added to the first feature map to generate the ninth feature map.

23. An apparatus comprising: processor; as well as A non-transitory computer-readable medium storing instructions, the instructions of which, when executed by a processor, cause the device to: Obtain the first feature map. The first feature map includes C channels, and The first feature map is generated by one or more previous neural network layers; The first set of distribution parameters is generated based on the first feature map using the first set of neural network layers; The first feature map is transformed into the second feature map based on the first set of distribution parameters; as well as The bitstream is encoded based on the second feature map.

24. An apparatus comprising: processor; as well as A non-transitory computer-readable medium storing instructions, the instructions of which, when executed by a processor, cause the device to: Obtain the first feature map. The first feature map includes C channels, and The first feature map is generated by one or more previous neural network layers; The first set of distribution parameters is generated based on the first feature map using the first set of neural network layers; The first feature map is transformed into the second feature map based on the first set of distribution parameters; as well as The second feature map is then output to subsequent neural network layers.

25. The apparatus of claim 24, wherein the first feature map further comprises n feature vectors corresponding to n different points in 3D space.

26. The apparatus according to claim 24, Each feature vector represents a feature of the corresponding point in 3D space; and The first feature map has a dimension of n×C.

27. A learning-based point cloud geometry method, the method comprising: Obtain the first feature map. The first feature map includes C channels, and The first feature map is generated by one or more previous neural network layers; The first set of distribution parameters is generated based on the first feature map using the first set of neural network layers; Expand the first set of distribution parameters to match the first feature map in dimensionality; Subtract the extended first set of distribution parameters from the first feature map to generate a centered feature map; A second set of distribution parameters is generated based on the first feature map using a second set of neural network layers; The second set of distribution parameters is expanded to match the centered feature map in dimensions; Divide the centered feature map by the extended second set of distribution parameters to generate a normalized feature map; and Bitstream encoded based on normalized feature maps.

28. A learning-based point cloud geometry method, the method comprising: Obtain the first feature map. The first feature map includes C channels, and The first feature map is generated by one or more previous neural network layers; The first set of distribution parameters is generated based on the first feature map using the first set of neural network layers; Expand the first set of distribution parameters to match the first feature map in dimensionality; Subtract the extended first set of distribution parameters from the first feature map to generate a centered feature map; A second set of distribution parameters is generated based on the first feature map using a second set of neural network layers; The second set of distribution parameters is expanded to match the centered feature map in dimensions; Divide the centered feature map by the extended second set of distribution parameters to generate a normalized feature map; as well as The normalized feature map is then output to subsequent neural network layers.

29. The method of claim 28, wherein the first feature map further comprises n feature vectors corresponding to n different points in 3D space.

30. The method according to claim 28, Each feature vector represents a feature of the corresponding point in 3D space; and The first feature map has a dimension of n×C.

31. The method according to claim 28, Expanding the first set of distribution parameters to match the first feature map in dimension includes copying the first distribution parameter vector once or multiple times to match the corresponding dimension of the first feature map in dimension. The first distribution parameter vector includes the first set of distribution parameters. Expanding the second set of distribution parameters to match the dimensionality of the centered feature map includes copying the second distribution parameter vector once or multiple times to match the corresponding dimension of the centered feature map. The second distribution parameter vector includes the second set of distribution parameters.

32. An apparatus comprising: processor; as well as A non-transitory computer-readable medium storing instructions, the instructions of which, when executed by a processor, cause the device to: Obtain the first feature map. The first feature map includes C channels, and The first feature map is generated by one or more previous neural network layers; The first set of distribution parameters is generated based on the first feature map using the first set of neural network layers; Expand the first set of distribution parameters to match the first feature map in dimensionality; Subtract the extended first set of distribution parameters from the first feature map to generate a centered feature map; A second set of distribution parameters is generated based on the first feature map using a second set of neural network layers; The second set of distribution parameters is expanded to match the centered feature map in dimensions; Divide the centered feature map by the extended second set of distribution parameters to generate a normalized feature map; and Bitstream encoded based on normalized feature maps.

33. An apparatus comprising: processor; as well as A non-transitory computer-readable medium storing instructions, the instructions of which, when executed by a processor, cause the device to: Obtain the first feature map. The first feature map includes C channels, and The first feature map is generated by one or more previous neural network layers; The first set of distribution parameters is generated based on the first feature map using the first set of neural network layers; Expand the first set of distribution parameters to match the first feature map in dimensionality; Subtract the extended first set of distribution parameters from the first feature map to generate a centered feature map; A second set of distribution parameters is generated based on the first feature map using a second set of neural network layers; The second set of distribution parameters is expanded to match the centered feature map in dimensions; Divide the centered feature map by the extended second set of distribution parameters to generate a normalized feature map; as well as The normalized feature map is then output to subsequent neural network layers.

34. A learning-based point cloud geometry method, the method comprising: Obtain the normalized feature map. The normalized feature map includes C channels corresponding to the original feature map, and The original feature map and the normalized feature map are generated by one or more previous neural network layers respectively; The first set of neural network layers is used to generate the first set of distribution parameters based on the normalized feature map; The first set of distribution parameters is expanded to match the normalized feature map in dimensions; The normalized feature map is scaled using the expanded first set of distribution parameters to generate a scaled feature map; A second set of distribution parameters is generated based on the normalized feature map using a second set of neural network layers; Expand the second set of distribution parameters to match the scaled feature map in dimensions; The scaled feature map is added to the expanded second set of distribution parameters to generate a transformed feature map; as well as Bitstream encoding is based on transform feature maps.

35. A learning-based point cloud geometry method, the method comprising: Obtain the normalized feature map. The normalized feature map includes C channels corresponding to the original feature map, and The original feature map and the normalized feature map are generated by one or more previous neural network layers respectively; The first set of neural network layers is used to generate the first set of distribution parameters based on the normalized feature map; The first set of distribution parameters is expanded to match the normalized feature map in dimensions; The normalized feature map is scaled using the expanded first set of distribution parameters to generate a scaled feature map; A second set of distribution parameters is generated based on the normalized feature map using a second set of neural network layers; Expand the second set of distribution parameters to match the scaled feature map in dimensions; The scaled feature map is added to the expanded second set of distribution parameters to generate a transformed feature map; as well as The transformed feature map is then output to subsequent neural network layers.

36. The method of claim 35, wherein the first feature map further comprises n feature vectors corresponding to n different points in 3D space.

37. The method according to claim 35, Each feature vector represents a feature of the corresponding point in 3D space; and The first feature map has a dimension of n×C.

38. The method of claim 35, wherein the input to the second set of neural network layers is a scaled feature map.

39. The method of claim 35, wherein the input to the second set of neural network layers is a normalized feature map.

40. The method according to any one of claims 35-39, further comprising: The transformed feature map is downsampled to generate the first feature vector. The pooling function is used to downsample the transformed feature map.

41. The method of claim 40, wherein the pooling function is selected from the group consisting of average pooling and max pooling.

42. The method according to any one of claims 40-41, further comprising: An extended feature map is generated by extending the first feature vector toward the channel dimension; The first feature map is concatenated with the expanded feature map to generate a concatenated feature map; as well as The concatenated feature map is passed through a filtering neural network layer to generate a filtered feature map.

43. The method according to any one of claims 35-42, further comprising using an additional neural network to aggregate the first feature map.

44. The method of claim 43, wherein the additional neural network comprises at least one of a sparse convolutional neural network (CNN) and a multilayer perceptron (MLP).

45. The method according to any one of claims 35-42, further comprising using a residual network to aggregate the first feature map.

46. ​​The method of claim 45, wherein the residual network is a ResNet block.

47. The method according to any one of claims 35-46, further comprising: Repeat the learning-based point cloud geometry method one or more times to generate enhanced feature maps; as well as The enhanced feature map is added to the first feature map to generate the enhanced output feature map.

48. The method according to any one of claims 35-47, further comprising repeating the learning-based point cloud geometry method once or multiple times within the point cloud encoder.

49. The method according to any one of claims 35-47, further comprising repeating the learning-based point cloud geometry method once or multiple times within the point cloud decoder.

50. The method according to any one of claims 35-48, wherein the point cloud encoder processes voxels.

51. The method according to any one of claims 35-48, wherein the point cloud encoder performs point and feature analysis.

52. An apparatus comprising: processor; as well as A non-transitory computer-readable medium storing instructions, the instructions of which, when executed by a processor, cause the device to: Obtain the normalized feature map. The normalized feature map includes C channels corresponding to the original feature map, and The original feature map and the normalized feature map are generated by one or more previous neural network layers respectively; The first set of neural network layers is used to generate the first set of distribution parameters based on the normalized feature map; The first set of distribution parameters is expanded to match the normalized feature map in dimensions; The normalized feature map is scaled using the expanded first set of distribution parameters to generate a scaled feature map; A second set of distribution parameters is generated based on the normalized feature map using a second set of neural network layers; Expand the second set of distribution parameters to match the scaled feature map in dimensions; The scaled feature map is added to the expanded second set of distribution parameters to generate a transformed feature map; as well as Bitstream encoding is based on transform feature maps.

53. An apparatus comprising: processor; as well as A non-transitory computer-readable medium storing instructions, the instructions of which, when executed by a processor, cause the device to: Obtain the normalized feature map. The normalized feature map includes C channels corresponding to the original feature map, and The original feature map and the normalized feature map are generated by one or more previous neural network layers respectively; The first set of neural network layers is used to generate the first set of distribution parameters based on the normalized feature map; The first set of distribution parameters is expanded to match the normalized feature map in dimensions; The normalized feature map is scaled using the expanded first set of distribution parameters to generate a scaled feature map; A second set of distribution parameters is generated based on the normalized feature map using a second set of neural network layers; Expand the second set of distribution parameters to match the scaled feature map in dimensions; The scaled feature map is added to the expanded second set of distribution parameters to generate a transformed feature map; as well as The transformed feature map is then output to subsequent neural network layers.

54. An apparatus comprising at least one processor configured to perform the method of any one of claims 1-22, 27-31 and 34-51.

55. An apparatus comprising a computer-readable medium storing instructions for causing one or more processors to perform the method of any one of claims 1-22, 27-31, and 34-51.

56. An apparatus comprising at least one processor and at least one non-transitory computer-readable medium storing instructions for causing the at least one processor to perform the method of any one of claims 1-22, 27-31 and 34-51.

57. A computer-readable medium storing a feature map according to any one of claims 1-22, 27-31 and 34-51.

58. A signal comprising a feature map according to any one of claims 1-22, 27-31 and 34-51.

Images