Adaptive geometric metric for 3D point clouds

Abstract

Some embodiments of a method can include performing a processing loop for each point in a reference point cloud, wherein the processing loop includes, for a current iteration through the processing loop, selecting a first point equal to a respective point in the reference point cloud, obtaining a local covariance, determining a set of distances, wherein the set of distances includes Mahalanobis distances between the first point and each of a plurality of nearest neighbor points in a test point cloud, and determining a first distance by aggregating the set of distances associated with each point in the reference point cloud, and determining a second distance between the test point cloud and the reference point cloud by aggregating each of the first distances associated with each respective point of the reference point cloud.

Description

[0001] Cross-reference to related applications This application claims the benefit of U.S. Provisional Patent Application No. 63 / 597,549, filed November 9, 2023, entitled “AN ADAPTIVE GEOMETRY METRIC FOR 3D POINT CLOUDS”, which is incorporated herein by reference in its entirety.

[0002] By referencing the merging The following applications are incorporated herein by reference in their entirety: U.S. Provisional Patent Application No. 63 / 291,015, filed December 17, 2021, entitled “Hybrid Framework for Point Cloud Compression” (“'015 application”); U.S. Provisional Patent Application No. 63 / 297,869, filed January 10, 2022, entitled “A Scalable Framework for Point Cloud Compression” (“'869 application”); U.S. Provisional Patent Application No. 63 / 388,087, filed July 11, 2022, entitled “A Scalable Framework for Point Cloud Compression” (“'087 application”); and U.S. Provisional Patent Application No. 63 / 388,087, filed July 11, 2022, entitled “Method and Apparatus for Point Cloud Compression Using Hybrid Deep Entropy”. The U.S. Provisional Patent Application No. 63 / 252,482, filed on October 5, 2021, entitled “Coding” (“'482 Application”); the U.S. Provisional Patent Application No. 63 / 297,894, filed on January 10, 2022, entitled “Coordinate Refinement and Upsampling from Quantized Point Cloud Reconstruction” (“'894 Application”); the U.S. Provisional Patent Application No. 63 / 388,600, filed on July 12, 2022, entitled “DeepDistribution-Aware Point Feature Extractor for AI-Based Point Cloud Compression” (“'600 Application”); and the U.S. Provisional Patent Application No. 63 / 388,600, filed on July 12, 2022, entitled “CONTEXT-AWARE VOXEL-BASED UPSAMPLING FOR” (“CONTEXT-AWARE VOXEL-BASED UPSAMPLING FOR”). The U.S. Provisional Patent Application No. 63 / 438,212, filed on January 10, 2023, entitled “POINT CLOUDPROCESSING” (“212 Application”); and the U.S. Provisional Patent Application No. 63 / 417,284, filed on October 18, 2022, entitled “CONTEXT-AWARE VOXEL-BASED UPSAMPLING FOR POINT CLOUDPROCESSING” (“284 Application”). Background Technology

[0003] Point cloud (PC) data format is a common data format across several business areas, including autonomous driving, robotics, augmented reality / virtual reality (AR / VR), civil engineering, computer graphics, and the animation / film industry. 3D LiDAR (light detection and ranging) sensors have already been deployed in autonomous vehicles, and affordable LiDAR sensors have been released from Velodyne Velabit, Apple iPad Pro 2020, and Intel RealSense LiDAR camera L515. With advancements in sensing technology, 3D point cloud data is becoming more practical than ever before. Summary of the Invention

[0004] The embodiments described herein include methods used in video encoding and decoding (collectively, “encoding”).

[0005] A first example method according to some embodiments may include: for each point in a reference point cloud, accessing local covariance; for each point in the reference point cloud, accessing a plurality of nearest neighbor points in a test point cloud; for each point in the reference point cloud, determining a distance set, wherein the distance set includes Mahalanobis distance between a first point in the reference point cloud and a second point among the plurality of nearest neighbor points in the test point cloud; for each point in the reference point cloud, determining a first distance by aggregating the distance set associated with each point in the reference point cloud; and determining a second distance between the test point cloud and the reference point cloud by aggregating each of the first distances associated with each corresponding point in the reference point cloud.

[0006] For some embodiments of the first example method, aggregating the set of distances associated with each point in the reference point cloud includes averaging the set of distances associated with each point in the reference cloud.

[0007] For some embodiments of the first example method, aggregating each of the first distances associated with each corresponding point in the reference point cloud includes: determining the maximum value of the first distances associated with each corresponding point in the reference point cloud.

[0008] For some embodiments of the first example method, aggregating each of the first distances associated with each corresponding point in the reference point cloud includes: averaging each of the first distances associated with each corresponding point in the reference point cloud.

[0009] For some embodiments of the first example method, accessing the local covariance includes one of the following: determining the local covariance; and accessing a predetermined value of the local covariance.

[0010] A second example method according to some embodiments may include, for each point in a reference point cloud, performing: obtaining a local covariance for the point in the reference point cloud; determining a set of distances for the point in the reference point cloud, wherein the set of distances includes Mahalanobis distances between the point in the reference point cloud and each of a plurality of nearest neighbor points in a test point cloud; determining a first distance corresponding to the point in the reference point cloud by aggregating the set of distances associated with the point in the reference point cloud; and determining a second distance between the test point cloud and the reference point cloud by aggregating each of the first distances associated with each corresponding point in the reference point cloud.

[0011] For some embodiments of the second example method, aggregating the set of distances associated with the points in the reference point cloud includes: determining the maximum value of the set of distances associated with the points in the reference point cloud.

[0012] For some embodiments of the second example method, aggregating the set of distances associated with the points in the reference point cloud includes averaging the set of distances associated with the points in the reference point cloud.

[0013] In some embodiments of the second example method, averaging the set of distances associated with the points in the reference point cloud includes averaging the Mahalanobis distances to the set of distances associated with the points in the reference point cloud.

[0014] For some embodiments of the second example method, aggregating each of the first distances associated with each corresponding point in the reference point cloud includes: determining the maximum value of the first distances associated with each corresponding point in the reference point cloud.

[0015] For some embodiments of the second example method, aggregating each of the first distances associated with each corresponding point in the reference point cloud includes: averaging each of the first distances associated with each corresponding point in the reference point cloud.

[0016] In some embodiments of the second example method, obtaining the local covariance includes: determining the local covariance.

[0017] In some embodiments of the second example method, the local covariance of the point in the reference point cloud includes a covariance matrix, which includes the distance between the point in the reference point cloud and each of a plurality of nearest neighbor points in the reference point cloud.

[0018] A third example method according to some embodiments may include: performing a processing loop for each point in a reference point cloud, wherein the processing loop includes: selecting a first point equal to a corresponding point in the reference point cloud for a current traversal of the processing loop; obtaining a local covariance; determining a set of distances, wherein the set of distances includes Mahalanobis distances between the first point and each of a plurality of nearest neighbor points in a test point cloud; determining a first distance by aggregating the set of distances associated with each point in the reference point cloud; and determining a second distance between the test point cloud and the reference point cloud by aggregating each of the first distances associated with each corresponding point in the reference point cloud.

[0019] For some embodiments of the third example method, aggregating the set of distances associated with each point in the reference point cloud includes: determining the maximum value of the set of distances associated with each point in the reference point cloud.

[0020] For some embodiments of the third example method, aggregating the set of distances associated with each point in the reference point cloud includes averaging the set of distances associated with each point in the reference point cloud.

[0021] For some embodiments of the third example method, aggregating each of the first distances associated with each corresponding point in the reference point cloud includes: averaging each of the first distances associated with each corresponding point in the reference point cloud.

[0022] In some embodiments of the third example method, the processing loop is executed for each point in the reference point cloud in a manner that is performed simultaneously for one or more points in the reference point cloud, at least in partial parallelism.

[0023] In some embodiments of the third example method, the processing loop is executed serially for each point in the reference point cloud.

[0024] In some embodiments of the third example method, obtaining the local covariance includes: determining the local covariance.

[0025] For some embodiments of the third example method, the local covariance of the point in the reference point cloud includes a covariance matrix, which includes the distance between the point in the reference point cloud and each of a plurality of nearest neighbor points in the reference point cloud.

[0026] In some embodiments of the third example method, determining the local covariance includes using a covariance matrix corresponding to a plurality of nearest neighbors of the first point.

[0027] In some embodiments of the third example method, determining the covariance matrix includes using the average of multiple nearest neighbors of the first point.

[0028] In some embodiments of the third example method, determining the covariance matrix includes determining a series of differences between the first point and each of a plurality of nearest neighbors of the first point.

[0029] In some embodiments of the third example method, determining the local covariance includes adding a small offset to one or more elements of the covariance matrix.

[0030] Some embodiments of the third example method may further include: using the second distance as a loss function to train a deep neural network, wherein the covariance matrix is ​​determined prior to training the deep neural network.

[0031] Some embodiments of the third example method may further include dynamically changing the second distance during the training of the deep neural network.

[0032] In some embodiments of the third example method, dynamically changing the second distance includes adjusting how many nearest neighbors are in the test point cloud.

[0033] For some embodiments of the third example method, dynamically changing the second distance includes: reducing how many nearest neighbor points are in the test point cloud.

[0034] In some embodiments of the third example method, determining the local covariance includes using the inverse of the covariance matrix corresponding to the plurality of nearest neighbors of the first point.

[0035] Some embodiments of the third example method may further include: determining a second local covariance based on the reference point cloud; determining a third distance based on the second local covariance; and adding the third distance to the second distance to obtain an enhanced distance.

[0036] A third example apparatus according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, operate to cause the apparatus to: perform a processing loop for each point in a reference point cloud, wherein the processing loop includes: selecting a first point equal to a corresponding point in the reference point cloud for a current traversal of the processing loop; obtaining a local covariance; determining a set of distances, wherein the set of distances includes Mahalanobis distances between the first point and each of a plurality of nearest neighbor points in a test point cloud; determining a first distance by aggregating the set of distances associated with each point in the reference point cloud; and determining a second distance between the test point cloud and the reference point cloud by aggregating each of the first distances associated with each corresponding point in the reference point cloud.

[0037] A fourth example method according to some embodiments may include: performing a processing loop for each point in a reference point cloud, wherein the processing loop includes: selecting a first point equal to a corresponding point in the reference point cloud for a current traversal of the processing loop; obtaining a local covariance; determining a set of distances, wherein the set of distances includes the square root of the Mahalanobis distance between the first point and each of a plurality of nearest neighbors in a test point cloud; determining a first distance by aggregating the set of distances associated with each point in the reference point cloud; and determining a second distance between the test point cloud and the reference point cloud by aggregating each of the first distances associated with each corresponding point in the reference point cloud.

[0038] For some embodiments of the fourth example method, aggregating the set of distances associated with each point in the reference point cloud includes: determining the maximum value of the set of distances associated with each point in the reference point cloud.

[0039] For some embodiments of the fourth example method, aggregating the set of distances associated with each point in the reference point cloud includes averaging the set of distances associated with each point in the reference point cloud.

[0040] For some embodiments of the fourth example method, averaging the set of distances associated with each point in the reference point cloud includes averaging over the plurality of nearest neighbor points in the reference point cloud.

[0041] In some embodiments of the fourth example method, determining the local covariance includes using a covariance matrix corresponding to a plurality of nearest neighbors of the first point.

[0042] In some embodiments of the fourth example method, determining the covariance matrix includes using the average of multiple nearest neighbors of the first point.

[0043] In some embodiments of the fourth example method, determining the covariance matrix includes determining a series of differences between the first point and each of a plurality of nearest neighbors of the first point.

[0044] In some embodiments of the fourth example method, determining the local covariance includes adding a small offset to one or more elements of the covariance matrix.

[0045] Some embodiments of the fourth example method may further include: using the second distance as a loss function to train a deep neural network, wherein the covariance matrix is ​​determined prior to training the deep neural network.

[0046] Some embodiments of the fourth example method may further include dynamically changing the second distance during the training of the deep neural network.

[0047] In some embodiments of the fourth example method, dynamically changing the second distance includes adjusting how many nearest neighbors are in the test point cloud.

[0048] For some embodiments of the fourth example method, dynamically changing the second distance includes: reducing how many nearest neighbor points are in the test point cloud.

[0049] In some embodiments of the fourth example method, determining the local covariance includes using the inverse of the covariance matrix corresponding to the nearest neighbors of the first point.

[0050] Some embodiments of the fourth example method may further include: determining a second local covariance based on the reference point cloud; determining a third distance based on the second local covariance; and adding the third distance to the second distance to obtain an enhanced distance.

[0051] A fourth example apparatus according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, operate to cause the apparatus to: perform a processing loop for each point in a reference point cloud, wherein the processing loop includes: selecting a first point equal to a corresponding point in the reference point cloud for a current traversal of the processing loop; obtaining a local covariance; determining a set of distances, wherein the set of distances includes the square root of the Mahalanobis distance between the first point and each of a plurality of nearest neighbors in a test point cloud; determining a first distance by aggregating the set of distances associated with each point in the reference point cloud; and determining a second distance between the test point cloud and the reference point cloud by aggregating each of the first distances associated with each corresponding point in the reference point cloud.

[0052] A fifth example method according to some embodiments may include: for each point in a first point cloud, accessing local covariance; for each point in the first point cloud, accessing a plurality of nearest neighbor points in a second point cloud; for each point in the first point cloud, determining a distance set, wherein the distance set includes Mahalanobis distances between a first point in the first point cloud and a second point among the plurality of nearest neighbor points in the second point cloud; for each point in the first point cloud, determining a first distance by aggregating the distance set associated with each point in the first point cloud; and determining a second distance between the second point cloud and the first point cloud by aggregating each of the first distances associated with each corresponding point in the first point cloud.

[0053] A fifth example apparatus according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, operate to cause the apparatus to: access a local covariance for each point in a first point cloud; access a plurality of nearest neighbor points in a second point cloud for each point in the first point cloud; determine a set of distances for each point in the first point cloud, wherein the set of distances includes Mahalanobis distances between a first point in the first point cloud and a second point among the plurality of nearest neighbor points in the second point cloud; determine a first distance for each point in the first point cloud by aggregating the set of distances associated with each point in the first point cloud; and determine a second distance between the second point cloud and the first point cloud by aggregating each of the first distances associated with each corresponding point in the first point cloud.

[0054] A sixth example apparatus according to some embodiments may include at least one processor configured to perform any of the methods described above.

[0055] A seventh example apparatus according to some embodiments may include a computer-readable medium storing instructions for causing one or more processors to perform any of the methods described above.

[0056] An eighth example apparatus according to some embodiments may include at least one processor and at least one non-transitory computer-readable medium storing instructions for causing the at least one processor to perform any of the methods described above.

[0057] Example signals according to some embodiments may include a bitstream generated according to any of the methods described above.

[0058] In additional embodiments, encoder and decoder devices are provided to perform the methods described herein. The encoder or decoder device may include a processor configured to perform the methods described herein. The device may include a computer-readable medium (e.g., a non-transitory medium) storing instructions for performing the methods described herein. In some embodiments, the computer-readable medium (e.g., a non-transitory medium) stores video encoded using any of the methods described herein.

[0059] One or more of these embodiments also provide a computer-readable storage medium having instructions stored thereon for executing a bidirectional optical stream to encode or decode video data according to any of the methods described above. This embodiment also provides a computer-readable storage medium having a bitstream generated according to the methods described above stored thereon. This embodiment also provides a method and apparatus for transmitting a bitstream generated according to the methods described above. This embodiment also provides a computer program product including instructions for executing any of the described methods. Attached Figure Description

[0060] Figure 1A This is a system diagram illustrating an example communication system according to some embodiments.

[0061] Figure 1B The illustration shows a configuration according to some embodiments. Figure 1A The diagram shows a system diagram of an example wireless transmit / receive unit (WTRU) used in a communication system.

[0062] Figure 1C This is a system diagram illustrating a set of example interfaces for a system according to some embodiments.

[0063] Figure 2A This is a schematic side view illustrating an example waveguide display that can be used with extended reality (XR) applications according to some embodiments.

[0064] Figure 2B This is a schematic side view illustrating an example of an alternative display type that can be used with extended reality applications according to some embodiments.

[0065] Figure 2C This is a schematic side view illustrating an example of an alternative display type that can be used with extended reality applications according to some embodiments.

[0066] Figure 3 This is a schematic diagram illustrating an example estimation of the covariance matrix for a reference point cloud according to some embodiments.

[0067] Figure 4This is a schematic diagram illustrating an example set of contour lines for Mahalanobis distance metric according to some embodiments.

[0068] Figure 5 This is a flowchart illustrating an example process for determining an LMD metric according to some embodiments.

[0069] Figure 6 This is a flowchart illustrating an example process for dynamically training a deep neural network using LMD according to some embodiments.

[0070] Figure 7 This is a flowchart illustrating an example process for determining a local Mahalanobis distance (LMD) metric according to some embodiments.

[0071] As examples, not limitations, are presented the entities, connections, arrangements, etc., depicted in—and described in conjunction with—the various figures. Therefore, any and all statements or other indications concerning what is “depicted” in a particular figure, what a particular element or entity “is” or “has” in a particular figure, and any and all similar statements—which may be interpreted in isolation and out of context as absolute and therefore restrictive—may be properly interpreted only as being preceded by a clause such as “In at least one embodiment, …”. For the sake of brevity and clarity, this implies the unconventional repetition of the preceding clause in the specific embodiments. Detailed Implementation

[0072] Figure 1A This diagram illustrates an example communication system 100 in which one or more of the disclosed embodiments may be implemented. The communication system 100 may be a multiple access system that provides content (such as voice, data, video, messaging, broadcasting, etc.) to multiple wireless users. The communication system 100 enables multiple wireless users to access such content by sharing system resources (including wireless bandwidth). For example, the communication system 100 may employ one or more channel access methods, such as Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal FDMA (OFDMA), Single Carrier FDMA (SC-FDMA), Zero-Tail Unique Word DFT Extended OFDM (ZT UW DTS-s OFDM), Unique Word OFDM (UW-OFDM), Resource Block Filtered OFDM, Filter Bank Multicarrier (FBMC), etc.

[0073] like Figure 1AAs shown, the communication system 100 may include wireless transmit / receive units (WTRUs) 102a, 102b, 102c, 102d, RAN 104 / 113, CN 106, Public Switched Telephone Network (PSTN) 108, Internet 110, and other networks 112. Although it will be appreciated, the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and / or network elements. Each of the WTRUs 102a, 102b, 102c, 102d can be any type of device configured to operate and / or communicate in a wireless environment. As an example, WTRUs 102a, 102b, 102c, and 102d (any of which may be referred to as a “station” and / or “STA”) may be configured to transmit and / or receive wireless signals and may include user equipment (UE), mobile stations, fixed or mobile subscriber units, subscription-based units, pagers, cellular phones, personal digital assistants (PDAs), smartphones, laptops, netbooks, personal computers, wireless sensors, hotspots or Mi-Fi devices, Internet of Things (IoT) devices, watches or other wearable devices, head-mounted displays (HMDs), vehicles, drones, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices operating in industrial and / or automated processing chain scenarios), consumer electronics devices, devices operating on commercial and / or industrial wireless networks, etc. Any of WTRUs 102a, 102b, 102c, and 102d may be interchangeably referred to as a UE.

[0074] The communication system 100 may also include base station 114a and / or base station 114b. Each of base stations 114a and 114b may be any type of device configured to wirelessly interface with at least one of WTRUs 102a, 102b, 102c, and 102d to facilitate access to one or more communication networks, such as CN 106, the Internet 110, and / or other networks 112. As an example, base stations 114a and 114b may be any of a base transceiver station (BTS), Node-B, eNode B, home node B, home eNode B, gNB, NR NodeB, site controller, access point (AP), wireless router, etc. Although base stations 114a and 114b are depicted as single elements, it will be understood that base stations 114a and 114b may include any number of interconnected base stations and / or network elements.

[0075] Base station 114a may be part of RAN 104 / 113, which may also include other base stations and / or network elements (not shown), such as base station controllers (BSCs), radio network controllers (RNCs), relay nodes, etc. Base station 114a and / or base station 114b may be configured to transmit and / or receive radio signals on one or more carrier frequencies, which may be referred to as cells (not shown). These frequencies may be licensed spectrum, unlicensed spectrum, or a combination of licensed and unlicensed spectrum. A cell may provide coverage for a specific geographic area for a radio service, which may be relatively fixed or may change over time. A cell may be further divided into cell sectors. For example, the cell associated with base station 114a may be divided into three sectors. Therefore, in one embodiment, base station 114a may include three transceivers, i.e., one transceiver for each sector of the cell. In one embodiment, base station 114a may employ multiple-input multiple-output (MIMO) technology, and multiple transceivers may be used for each sector of the cell. For example, beamforming can be used to transmit and / or receive signals in a desired spatial direction.

[0076] Base stations 114a and 114b can communicate with one or more of WTRUs 102a, 102b, 102c, and 102d via air interface 116, which can be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, millimeter wave, infrared (IR), ultraviolet (UV), visible light, etc.). Air interface 116 can be established using any suitable radio access technology (RAT).

[0077] More specifically, as noted above, communication system 100 can be a multiple access system and can employ one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, etc. For example, base station 114a in RAN 104 / 113, and WTRUs 102a, 102b, and 102c can implement radio technologies such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which can use Wideband CDMA (WCDMA) to establish the air interface 116. WCDMA can include communication protocols such as High-Speed ​​Packet Access (HSPA) and / or Evolved HSPA (HSPA+). HSPA can include High-Speed ​​Downlink (DL) Packet Access (HSDPA) and / or High-Speed ​​UL Packet Access (HSUPA).

[0078] In one embodiment, base station 114a and WTRUs 102a, 102b, 102c can implement radio technologies such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which can use Long Term Evolution (LTE) and / or Advanced LTE (LTE-A) and / or Advanced LTE Pro (LTE-A Pro) to establish air interface 116.

[0079] In one embodiment, base station 114a and WTRUs 102a, 102b, 102c can implement radio technologies such as NR radio access, which can use a new radio (NR) to establish an air interface 116.

[0080] In one embodiment, base station 114a and WTRUs 102a, 102b, and 102c can implement multiple radio access technologies. For example, base station 114a and WTRUs 102a, 102b, and 102c can jointly implement LTE radio access and NR radio access, for example, using the dual connectivity (DC) principle. Therefore, the air interface utilized by WTRUs 102a, 102b, and 102c can be characterized by multiple types of radio access technologies and / or transmissions sent to / from multiple types of base stations (e.g., eNBs and gNBs).

[0081] In other embodiments, base station 114a and WTRUs 102a, 102b, 102c can implement the following radio technologies, such as IEEE 802.11 (i.e., WiFi), IEEE 802.16 (i.e., WiMAX), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Provisional Standard 2000 (IS-2000), Provisional Standard 95 (IS-95), Provisional Standard 856 (IS-856), Global System for Mobile Communications (GSM), Enhanced Data Rate GSM Evolution (EDGE), GSMEDGE (GERAN), etc.

[0082] Figure 1ABase station 114b can be, for example, a wireless router, a home node B, a home eNode B, or an access point, and can utilize any suitable RAT to facilitate wireless connectivity in a local area, such as a commercial area, home, vehicle, campus, industrial facility, air corridor (e.g., for drone use), road, etc. In one embodiment, base station 114b and WTRUs 102c, 102d can implement radio technologies such as IEEE 802.11 to establish a wireless local area network (WLAN). In one embodiment, base station 114b and WTRUs 102c, 102d can implement radio technologies such as IEEE 802.15 to establish a wireless personal area network (WPAN). In yet another embodiment, base station 114b and WTRUs 102c, 102d can utilize a cellular-based RAT (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR, etc.) to establish a picocell or femtocell. Figure 1A As shown, base station 114b may have a direct connection to Internet 110. Therefore, base station 114b may not be required to access Internet 110 via CN 106.

[0083] RAN 104 / 113 can communicate with CN 106, which can be any type of network configured to provide voice, data, application, and / or Voice over Internet Protocol (VoIP) services to one or more of WTRUs 102a, 102b, 102c, and 102d. Data can have different Quality of Service (QoS) requirements, such as different throughput requirements, latency requirements, error tolerance requirements, reliability requirements, data throughput requirements, mobility requirements, etc. CN 106 can provide call control, billing services, location-based services, prepaid calling, internet connectivity, video distribution, and / or perform advanced security functions, such as user authentication. Although... Figure 1A As not shown, but will be understood, RAN 104 / 113 and / or CN106 can communicate directly or indirectly with other RANs that use the same RAT as or a different RAT than RAN 104 / 113. For example, in addition to being connected to RAN 104 / 113, which can utilize NR radio technology, CN 106 can also communicate with another RAN (not shown) that uses GSM, UMTS, CDMA 2000, WiMAX, E-UTRA, or WiFi radio technology.

[0084] CN 106 may also act as a gateway for WTRUs 102a, 102b, 102c, and 102d to access PSTN 108, the Internet 110, and / or other networks 112. PSTN 108 may include a circuit-switched telephone network providing Common Old-Style Telephone Service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices using common communication protocols such as Transmission Control Protocol (TCP), User Datagram Protocol (UDP), and / or Internet Protocol (IP) from the TCP / IP Internet Protocol suite. Network 112 may include wired and / or wireless communication networks owned and / or operated by other service providers. For example, network 112 may include another CN connected to one or more RANs, which may use the same RAT as RAN 104 / 113 or a different RAT.

[0085] Some or all of the WTRUs 102a, 102b, 102c, and 102d in communication system 100 may include multi-mode capabilities (e.g., WTRUs 102a, 102b, 102c, and 102d may include multiple transceivers for communicating with different wireless networks via different wireless links). For example, Figure 1A The WTRU 102c shown can be configured to communicate with a base station 114a that can use cellular-based radio technology and a base station 114b that can use IEEE 802 radio technology.

[0086] Figure 1B This is a system diagram illustrating example WTRU 102. (Example:) Figure 1B As shown, WTRU 102 may include a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a keypad 126, a display / touchpad 128, non-removable memory 130, removable memory 132, a power supply 134, a global positioning system (GPS) chipset 136, and / or other peripherals 138, etc. It will be appreciated that WTRU 102 may include any sub-combination of the above-described elements while remaining consistent with the embodiments.

[0087] Processor 118 may be a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc. Processor 118 may perform signal encoding, data processing, power control, input / output processing, and / or any other functions that enable WTRU 102 to operate in a wireless environment. Processor 118 may be coupled to transceiver 120, which may be coupled to transmitting / receiving element 122. Although... Figure 1B The processor 118 and transceiver 120 are depicted as separate components, but it will be understood that the processor 118 and transceiver 120 can be integrated together in an electronic package or chip.

[0088] Transmitting / receiving element 122 can be configured to transmit signals to or receive signals from a base station (e.g., base station 114a) via air interface 116. For example, in one embodiment, transmitting / receiving element 122 can be an antenna configured to transmit and / or receive RF signals. In one embodiment, transmitting / receiving element 122 can be a transmitter / detector configured to transmit and / or receive, for example, IR, UV, or visible light signals. In yet another embodiment, transmitting / receiving element 122 can be configured to transmit and / or receive both RF and optical signals. It will be appreciated that transmitting / receiving element 122 can be configured to transmit and / or receive any combination of wireless signals.

[0089] Although the transmitting / receiving element 122 is in Figure 1B While depicted as a single element, WTRU 102 may include any number of transmitting / receiving elements 122. More specifically, WTRU 102 may employ MIMO technology. Thus, in one embodiment, WTRU 102 may include two or more transmitting / receiving elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals via air interface 116.

[0090] Transceiver 120 can be configured to modulate signals to be transmitted by transmitting / receiving element 122 and demodulate signals received by transmitting / receiving element 122. As noted above, WTRU 102 can have multi-mode capability. Thus, for example, transceiver 120 may include multiple transceivers for enabling WTRU 102 to communicate via multiple RATs (such as NR and IEEE 802.11).

[0091] The processor 118 of WTRU 102 can be coupled to the speaker / microphone 124, keypad 126, and / or display / touchpad 128 (e.g., a liquid crystal display (LCD) unit or an organic light-emitting diode (OLED) display unit), and can receive user input data from them. The processor 118 can also output user data to the speaker / microphone 124, keypad 126, and / or display / touchpad 128. Additionally, the processor 118 can access information from any type of suitable memory (such as non-removable memory 130 and / or removable memory 132), and store data in that memory. Non-removable memory 130 may include random access memory (RAM), read-only memory (ROM), hard disk, or any other type of memory storage device. Removable memory 132 may include a subscriber identity module (SIM) card, memory stick, secure digital storage (SD) card, etc. In other embodiments, processor 118 may access information from memory that is not physically located on WTRU 102 (such as on a server or home computer (not shown)) and store data in that memory.

[0092] The processor 118 can receive power from the power supply 134 and can be configured to distribute and / or control the power going to other components in the WTRU 102. The power supply 134 can be any suitable device for powering the WTRU 102. For example, the power supply 134 may include one or more dry cell batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, etc.

[0093] The processor 118 may also be coupled to a GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) about the current location of the WTRU 102. In addition to, or instead of, information from the GPS chipset 136, the WTRU 102 may receive location information from base stations (e.g., base stations 114a, 114b) via air interface 116 and / or determine its location based on the timing of signals received from two or more nearby base stations. It will be understood that the WTRU 102 may acquire location information using any suitable location determination method, while remaining consistent with the embodiments.

[0094] The processor 118 may be further coupled to other peripherals 138, which may include one or more software and / or hardware modules providing additional features, functions, and / or wired or wireless connectivity. For example, peripherals 138 may include accelerometers, electronic compasses, satellite transceivers, digital cameras (for photos and / or videos), Universal Serial Bus (USB) ports, vibration devices, television transceivers, hands-free headsets, Bluetooth® modules, FM radio units, digital music players, media players, video game player modules, internet browsers, virtual reality and / or augmented reality (VR / AR) devices, activity trackers, etc. Peripherals 138 may include one or more sensors, which may be one or more of the following: gyroscopes, accelerometers, Hall effect sensors, magnetometers, orientation sensors, proximity sensors, temperature sensors, time sensors, geolocation sensors, altimeters, light sensors, touch sensors, magnetometers, barometers, gesture sensors, biometric sensors, and / or humidity sensors.

[0095] WTRU 102 may include a full-duplex radio, for which the transmission and reception of some or all signals (e.g., associated with specific subframes for both UL (e.g., for transmission) and downlink (e.g., for reception)) may be concurrent and / or simultaneous. The full-duplex radio may include an interference management unit to reduce and / or substantially eliminate self-interference via hardware (e.g., a choke) or via signal processing (e.g., a separate processor (not shown) or via processor 118). In one embodiment, WTRU 102 may include a half-duplex radio, for which the transmission and reception of some or all signals (e.g., associated with specific subframes for UL (e.g., for transmission) or downlink (e.g., for reception)) may be concurrent and / or simultaneous.

[0096] Despite WTRU in Figures 1A to 1B While described as a wireless terminal, it is envisioned that, in some representative embodiments, such a terminal may use (e.g., temporarily or permanently) a wired communication interface with a communication network.

[0097] In a representative embodiment, the other network 112 may be a WLAN.

[0098] Given Figures 1A to 1B The description, along with its corresponding information, indicates that one or more of the functions described herein may be performed by one or more emulation devices (not shown). An emulation device may be one or more devices configured to emulate one or more of the functions described herein. For example, an emulation device may be used to test other devices and / or simulate network and / or WTRU functions.

[0099] Simulation devices can be designed to perform tests on one or more other devices in a laboratory environment and / or a carrier network environment. For example, the one or more simulation devices can perform one or more or all of their functions while being fully or partially implemented and / or deployed as part of a wired and / or wireless communication network to test other devices within the communication network. The one or more simulation devices can perform one or more or all of their functions while being temporarily implemented / deployed as part of a wired and / or wireless communication network. Simulation devices can be directly coupled to another device for testing purposes and / or can use over-the-air wireless communication to perform tests.

[0100] The one or more simulation devices can perform one or more (including all) functions without being implemented / deployed as part of a wired and / or wireless communication network. For example, the simulation devices can be used in test scenarios in a test laboratory and / or in non-deployed (e.g., testing) wired and / or wireless communication networks to perform testing on one or more components. The one or more simulation devices can be test rigs. Direct RF coupling and / or wireless communication via RF circuitry (e.g., which may include one or more antennas) can be used by the simulation devices to transmit and / or receive data.

[0101] Figure 1C This is a system diagram illustrating an example set of interfaces for a system according to some embodiments. In some embodiments, an extended reality display device may be implemented together with its control electronics. System 150 may be embodied as a device including the various components described below and configured to perform one or more of the aspects described in this document. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Elements of system 150 may be embodied individually or in combination in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 150 are distributed across multiple ICs and / or discrete components. In various embodiments, system 150 is communicatively coupled to one or more other systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports. In various embodiments, system 150 is configured to implement one or more of the aspects described in this document.

[0102] System 150 includes: at least one processor 152 configured to execute instructions loaded therein for implementing various aspects, such as those described in this document. Processor 152 may include embedded memory, input / output interfaces, and various other circuitry as known in the art. System 150 includes at least one memory 154 (e.g., a volatile memory device and / or a non-volatile memory device). System 150 may include: a storage device 158 which may include non-volatile memory and / or volatile memory, including but not limited to electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, disk drives, and / or optical disk drives. Storage device 158 may include internal storage devices, attached storage devices (including removable and non-removable storage devices), and / or network-accessible storage devices, as non-limiting examples.

[0103] System 150 includes an encoder / decoder module 156 configured to, for example, process data to provide encoded or decoded video, and the encoder / decoder module 156 may include its own processor and memory. The encoder / decoder module 156 represents one or more modules that may be included in a device to perform encoding and / or decoding functions. It is well known that a device may include one or both encoding and decoding modules. Furthermore, the encoder / decoder module 156 may be implemented as a separate element of system 150, or may be incorporated into processor 152 as a combination of hardware and software as known to those skilled in the art.

[0104] Program code to be loaded onto processor 152 or encoder / decoder 156 to execute the various aspects described herein may be stored in storage device 158 and subsequently loaded onto memory 154 for execution by processor 152. According to various embodiments, one or more of processor 152, memory 154, storage device 158, and encoder / decoder module 156 may store one or more items of various kinds during the execution of the processes described herein. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0105] In some embodiments, the memory within processor 152 and / or encoder / decoder module 156 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, external memory (e.g., processor 152 or encoder / decoder module 152) is used for one or more of these functions. External memory may be memory 154 and / or storage device 158, such as volatile memory and / or non-volatile flash memory. In several embodiments, external non-volatile flash memory is used to store, for example, the operating system of a television. In at least one embodiment, a fast external dynamic volatile memory, such as RAM, is used as working memory for video encoding and decoding operations, such as for MPEG-2 (MPEG stands for Moving Picture Experts Group, MPEG-2 is also known as ISO / IEC 13818, and 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC stands for High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Universal Video Coding: a new standard developed by the Joint Video Experts Team JVET)).

[0106] Inputs to the components of system 172 can be provided through various input devices as indicated in box 150. Such input devices include, but are not limited to: (i) a radio frequency (RF) section that receives RF signals transmitted over the air, for example, by a broadcaster; (ii) component (COMP) input terminals (or a collection of COMP input terminals); (iii) a universal serial bus (USB) input terminal; and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Figure 1C Other examples not shown include composite video.

[0107] In various embodiments, the input device of block 172 has associated corresponding input processing elements as known in the art. For example, the RF section may be associated with elements suitable for: (i) selecting a desired frequency (also referred to as selecting a signal or limiting a signal to a frequency band); (ii) down-converting the selected signal; (iii) further limiting the frequency band to a narrower band to select, for example, a signal band that may be referred to as a channel in some embodiments; (iv) demodulating the down-converted and band-limited signal; (v) performing error correction; and (vi) demultiplexing to select the desired stream of data packets. The RF section in various embodiments includes one or more elements to perform these functions, such as frequency selectors, signal selectors, band limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF section may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., intermediate frequency or near-baseband frequency) or to baseband. In one set-top box embodiment, the RF section and its associated input processing elements receive RF signals transmitted over a wired (e.g., cable) medium and perform frequency selection by filtering, down-converting, and filtering again to the desired frequency band. Various embodiments rearrange the order of the components described above (and others), remove some of these components, and / or add other components that perform similar or different functions. Adding components may include inserting components between existing components, such as, for example, inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0108] Additionally, the USB and / or HDMI endpoints may include corresponding interface processors for connecting system 150 to other electronic devices across USB and / or HDMI connections. It should be understood that various aspects of input processing, such as Reed-Solomon error correction, may be implemented, for example, within a separate input processing IC or, if necessary, within processor 152. Similarly, aspects of USB or HDMI interface processing may be implemented, either within a separate interface IC or, if necessary, within processor 152. The demodulated, error-corrected, and demultiplexed stream is provided to various processing elements, including, for example, processor 152 and encoder / decoder 156, which operate in conjunction with memory and storage elements to process the data stream as needed for presentation on an output device.

[0109] Various components of system 150 can be provided within an integrated housing, where the components can be interconnected and data can be transmitted therebetween using a suitable connection arrangement 174, such as an internal bus as known in the art, including inter-IC (I2C) bus, wiring, and printed circuit board.

[0110] System 150 includes a communication interface 160, which enables communication with other devices via a communication channel 162. The communication interface 160 may include, but is not limited to, a transceiver configured to transmit and receive data on the communication channel 162. The communication interface 160 may include, but is not limited to, a modem or network interface card (NIC), and the communication channel 162 may be implemented over, for example, wired and / or wireless media.

[0111] In various embodiments, data is streamed or otherwise provided to system 150 using a wireless network such as WiFi (e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers)). In these embodiments, the Wi-Fi signal is received on a communication channel 162 and a communication interface 160 adapted for Wi-Fi communication. The communication channel 162 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other over-the-top communications. Other embodiments use a set-top box that delivers data over an HDMI connection in input box 172 to provide streaming data to system 150. Still other embodiments use an RF connection in input box 172 to provide streaming data to system 150. As indicated above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, such as cellular networks or Bluetooth networks.

[0112] System 150 can provide output signals to various output devices, including display 176, speaker 178, and other peripheral devices 180. Display 176 in various embodiments includes one or more of the following: for example, a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. Display 176 can be used in televisions, tablets, laptops, cellular phones (mobile phones), or other devices. Display 176 can also be integrated with other components (e.g., as in smartphones) or separate (e.g., an external monitor for a laptop). In various examples of embodiments, other peripheral devices 180 include one or more of a standalone digital video disc (or digital multifunction disc) (DVR, for both terms), disc player, stereo system, and / or lighting system. Various embodiments use one or more peripheral devices 180 that provide functionality based on the output of system 150. For example, a disc player performs the function of playing the output of system 150.

[0113] In various embodiments, signaling such as AV is used to transmit control signals between system 150 and display 176, speaker 178, or other peripheral devices 180. Device-to-device control links, consumer electronics control (CEC), or other communication protocols are implemented with or without user intervention. Output devices can be communicatively coupled to system 150 via dedicated connections through corresponding interfaces 164, 166, and 168. Alternatively, output devices can be connected to system 150 via communication interface 160 using communication channel 162. Display 176 and speaker 178 can be integrated into a single unit with other components of system 150 in electronic devices such as, for example, televisions. In various embodiments, display interface 164 includes a display driver, such as, for example, a timing controller (TCon) chip.

[0114] Display 176 and speaker 178 can alternatively be separated from one or more other components, for example, if the RF section of input 172 is part of a separate set-top box. In various embodiments where display 176 and speaker 178 are external components, output signals can be provided via dedicated output connections, including, for example, HDMI ports, USB ports, or COMP outputs.

[0115] System 150 may include one or more sensor devices 168. Examples of usable sensor devices include one or more GPS sensors, gyroscope sensors, accelerometers, light sensors, cameras, depth sensors, microphones, and / or magnetometers. Such sensors can be used to obtain information such as the user's position and orientation. Where system 150 is used as a control module (such as control modules 124, 132) for an extended reality display, the user's position and orientation can be used to determine how image data is presented, so that the user perceives the correct portion of a virtual object or scene from the correct viewpoint. In the case of a head-mounted display device, the device's own position and orientation can be used to determine the user's position and orientation for the purpose of presenting virtual content. In the case of other display devices such as telephones, tablets, computer monitors, or televisions, other inputs can be used to determine the user's position and orientation for the purpose of presenting content. For example, a user can select and / or adjust the desired viewpoint and / or viewing direction using a touchscreen, keypad or keyboard, trackball, joystick, or other inputs. When the display device has sensors such as accelerometers and / or gyroscopes, the viewpoint and orientation used for the purpose of presenting content can be selected and / or adjusted based on the movement of the display device.

[0116] The embodiments may be implemented by computer software, hardware, or a combination of hardware and software, as implemented by processor 152. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. As a non-limiting example, memory 154 may be of any type suitable for the technical environment and may be implemented using any suitable data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. As a non-limiting example, processor 152 may be of any type suitable for the technical environment and may encompass one or more of microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures.

[0117] Figure 2A This is a schematic side view illustrating an example waveguide display that can be used with extended reality (XR) applications according to some embodiments. The image is projected by an image generator 202. The image generator 202 can use one or more of a variety of techniques for projecting images. For example, the image generator 202 can be a laser beam scanning (LBS) projector, a liquid crystal display (LCD), a light-emitting diode (LED) display (including organic LED (OLED) or micro LED (μLED) displays), a digital light processor (DLP), a liquid crystal on silicon (LCoS) display, or other types of image generators or light engines.

[0118] Light representing image 212 generated by image generator 202 is coupled into waveguide 204 via diffraction coupler 206. Coupler 206 diffracts the light representing image 212 into one or more diffraction orders. For example, ray 208, which is one of the rays representing the bottom portion of the image, is diffracted by coupler 206, and one of the diffraction orders 210 (e.g., second order) is at an angle that can propagate through waveguide 204 by total internal reflection. Image generator 202 displays the image as directed by control module 224, which operates to render image data, video data, point cloud data, or other displayable data.

[0119] At least a portion of the light 210, which has been coupled into waveguide 204 by diffraction coupler 206, is coupled out of the waveguide by diffraction coupler 214. At least some of the light coupled out of waveguide 204 replicates the angle of incidence of the light coupled into the waveguide. For example, in the illustration, the coupled rays 216a, 216b, and 216c replicate the angle of the coupled ray 208. Because the light leaving the coupler replicates the direction of the light entering the coupler, the waveguide essentially replicates the original image 212. The user's eye 218 can focus on the replicated image.

[0120] exist Figure 2AIn the example, the output 214 couples out only a portion of the light, where each reflection allows a single input beam (e.g., beam 208) to generate multiple parallel output beams (e.g., beams 216a, 216b, and 216c). In this way, at least some of the light originating from each part of the image is likely to reach the user's eye, even if the eye is not perfectly aligned with the center of the output. For example, if the eye 218 moves downward, beam 216c can enter the eye, even if beams 216a and 216b do not, so the user can still perceive the bottom of image 212 despite the positional shift. The output 214 thus partially operates as an exit pupil expander in the vertical direction. The waveguide may also include one or more additional exit pupil expanders ( Figure 2A (Not shown in the image) to expand the exit pupil in the horizontal direction.

[0121] In some embodiments, waveguide 204 is at least partially transparent to light originating outside the waveguide display. For example, at least some of the light 220 from a real-world object (such as object 222) passes through waveguide 204, allowing the user to see the real-world object while using the waveguide display. As the light 220 from the real-world object also passes through diffraction grating 214, multiple diffraction orders will exist, and therefore multiple images will exist. To minimize the visibility of multiple images, it is desirable that diffraction order 0 (not deflected by 214) has high diffraction efficiency for light 220 and order 0, while higher diffraction orders are lower in energy. Therefore, in addition to expanding and coupling out virtual images, coupler 214 is also preferably configured to allow order 0 of the real-world image to pass through. In this embodiment, the image displayed by the waveguide display can appear as an overlay on the real world.

[0122] Figure 2B This is a schematic side view illustrating an example of an alternative display type that can be used with extended reality applications according to some embodiments. In an XR head-mounted display device 230, a control module 232 controls a display 234 to display an image; the display 234 may be an LCD. The head-mounted display includes a partially reflective surface 236 that reflects (and in some embodiments, both reflects and focuses) the image displayed on the LCD to make the image visible to the user. The partially reflective surface 236 also allows at least some external light to pass through, thereby allowing the user to see their surroundings.

[0123] Figure 2C This is a schematic side view illustrating an example of an alternative display type that can be used with extended reality applications according to some embodiments. In an XR head-mounted display device 240, a control module 242 controls a display 244, which may be an LCD, to display an image. The image is focused by one or more lenses of a display optics 246 to make the image visible to the user. Figure 2CIn the image, external light does not directly reach the user's eyes. However, in some such embodiments, an external camera 248 can be used to capture an image of the external environment and display this image on a display 244 along with any virtual content that may also be displayed.

[0124] The embodiments described herein are not limited to any particular type or structure of XR display device.

[0125] This application discusses point cloud compression and processing. The aim of this field is to develop tools for the compression, analysis, interpolation, representation, and understanding of point cloud signals.

[0126] Point cloud data format Point cloud (PC) data format is a common data format across several business areas, including autonomous driving, robotics, augmented reality / virtual reality (AR / VR), civil engineering, computer graphics, and the animation / film industry. 3D LiDAR (light detection and ranging) sensors have already been deployed in autonomous vehicles, and affordable LiDAR sensors have been released from Velodyne Velabit, Apple iPad Pro 2020, and Intel RealSense LiDAR camera L515. With advancements in sensing technology, 3D point cloud data is becoming more practical than ever before.

[0127] Point cloud data can consume a significant portion of network traffic, for example, in cars connected to 5G networks and in immersive communications (VR / AR). Efficient representation formats are essential for point cloud understanding and communication. Specifically, raw point cloud data can be appropriately organized and processed for world modeling and sensing purposes. Compression of the raw point cloud can be used when data storage and transmission are required in the relevant scenario.

[0128] Furthermore, point clouds can represent sequential scans of the same scene, which may contain multiple moving objects. These are called dynamic point clouds in contrast to static point clouds captured from static scenes or static objects. Dynamic point clouds are typically organized into frames, with different frames captured at different times. Dynamic point clouds may require processing and compression to be real-time or with low latency.

[0129] Point cloud data usage The automotive industry and autonomous vehicles are among the sectors where point clouds can be used. Autonomous vehicles should be able to "detect" their environment to make sound driving decisions based on the reality of their immediate surroundings. Typical sensors like LiDAR generate (dynamic) point clouds used by perception engines. These point clouds are not intended for human viewing, and they are typically sparse, not necessarily colored, and dynamic at a high capture frequency. They can possess other properties, such as reflectivity provided by LiDAR, as this indicates the material of the sensed object and can aid in decision-making.

[0130] Virtual reality (VR) and immersive worlds are envisioned by many as the future of 2D flat video. In VR and immersive worlds, the viewer is immersed in an environment surrounding them, contrasting with standard TVs where the viewer can only see a virtual world in front of them. Several levels of immersion exist, depending on the viewer's freedom within the environment. Point clouds are a good candidate format for distributing VR worlds. Point clouds used in VR can be static or dynamic and are typically of average size, for example, no more than a few million points at a time.

[0131] Point clouds can also be used for a variety of purposes, such as cultural heritage / buildings, where objects like sculptures or buildings are scanned in 3D to share their spatial configuration without sending or accessing them. Furthermore, point clouds can be used to preserve knowledge of objects in the event that they may be destroyed (e.g., a temple destroyed by an earthquake). Such point clouds are typically static, colored, and large.

[0132] Another use case is in topography and cartography, where 3D representation is used, and maps are not limited to flat surfaces and can include reliefs. Google Maps is a good example of a 3D map, but it uses a grid instead of point clouds. However, point clouds can be a suitable data format for 3D maps, and such point clouds are typically static, colored, and large.

[0133] World modeling and sensing via point clouds can be a useful technique that allows machines to acquire knowledge about the 3D world around them for the applications discussed in this paper.

[0134] 3D point cloud data can be discrete samples of the surface of an object or scene. Such point clouds have several characteristics. Unlike 2D images, 3D points in a point cloud are unorganized. Second, depending on how the point cloud is retrieved, the points can be distributed in very different ways, ranging from very dense (e.g., point clouds used for VR / AR / games) to very sparse (e.g., LiDAR point clouds). Even within the same point cloud, the point distribution can vary greatly. For example, in a LiDAR point cloud, there are often more points closer to the LiDAR sensor. Furthermore, to fully represent the real world with point samples, such point clouds actually require a huge number of points. For example, a typical VR immersive scene contains millions of points, while a point cloud typically contains hundreds of millions.

[0135] Geometric metrics can be used to compare different point clouds. Considering two point clouds, such as a test point cloud and a reference point cloud, the metric can output a number indicating the geometric similarity between the two input point clouds. This metric can be used as an objective measure to assess the performance of point cloud processing algorithms. Second, considering the deep neural network that generates the point clouds (e.g., a point cloud compression (PCC) network), the metric can be used as a loss function to compute the loss value between the reconstructed / test point cloud and the reference point cloud for training the neural network.

[0136] However, due to the characteristics of the point cloud format mentioned above, creating an effective geometric metric to compare two different point clouds can be difficult.

[0137] As a popular 3D data format, point clouds typically consist of discrete samples of the surfaces of objects or scenes. Geometric metrics can be used to evaluate point cloud quality for assessment of point cloud processing / compression algorithms or for loss calculation in deep neural network training. However, creating such metrics can be challenging due to the specific characteristics of point cloud formats. This work specifically focuses on this problem of designing similarity metrics for point clouds.

[0138] Geometric metrics can be used to evaluate the quality of a test point cloud while taking into account a reference point cloud. Such metrics can respect the local geometry of the point cloud while remaining computationally efficient. Before delving into the design of such metrics, we will review several popular geometric metrics for point clouds.

[0139] Chamfer distance (CD) Chamfer distance (CD) is a popular metric for comparing point clouds. See Fan, Haoqiang, et al. A Point Set Generation Network for 3D Object Reconstruction from a Single Image, Proceedings of the IEEE Conf. on Comp. Vision and Pattern Recognition(2017) (“ Fan Assume the 3D point clouds being compared are X and Y, where X contains... n There are points and Y contains m There are several points. The chamfer distance between X and Y is shown in Equation 1: Where x and y are 3D points in X and Y, respectively. For each point in X and Y, the CD algorithm finds the nearest neighbor from the other point cloud, calculates the distance, and sums the distances.

[0140] Using chamfer distance (CD) as the loss function in training deep neural networks can be problematic. In particular, neural networks trained using chamfer distance often output point clouds with highly imbalanced densities; for example, some regions may have much higher densities than others. (Achlioptas, Panos, et al.) Learning Representations and Generative Models for 3D Point Clouds , Intern'l Conf. on Mach. Learning, PMLR(2018).

[0141] Bulldozer Distance (EMD) Besides CD, another popular metric is bulldozer distance (EMD). See also Fan and Rubner, Yossi, et al., The Earth Mover's Distance as a Metric for Image Retrieval , 40 Intern'lJ. Comp. Vision 99-121 (2000). Assume point clouds X and Y have the same number of points. The EMD between X and Y is shown in Equation 2: .

[0142] The EMD metric seeks a one-to-one mapping between points in X and Y with the shortest aggregation distance. When applied to point clouds, the EMD distance is understood to have two constraints: the number of points in X and the number of points in Y must be the same. Furthermore, calculating the EMD distance is computationally very expensive. These issues limit the practical use of the EMD distance as a geometric metric or as a loss function in deep neural network training.

[0143] Distance from point to plane Although CD measures the point-to-point distance between two comparative point clouds, the point-to-plane distance, as its name suggests, measures the average point-to-plane distance by utilizing the surface normal of the reference point cloud. (Tian, ​​Dong, et al.) Geometric Distortion Metrics for Point Cloud Compression , 2017 IEEE Intern'lConf. on ImageProc. (ICIP), IEEE (2017). Assume point cloud X is the reference point cloud (which can be called the ground truth point cloud) and point cloud Y is the test point cloud (which can be called the reconstructed point cloud). The distance from a point to a plane can be expressed as shown in Equation 3: Where n x Let X be the surface normal vector at point x. The distance from a point to the plane passes through every point on the ground truth X, and the point in Y with the minimum projected distance is found based on the surface normal. However, the surface normal of X may not be available and may need to be estimated. Furthermore, in some cases, the normal vector may not even be explicitly defined. In particular, there is no normal vector for line or curve structures.

[0144] A geometric metric can be used to evaluate the quality of a test point cloud Y while taking into account a reference point cloud X. This metric will be called Local Mahalanobis Distance (LMD). By utilizing the local covariance of the reference point cloud, this metric adaptively modifies the distance calculation, respecting the local geometry of the reference point cloud. This metric is more computationally efficient for training deep neural networks. Moreover, it yields trained networks that can be used to reconstruct higher-quality point clouds.

[0145] Local Mahalanobis distance (LMD) Figure 3 This is a schematic diagram illustrating an example estimation of the covariance matrix for a reference point cloud according to some embodiments. Considering that... n The reference point cloud X (302) and the reference point cloud with points ... m If there are 1 test point cloud Y, then the LMD distance can be used to evaluate the quality of the test point cloud Y.

[0146] For each point in X, a 3x3 covariance matrix S can be calculated. i (308), of which i = 0, 1, 2, …, ( n -1), to describe the local geometry, where once again, n This is the number of points in X. Example procedure 300 is... Figure 3 As shown in the diagram. To obtain the result for point x... i covariance matrix S i , point x i of K The nearest neighbor search of the nearest neighbors forms a set of points. In the middle. And x i The associated estimated covariance matrix S i As given by Equation 4: .

[0147] In some embodiments, In some embodiments, , it is M i The average value of the points in the matrix.

[0148] For each point x in the reference point cloud X i For the test point cloud Y k The search is performed using the nearest neighbor. (The last part, "with point x," appears to be a typo and can be omitted i The set of neighboring points in the associated Y is denoted as N i The LMD between X and Y can be determined as shown in Equation 5: Where y is from x i The points in the neighborhood of and It is x i The Mahalanobis distance between x and y. i The Mahalanobis distance between y and y can be expressed as shown in Equation 6: Where "T" refers to the transpose operation, and It is the covariance matrix S i The inverse matrix of X. The calculation of LMD can be broken down into two steps, as shown in the first equality in Equation 5. For each point x in X. i Determine x i With neighboring regions N i All of Y in k The Mahalanobis distance between points, followed by a pair k The average of the distance values ​​can be calculated. Figure 3 This is part of the PCA process 306 shown. For all x from X... i And to calculate the average Mahalanobis distance The average is then calculated to obtain the LMD between X and Y. In some embodiments, instead of averaging the distance values ​​for aggregation in the above two steps, the maximum value among the distances is used. In this case, the LMD metric is as shown in Equation 7: .

[0149] Figure 4 This is a schematic diagram illustrating an example set of contour lines for a Mahalanobis distance metric according to some embodiments. Consider the reference point cloud X402 at position x i Local geometry at x, Mahalanobis distance measurement point iThe distance between point y and point y. For illustration purposes, Figure 4 It shows the point A and B The environment 400 has contour lines 404 and 406 with Mahalanobis distance. The Mahalanobis distance contour lines 404 and 406 are respectively along... A and B An ellipse oriented at the surface. In this way, with A or B Points with the same Euclidean distance can have very different Mahalanobis distances, depending on how close the point is to the surface. For Figure 4 Example scenario 400, C and D Both have with A The same Euclidean distance. However, the point D Will get and A The Mahalanobis distance is much smaller. (Line segment) It intersects with only one contour line because the point D Comparison point C It's closer to the surface of a point cloud. In other words, points... C With A The Mahalanobis distance is much larger because of the line segment It intersects with four contour lines at 404. The reason for this relationship is the point... C Comparison point D Further away from the point cloud surface. In this way, Mahalanobis distance is used to calculate the distance. A Distance adhered to A The local surface geometry at that location.

[0150] Figure 5 This is a flowchart illustrating an example process for determining the LMD metric according to some embodiments. The steps used to reflect the calculation of the LMD using Equation 5 are... Figure 5 Example procedure 500 is shown. Constructing the 502 covariance matrix S. i And construct 504 for each point x in X. i nearest neighbor set N i Using S i To calculate 506x i and N i Mahalanobis distance between points d M The set, which yields for each point x in the reference point cloud X. i distance set D i For each point x in X i By analyzing the distance set Di The average distance between points X and Y is used to calculate the distance from point X to Y. Finally, for point X in X... i The average of the associated distance values ​​is 510, which yields the LMD value between X and Y. .

[0151] In some embodiments, LMD can be written as shown in Equation 8: Compared to Equation 5, Equation 8 adds the application of the square root. .

[0152] In some embodiments, the covariance matrix S i Versions that can be modified The alternative is shown in Equation 9: Where I is a 3x3 identity matrix and ε It is a small normal quantity. (The term is missing from the original text.) ε Adding I to the covariance matrix makes the covariance matrix more stable during matrix inversion, where ε It is a small positive real number, such as 0.001.

[0153] LMD can be applied as a loss function to train a deep neural network for point cloud generation. In this case, the covariance matrix S associated with the reference point cloud (from the training dataset) i It can be pre-computed before training. In this way, there is no need to compute the covariance matrix on the fly during training, which reduces computational costs.

[0154] Enhanced LMD Equation 5 iterates over each point in the reference point cloud X. However, the calculation of Equation 5 may not cover all points in the test point cloud Y, especially those points in Y that are far from X. Therefore, some points in Y may not be considered in the evaluation. To address this issue, an additional term can be added to the LMD metric shown in Equation 5, which iterates over the points in the test point cloud Y.

[0155] For each point in the test point cloud Y y j Execute targeting from reference point cloud X k (tst) The search for the nearest neighbor in X. y j The set of related neighboring points is denoted as The second term can be computed from the perspective of the test point cloud Y, and this second term can be included in the LMD function. The updated LMD is shown in Equation 10: .

[0156] For equation 10 Points in x Using indexing functions ID ( x To track the indices of points in point cloud X. For example, ID (x i ) = i Therefore, S ID(x) It is related to 3D points x The associated covariance matrix. In fact, the numbers... k and k (tst) These can be different numbers, for example k = 7 and k (tst) = 1.

[0157] In the newly added second item, instead of estimating a new set of covariance matrices based on the test point cloud Y, the covariance matrix S obtained from the reference point cloud X is reused. i To calculate the Mahalanobis distance. The covariance matrix S estimated from X. i They reflect the true statistics of the reference point cloud X, so they are reused. They can be reliably used to measure or quantify the quality of the test point cloud Y.

[0158] Dynamic LMD Figure 6 This is a flowchart illustrating an example process for dynamically training a deep neural network using LMD, according to some embodiments. Figure 6 In this process, LMD is dynamically modified for better geometric detail reconstruction.

[0159] In some embodiments, LMD can be applied as a loss function for training deep neural networks. The parameter K for nearest neighbor search plays a crucial role when constructing the covariance matrix for LMD computation. If K is a large number, the covariance matrix represents the geometry of a larger region of the reference point cloud. In this first scenario, LMD is more suitable for training deep neural networks to reconstruct a coarse point cloud shape. In other words, if K is a small number, the covariance matrix represents only a very local region. In this second scenario, LMD is more suitable for training deep neural networks to reconstruct finer details.

[0160] Considering this behavior, the LMD loss can be dynamically changed during neural network training, which makes the numbers... K Gradually decrease. The LMD based on the covariance of its K nearest neighbors is labeled as LMD. K The overall training process is divided into... tThe step, in which the first step i At each training step, execute the targeting e i Neural network training in epochs. i = 1, 2,…, t .

[0161] For example process 600, it can be targeted at e One cycle of use Training the 602 network, where K 1 is relatively large. At the beginning of training, the network can learn how to roughly reconstruct the point cloud. This can be targeted at... e 2 cycles of utilization Training a 604 network, where K 2< K 1. Generally, in the first i At each training step, it can be targeted e i Utilization of each cycle Training a 606 neural network, where K i < K i-1 In the final step (the... t (Step 1) can be used to target e t Utilization of each cycle Train a 608-bit neural network. Deep neural networks can gradually learn how to reconstruct the geometric details of a point cloud.

[0162] Figure 7This is a flowchart illustrating an example process for determining a local Mahalanobis distance (LMD) metric according to some embodiments. In some embodiments, example process 700 may include: performing a processing loop for each point in a reference point cloud. In some embodiments, example process 700 may further include: selecting 702 a first point equal to a corresponding point in the reference point cloud for the current traversal of the processing loop. In some embodiments, example process 700 may further include: obtaining 704 the local covariance. In some embodiments, example process 700 may further include: determining 706 a distance set, wherein the distance set includes Mahalanobis distances between the first point and each of a plurality of nearest neighbors in the test point cloud. In some embodiments, example process 700 may determine 708 whether the processing loop has more traversals to complete. If the processing loop has more traversals to complete, control returns to complete the next traversal; otherwise, the processing loop exits. In some embodiments, example process 700 may further include: determining 710 a first distance by aggregating the distance set associated with each point in the reference point cloud. In some embodiments, example process 700 may further include determining a second distance 712 between the test point cloud and the reference point cloud by aggregating each of the first distances associated with each corresponding point in the reference point cloud. Although the example processing loop is presented as occurring serially, in some embodiments, the processing of individual corresponding points in the point cloud and the processing loop may generally occur, for example, in parallel or, for example, in a combination of serial and parallel processing.

[0163] While methods and systems according to some embodiments have been generally discussed in the context of extended reality (XR), some embodiments can be applied to any XR context such as, for example, virtual reality (VR) / mixed reality (MR) / augmented reality (AR). Furthermore, although the term "head-mounted display (HMD)" is used herein according to some embodiments, some embodiments can be applied to wearable devices (which may or may not be attached to the head) that have, for example, XR, VR, AR, and / or MR capabilities for some embodiments.

[0164] A first example method according to some embodiments may include: for each point in a reference point cloud, accessing local covariance; for each point in the reference point cloud, accessing a plurality of nearest neighbor points in a test point cloud; for each point in the reference point cloud, determining a distance set, wherein the distance set includes Mahalanobis distance between a first point in the reference point cloud and a second point among the plurality of nearest neighbor points in the test point cloud; for each point in the reference point cloud, determining a first distance by aggregating the distance set associated with each point in the reference point cloud; and determining a second distance between the test point cloud and the reference point cloud by aggregating each of the first distances associated with each corresponding point in the reference point cloud.

[0165] For some embodiments of the first example method, aggregating the set of distances associated with each point in the reference point cloud includes averaging the set of distances associated with each point in the reference cloud.

[0166] For some embodiments of the first example method, aggregating each of the first distances associated with each corresponding point in the reference point cloud includes: determining the maximum value of the first distances associated with each corresponding point in the reference point cloud.

[0167] For some embodiments of the first example method, aggregating each of the first distances associated with each corresponding point in the reference point cloud includes: averaging each of the first distances associated with each corresponding point in the reference point cloud.

[0168] A second example method according to some embodiments may include, for each point in a reference point cloud, performing: obtaining a local covariance for the point in the reference point cloud; determining a set of distances for the point in the reference point cloud, wherein the set of distances includes Mahalanobis distances between the point in the reference point cloud and each of a plurality of nearest neighbor points in a test point cloud; determining a first distance corresponding to the point in the reference point cloud by aggregating the set of distances associated with the point in the reference point cloud; and determining a second distance between the test point cloud and the reference point cloud by aggregating each of the first distances associated with each corresponding point in the reference point cloud.

[0169] For some embodiments of the second example method, aggregating the set of distances associated with the points in the reference point cloud includes: determining the maximum value of the set of distances associated with the points in the reference point cloud.

[0170] For some embodiments of the second example method, aggregating the set of distances associated with the points in the reference point cloud includes averaging the set of distances associated with the points in the reference point cloud.

[0171] In some embodiments of the second example method, averaging the set of distances associated with the points in the reference point cloud includes averaging the Mahalanobis distances to the set of distances associated with the points in the reference point cloud.

[0172] For some embodiments of the second example method, aggregating each of the first distances associated with each corresponding point in the reference point cloud includes: determining the maximum value of the first distances associated with each corresponding point in the reference point cloud.

[0173] For some embodiments of the second example method, aggregating each of the first distances associated with each corresponding point in the reference point cloud includes: averaging each of the first distances associated with each corresponding point in the reference point cloud.

[0174] In some embodiments of the second example method, obtaining the local covariance includes: determining the local covariance.

[0175] In some embodiments of the second example method, the local covariance of the point in the reference point cloud includes a covariance matrix, which includes the distance between the point in the reference point cloud and each of a plurality of nearest neighbor points in the reference point cloud.

[0176] A third example method according to some embodiments may include: performing a processing loop for each point in a reference point cloud, wherein the processing loop includes: selecting a first point equal to a corresponding point in the reference point cloud for a current traversal of the processing loop; obtaining a local covariance; determining a set of distances, wherein the set of distances includes Mahalanobis distances between the first point and each of a plurality of nearest neighbor points in a test point cloud; determining a first distance by aggregating the set of distances associated with each point in the reference point cloud; and determining a second distance between the test point cloud and the reference point cloud by aggregating each of the first distances associated with each corresponding point in the reference point cloud.

[0177] For some embodiments of the third example method, aggregating the set of distances associated with each point in the reference point cloud includes: determining the maximum value of the set of distances associated with each point in the reference point cloud.

[0178] For some embodiments of the third example method, aggregating the set of distances associated with each point in the reference point cloud includes averaging the set of distances associated with each point in the reference point cloud.

[0179] For some embodiments of the third example method, aggregating each of the first distances associated with each corresponding point in the reference point cloud includes: averaging each of the first distances associated with each corresponding point in the reference point cloud.

[0180] In some embodiments of the third example method, the processing loop is executed for each point in the reference point cloud in a manner that is performed simultaneously for one or more points in the reference point cloud, at least in partial parallelism.

[0181] In some embodiments of the third example method, the processing loop is executed serially for each point in the reference point cloud.

[0182] In some embodiments of the third example method, obtaining the local covariance includes: determining the local covariance.

[0183] For some embodiments of the third example method, the local covariance of the point in the reference point cloud includes a covariance matrix, which includes the distance between the point in the reference point cloud and each of a plurality of nearest neighbor points in the reference point cloud.

[0184] In some embodiments of the third example method, determining the local covariance includes using a covariance matrix corresponding to a plurality of nearest neighbors of the first point.

[0185] In some embodiments of the third example method, determining the covariance matrix includes using the average of multiple nearest neighbors of the first point.

[0186] In some embodiments of the third example method, determining the covariance matrix includes determining a series of differences between the first point and each of a plurality of nearest neighbors of the first point.

[0187] In some embodiments of the third example method, determining the local covariance includes adding a small offset to one or more elements of the covariance matrix.

[0188] Some embodiments of the third example method may further include: using the second distance as a loss function to train a deep neural network, wherein the covariance matrix is ​​determined prior to training the deep neural network.

[0189] Some embodiments of the third example method may further include dynamically changing the second distance during the training of the deep neural network.

[0190] In some embodiments of the third example method, dynamically changing the second distance includes adjusting how many nearest neighbors are in the test point cloud.

[0191] For some embodiments of the third example method, dynamically changing the second distance includes: reducing how many nearest neighbor points are in the test point cloud.

[0192] In some embodiments of the third example method, determining the local covariance includes using the inverse of the covariance matrix corresponding to the plurality of nearest neighbors of the first point.

[0193] Some embodiments of the third example method may further include: determining a second local covariance based on the reference point cloud; determining a third distance based on the second local covariance; and adding the third distance to the second distance to obtain an enhanced distance.

[0194] A third example apparatus according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, operate to cause the apparatus to: perform a processing loop for each point in a reference point cloud, wherein the processing loop includes: selecting a first point equal to a corresponding point in the reference point cloud for a current traversal of the processing loop; obtaining a local covariance; determining a set of distances, wherein the set of distances includes Mahalanobis distances between the first point and each of a plurality of nearest neighbor points in a test point cloud; determining a first distance by aggregating the set of distances associated with each point in the reference point cloud; and determining a second distance between the test point cloud and the reference point cloud by aggregating each of the first distances associated with each corresponding point in the reference point cloud.

[0195] A fourth example method according to some embodiments may include: performing a processing loop for each point in a reference point cloud, wherein the processing loop includes: selecting a first point equal to a corresponding point in the reference point cloud for a current traversal of the processing loop; obtaining a local covariance; determining a set of distances, wherein the set of distances includes the square root of the Mahalanobis distance between the first point and each of a plurality of nearest neighbors in a test point cloud; determining a first distance by aggregating the set of distances associated with each point in the reference point cloud; and determining a second distance between the test point cloud and the reference point cloud by aggregating each of the first distances associated with each corresponding point in the reference point cloud.

[0196] For some embodiments of the fourth example method, aggregating the set of distances associated with each point in the reference point cloud includes: determining the maximum value of the set of distances associated with each point in the reference point cloud.

[0197] For some embodiments of the fourth example method, aggregating the set of distances associated with each point in the reference point cloud includes averaging the set of distances associated with each point in the reference point cloud.

[0198] For some embodiments of the fourth example method, averaging the set of distances associated with each point in the reference point cloud includes averaging over the plurality of nearest neighbor points in the reference point cloud.

[0199] In some embodiments of the fourth example method, determining the local covariance includes using a covariance matrix corresponding to a plurality of nearest neighbors of the first point.

[0200] In some embodiments of the fourth example method, determining the covariance matrix includes using the average of multiple nearest neighbors of the first point.

[0201] In some embodiments of the fourth example method, determining the covariance matrix includes determining a series of differences between the first point and each of a plurality of nearest neighbors of the first point.

[0202] In some embodiments of the fourth example method, determining the local covariance includes adding a small offset to one or more elements of the covariance matrix.

[0203] Some embodiments of the fourth example method may further include: using the second distance as a loss function to train a deep neural network, wherein the covariance matrix is ​​determined prior to training the deep neural network.

[0204] Some embodiments of the fourth example method may further include dynamically changing the second distance during the training of the deep neural network.

[0205] In some embodiments of the fourth example method, dynamically changing the second distance includes adjusting how many nearest neighbors are in the test point cloud.

[0206] For some embodiments of the fourth example method, dynamically changing the second distance includes: reducing how many nearest neighbor points are in the test point cloud.

[0207] In some embodiments of the fourth example method, determining the local covariance includes using the inverse of the covariance matrix corresponding to the nearest neighbors of the first point.

[0208] Some embodiments of the fourth example method may further include: determining a second local covariance based on the reference point cloud; determining a third distance based on the second local covariance; and adding the third distance to the second distance to obtain an enhanced distance.

[0209] A fourth example apparatus according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, operate to cause the apparatus to: perform a processing loop for each point in a reference point cloud, wherein the processing loop includes: selecting a first point equal to a corresponding point in the reference point cloud for a current traversal of the processing loop; obtaining a local covariance; determining a set of distances, wherein the set of distances includes the square root of the Mahalanobis distance between the first point and each of a plurality of nearest neighbors in a test point cloud; determining a first distance by aggregating the set of distances associated with each point in the reference point cloud; and determining a second distance between the test point cloud and the reference point cloud by aggregating each of the first distances associated with each corresponding point in the reference point cloud.

[0210] A fifth example apparatus according to some embodiments may include at least one processor configured to perform any of the methods described above.

[0211] A sixth example apparatus according to some embodiments may include a computer-readable medium storing instructions for causing one or more processors to perform any of the methods described above.

[0212] A seventh example apparatus according to some embodiments may include at least one processor and at least one non-transitory computer-readable medium storing instructions for causing the at least one processor to perform any of the methods described above.

[0213] Example signals according to some embodiments may include a bitstream generated according to any of the methods described above.

[0214] This disclosure describes a variety of aspects, including tools, features, embodiments, models, schemes, etc. Many of these aspects are described in detail and are often described in a manner that may sound limiting, at least to illustrate individual characteristics. However, this is for clarity of purpose and does not limit the disclosure or scope of those aspects. Indeed, all the different aspects can be combined and interchanged to provide further aspects. Furthermore, this aspect can also be combined and interchanged with aspects described in earlier filings.

[0215] The aspects described and contemplated in this disclosure can be implemented in many different forms. Although some embodiments are specifically illustrated, other embodiments are contemplated, and the discussion of particular embodiments does not limit the breadth of implementations. At least one of the aspects generally relates to video encoding and decoding, and at least one other aspect generally relates to the transmission of a generated or encoded bitstream. These and other aspects can be implemented as methods, apparatus, computer-readable storage media having instructions stored thereon for encoding or decoding video data according to any of the described methods, and / or computer-readable storage media having bitstreams generated according to any of the described methods stored thereon.

[0216] In this disclosure, the terms "reconstructed" and "decoded" may be used interchangeably, as may the terms "pixel" and "sample," and the terms "image," "picture," and "frame." Typically, but not necessarily, the term "reconstructed" is used on the encoder side and "decoded" is used on the decoder side.

[0217] The terms HDR (High Dynamic Range) and SDR (Standard Dynamic Range) often convey specific values ​​of dynamic range to those skilled in the art. However, additional embodiments are contemplated, wherein a reference to HDR is understood to mean "higher dynamic range" and a reference to SDR is understood to mean "lower dynamic range." Such additional embodiments are not constrained by any specific values ​​of dynamic range that may often be associated with the terms "high dynamic range" and "standard dynamic range."

[0218] This document describes various methods, and each method includes one or more steps or actions for implementing the described method. Unless a specific order of steps or actions is required for the proper operation of the method, the order and / or use of specific steps and / or actions may be modified or combined. Additionally, terms such as "first," "second," etc., may be used in various embodiments to modify elements, components, steps, operations, etc., such as, for example, "first decoding" and "second decoding." The use of such terms does not imply a sequence of modified operations unless specifically required. Thus, in this example, the first decoding need not be performed before the second decoding, but may occur, for example, before, during, or in the time period overlapping with the second decoding.

[0219] For example, various numerical values ​​may be used in this disclosure. Specific values ​​are for illustrative purposes only, and the aspects described are not limited to these specific values.

[0220] The embodiments described herein may be implemented by computer software or other hardware implemented by a processor, or by a combination of hardware and software. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. The processor may be of any type suitable for the technical environment and may encompass one or more of microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures, as a non-limiting example.

[0221] Various implementations involve decoding. As used in this disclosure, “decoding” can encompass all or part of a process performed, for example, on a received encoded sequence to produce a final output suitable for display. In various embodiments, such a process includes one or more processes typically performed by a decoder, such as entropy decoding, dequantization, inverse transform, and differential decoding. In various embodiments, such a process also or alternatively includes processes performed by a decoder of the various implementations described in this disclosure, such as: extracting images from chunks (packed) of images, determining an upsampling filter to use and then upsampling the images, and flipping the images back to their intended orientation.

[0222] As a further example, in one embodiment, "decoding" refers only to entropy decoding; in another embodiment, "decoding" refers only to differential decoding; and in yet another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" is intended to specifically refer to a subset of operations or generally to a broader decoding process will be clear based on the specific context of the description.

[0223] Various implementations involve encoding. In a manner similar to the above discussion of “decoding,” the term “encoding,” as used herein, can encompass all or part of a process performed, for example, on an input video sequence to produce an encoded bitstream. In various embodiments, such a process includes one or more processes typically performed by an encoder, such as partitioning, differential coding, transform, quantization, and entropy coding. In various embodiments, such a process also, or alternatively, includes processes performed by an encoder of the various implementations described herein.

[0224] As a further example, in one embodiment, "encoding" refers only to entropy encoding; in another embodiment, "encoding" refers only to differential encoding; and in yet another embodiment, "encoding" refers to a combination of differential and entropy encoding. Whether the phrase "encoding process" is intended to specifically refer to a subset of operations or generally to a broader encoding process will be clear based on the context of the specific description.

[0225] When a diagram is presented as a flowchart, it should be understood that it also provides a block diagram of the corresponding device. Similarly, when a diagram is presented as a block diagram, it should be understood that it also provides a flowchart of the corresponding method / process.

[0226] Various embodiments mention rate distortion optimization. Specifically, during the encoding process, a balance or trade-off between rate and distortion is often considered, taking into account computational complexity constraints. Rate distortion optimization is typically formulated as minimizing a rate distortion function, which is a weighted sum of rate and distortion. Different schemes exist for solving the rate distortion optimization problem. For example, a scheme can be based on a comprehensive test of all encoding options, including all considered modes or encoding parameter values, with their encoding costs and a complete evaluation of the correlated distortion of the reconstructed signal after encoding and decoding. Faster schemes can also be used to save encoding complexity, particularly by utilizing approximate distortion calculations based on prediction or prediction of the residual signal rather than the reconstructed signal. A hybrid of these two schemes can also be used, such as by applying approximate distortion to only some of the possible encoding options and full distortion to the others. Other schemes evaluate only a subset of the possible encoding options. More generally, many schemes employ any of a variety of techniques to perform optimization, but optimization is not necessarily a complete evaluation of both encoding costs and correlated distortion.

[0227] The implementations and aspects described herein can be implemented, for example, in methods or processes, apparatus, software programs, data streams, or signals. Even if discussed only in the context of a single form of implementation (e.g., discussed only as a method), the features in question can be implemented in other forms (e.g., apparatus or program). Apparatus can be implemented, for example, in appropriate hardware, software, and firmware. Methods can be implemented, for example, in a processor, where processor generally refers to a processing device, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices, such as, for example, computers, cellular phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate communication of information between end users.

[0228] References to "an embodiment" or "an embodiment" or "an implementation" or "an implementation," and other variations thereof, mean that a particular feature, structure, characteristic, etc., described in connection with an embodiment is included in at least one embodiment. Therefore, the phrases "in an embodiment" or "in one embodiment" or "in one implementation," and any variations appearing throughout this disclosure, do not necessarily all refer to the same embodiment.

[0229] Additionally, this disclosure may refer to "determining" each piece of information. Determining information may include one or more of, for example, estimation information, calculation information, prediction information, or information retrieved from memory.

[0230] Furthermore, this disclosure may refer to "accessing" each piece of information. Accessing information may include one or more of the following: receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0231] Additionally, this disclosure may refer to "receiving" individual pieces of information. As with "access," receiving is intended to be a broad term. Receiving information may include one or more of, for example, accessing information or retrieving information (e.g., from memory). Further, "receiving" typically refers to actions performed during operation, such as, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0232] It should be understood that the use of any of the following “ / ”, “and / or”, and “…at least one of” (e.g., in the cases of “A / B”, “A and / or B”, and “at least one of A and B”) is intended to cover the selection of only the first listed option (A), or only the second listed option (B), or the selection of both options (A and B). As a further example, in the cases of “A, B, and / or C” and “at least one of A, B, and C”, this phrase is intended to cover the selection of only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or the selection of all three options (A, B, and C). This can be extended to as many items as are listed.

[0233] Moreover, as used herein, the term “signaling” refers, among other things, to instructing the corresponding decoder to do something. For example, in some embodiments, the encoder signals a specific one of several parameters for selecting region-based filter parameters used for artifact removal filtering. In this way, in one embodiment, the same parameter is used at both the encoder and decoder sides. Thus, for example, the encoder can transmit (explicitly signal) the specific parameter to the decoder so that the decoder can use the same specific parameter. Conversely, if the decoder already has the specific parameter as well as other parameters, then signaling can be used without transmission (implicitly signal) to simply allow the decoder to know and select the specific parameter. Bit saving is achieved in various embodiments by avoiding the transmission of any actual functionality. It should be understood that signaling can be done in a variety of ways. For example, in various embodiments, one or more syntax elements, tags, etc., are used to signal information to the corresponding decoder. Although the signature refers to the verb form of the term “signaling,” the term “signaling” may also be used as a noun herein.

[0234] Implementations can generate various signals formatted to carry, for example, information that can be stored or transmitted. The information may include, for example, instructions for performing a method or data generated by one of the described implementations. For example, the signal may be formatted to carry a bit stream of the described embodiments. Such a signal may be formatted as, for example, electromagnetic waves (e.g., using the radio frequency portion of the spectrum) or baseband signals. Formatting may include, for example, encoding the data stream and modulating a carrier wave using the encoded data stream. The information carried by the signal may be, for example, analog or digital information. The signal may be transmitted over a variety of different wired or wireless links, as is well known. The signal may be stored on a processor-readable medium.

[0235] We have described several embodiments. These embodiments may be provided individually or in any combination across various claim classes and types. Furthermore, embodiments may include one or more of the following features, devices, or aspects, individually or in any combination, across various claim classes and types: • Adapt the residual at the encoder according to any of the embodiments discussed. • A bitstream or signal that includes one or more of the described syntax elements or their variants. • Bitstream or signal, which includes grammatical communication information generated according to any of the described embodiments. • Insert syntax elements into the signaling that enable the decoder to adapt the residuals in a manner corresponding to that used by the encoder. • Create and / or transmit and / or receive and / or decode bitstreams or signals, including one or more of the described syntax elements or their variants. • Create and / or transmit and / or receive and / or decode according to any of the described embodiments. • The method, process, apparatus, medium for storing instructions, medium for storing data, or signal according to any of the described embodiments. • TV, set-top box, cellular phone, tablet, or other electronic device that adapts the filter parameters according to any of the described embodiments. • TV, set-top box, cellular phone, tablet, or other electronic device that performs filter parameter adaptation and displays (e.g., using a monitor, screen, or other type of display) the resulting image according to any of the described embodiments. • TV, set-top box, cellular phone, tablet, or other electronic device that selects (e.g., uses a tuner) a channel to receive signals including encoded images and performs filter parameter adaptations according to any of the described embodiments. TVs, set-top boxes, cellular phones, tablets, or other electronic devices that receive signals that encode images and perform filter parameter adaptations according to any of the described embodiments.

[0236] Note that the various hardware elements of one or more in the described embodiments are referred to as “modules” that implement (i.e., perform, execute, etc.) the functions described herein in conjunction with the corresponding Module 2. As used herein, a module includes hardware (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), one or more memory devices) that are considered suitable by those skilled in the art for a given implementation. Each described module may also include instructions executable to implement one or more functions described as being implemented by the corresponding module, and it should be noted that such instructions may take the form of hardware (i.e., hardwired) instructions, firmware instructions, software instructions, etc., or include hardware (i.e., hardwired) instructions, firmware instructions, software instructions, etc., and may be stored in any one or more suitable non-transient computer-readable media (such as collectively referred to as RAM, ROM, etc.).

[0237] Although features and elements have been described above in specific combinations, those skilled in the art will appreciate that each feature or element may be used individually or in any combination with other features and elements. Furthermore, the methods described herein can be implemented in a computer program, software, or firmware incorporated in a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media (such as internal hard disks and removable disks), magnetic-optical media, and optical media (such as CD-ROMs and digital multifunction discs (DVDs)). The processor associated with the software can be used to implement a radio frequency transceiver for a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

1. A method comprising: For each point in the reference point cloud, access the local covariance; For each point in the reference point cloud, access multiple nearest neighbor points in the test point cloud; For each point in the reference point cloud, determine a distance set. The distance set includes the Mahalanobis distance between a first point in the reference point cloud and a second point among the plurality of nearest neighbors in the test point cloud; For each point in the reference point cloud, a first distance is determined by aggregating the set of distances associated with each point in the reference point cloud; as well as A second distance between the test point cloud and the reference point cloud is determined by aggregating each of the first distances associated with each corresponding point in the reference point cloud.

2. The method of claim 1, wherein aggregating the set of distances associated with each point in the reference point cloud comprises: Average the set of distances associated with each point in the reference cloud.

3. The method of claim 1, wherein aggregating each of the first distances associated with each corresponding point of the reference point cloud comprises: Determine the maximum value of the first distance associated with each corresponding point in the reference point cloud.

4. The method of claim 1, wherein aggregating each of the first distances associated with each corresponding point of the reference point cloud comprises: The average of each of the first distances associated with each corresponding point in the reference point cloud is calculated.

5. The method of claim 1, wherein accessing the local covariance comprises one of the following: determining the local covariance; and accessing a predetermined value of the local covariance.

6. A method comprising: For each point in the reference point cloud, execute: The local covariance is obtained for the points in the reference point cloud; Determine a distance set for the points in the reference point cloud. The distance set includes Mahalanobis distances between the point in the reference point cloud and each of a plurality of nearest neighbor points in the test point cloud; as well as A first distance corresponding to a point in the reference point cloud is determined by aggregating a set of distances associated with the point in the reference point cloud; as well as A second distance between the test point cloud and the reference point cloud is determined by aggregating each of the first distances associated with each corresponding point in the reference point cloud.

7. The method of claim 6, wherein aggregating the set of distances associated with the points in the reference point cloud comprises: Determine the maximum value of the set of distances associated with the points in the reference point cloud.

8. The method of claim 6, wherein aggregating the set of distances associated with the points in the reference point cloud comprises: The average of the set of distances associated with the points in the reference point cloud is calculated.

9. The method of claim 8, wherein averaging the set of distances associated with the points in the reference point cloud comprises: The Mahalanobis distances to the set of distances associated with the points in the reference point cloud are averaged.

10. The method of claim 6, wherein aggregating each of the first distances associated with each corresponding point of the reference point cloud comprises: Determine the maximum value of the first distance associated with each corresponding point in the reference point cloud.

11. The method of claim 6, wherein aggregating each of the first distances associated with each corresponding point of the reference point cloud comprises: The average of each of the first distances associated with each corresponding point in the reference point cloud is calculated.

12. The method of claim 6, wherein obtaining the local covariance comprises: Determine the local covariance.

13. The method of claim 6, wherein the local covariance of the point in the reference point cloud includes a covariance matrix, the covariance matrix including the distance between the point in the reference point cloud and each of a plurality of nearest neighbors in the reference point cloud.

14. A method comprising: A processing loop is performed for each point in the reference point cloud, wherein the processing loop includes: For the current traversal after the processing loop, select the first point that is equal to the corresponding point in the reference point cloud; Obtain the local covariance; Determine the distance set. The distance set includes the Mahalanobis distance between the first point and each of the plurality of nearest neighbors in the test point cloud; and A first distance is determined by aggregating the set of distances associated with each point in the reference point cloud; and A second distance between the test point cloud and the reference point cloud is determined by aggregating each of the first distances associated with each corresponding point in the reference point cloud.

15. The method of claim 14, wherein aggregating the set of distances associated with each point in the reference point cloud comprises: Determine the maximum value of the set of distances associated with each point in the reference point cloud.

16. The method of claim 14, wherein aggregating the set of distances associated with each point in the reference point cloud comprises: The average of the set of distances associated with each point in the reference point cloud is calculated.

17. The method of claim 14, wherein aggregating each of the first distances associated with each corresponding point of the reference point cloud comprises: The average of each of the first distances associated with each corresponding point in the reference point cloud is calculated.

18. The method of claim 14, wherein the processing loop is performed for each point in the reference point cloud in a manner that is performed simultaneously for one or more points in the reference point cloud, at least in partial parallelism.

19. The method of claim 14, wherein the processing loop is performed serially for each point in the reference point cloud.

20. The method of claim 14, wherein obtaining the local covariance comprises: Determine the local covariance.

21. The method of claim 14, wherein the local covariance of the point in the reference point cloud includes a covariance matrix, the covariance matrix including the distance between the point in the reference point cloud and each of a plurality of nearest neighbors in the reference point cloud.

22. The method of claim 14, wherein determining the local covariance comprises: Use the covariance matrix corresponding to the multiple nearest neighbors of the first point.

23. The method of claim 22, wherein determining the covariance matrix comprises: Use the average of the nearest neighbors of the first point.

24. The method of claim 23, wherein determining the covariance matrix comprises: Determine a series of differences between the first point and each of its nearest neighbors.

25. The method of any one of claims 22-24, wherein determining the local covariance comprises: Add a small offset to one or more elements of the covariance matrix.

26. The method of any one of claims 22-25, further comprising: The second distance is used as a loss function to train the deep neural network. The covariance matrix is ​​determined before training the deep neural network.

27. The method of claim 26, further comprising: The second distance is dynamically changed during the training of the deep neural network.

28. The method of claim 27, wherein dynamically changing the second distance comprises: Adjust how many nearest neighbor points are in the test point cloud.

29. The method of claim 27, wherein dynamically changing the second distance comprises: How much the nearest neighbor points in the test point cloud are reduced.

30. The method of any one of claims 14-29, wherein determining the local covariance comprises: Use the inverse of the covariance matrix corresponding to the nearest neighbors of the first point.

31. The method of any one of claims 14-30, further comprising: The second local covariance is determined based on the reference point cloud; The third distance is determined based on the second local covariance; as well as The third distance is added to the second distance to obtain an enhanced distance.

32. An apparatus comprising: processor; as well as A non-transitory computer-readable medium storing instructions that, when executed by the processor, operate to cause the device to: A processing loop is performed for each point in the reference point cloud, wherein the processing loop includes: For the current traversal after the processing loop, select the first point that is equal to the corresponding point in the reference point cloud; Obtain the local covariance; Determine the distance set. The distance set includes the Mahalanobis distance between the first point and each of the plurality of nearest neighbors in the test point cloud; and A first distance is determined by aggregating the set of distances associated with each point in the reference point cloud; and A second distance between the test point cloud and the reference point cloud is determined by aggregating each of the first distances associated with each corresponding point in the reference point cloud.

33. A method comprising: A processing loop is performed for each point in the reference point cloud, wherein the processing loop includes: For the current traversal after the processing loop, select the first point that is equal to the corresponding point in the reference point cloud; Obtain the local covariance; Determine the distance set. The distance set includes the square root of the Mahalanobis distance between the first point and each of the plurality of nearest neighbors in the test point cloud; and A first distance is determined by aggregating the set of distances associated with each point in the reference point cloud; and A second distance between the test point cloud and the reference point cloud is determined by aggregating each of the first distances associated with each corresponding point in the reference point cloud.

34. The method of claim 33, wherein aggregating the set of distances associated with each point in the reference point cloud comprises: Determine the maximum value of the set of distances associated with each point in the reference point cloud.

35. The method of claim 33, wherein aggregating the set of distances associated with each point in the reference point cloud comprises: The average of the set of distances associated with each point in the reference point cloud is calculated.

36. The method of claim 35, wherein averaging the set of distances associated with each point in the reference point cloud comprises: The average is calculated over the plurality of nearest neighbor points in the reference point cloud.

37. The method of claim 33, wherein determining the local covariance comprises: Use the covariance matrix corresponding to the multiple nearest neighbors of the first point.

38. The method of claim 37, wherein determining the covariance matrix comprises: Use the average of the nearest neighbors of the first point.

39. The method of claim 37, wherein determining the covariance matrix comprises: Determine a series of differences between the first point and each of its nearest neighbors.

40. The method of any one of claims 37-39, wherein determining the local covariance comprises: Add a small offset to one or more elements of the covariance matrix.

41. The method of any one of claims 37-39, further comprising: The second distance is used as a loss function to train the deep neural network. The covariance matrix is ​​determined before training the deep neural network.

42. The method of claim 40, further comprising: The second distance is dynamically changed during the training of the deep neural network.

43. The method of claim 42, wherein dynamically changing the second distance comprises: Adjust how many nearest neighbor points are in the test point cloud.

44. The method of claim 42, wherein dynamically changing the second distance comprises: How much the nearest neighbor points in the test point cloud are reduced.

45. The method of any one of claims 33-44, wherein determining the local covariance comprises: Use the inverse of the covariance matrix corresponding to the nearest neighbors of the first point.

46. ​​The method of any one of claims 33-45, further comprising: The second local covariance is determined based on the reference point cloud; The third distance is determined based on the second local covariance; as well as The third distance is added to the second distance to obtain an enhanced distance.

47. An apparatus comprising: processor; as well as A non-transitory computer-readable medium storing instructions that, when executed by the processor, operate to cause the device to: A processing loop is performed for each point in the reference point cloud, wherein the processing loop includes: For the current traversal after the processing loop, select the first point that is equal to the corresponding point in the reference point cloud; Obtain the local covariance; Determine the distance set. The distance set includes the square root of the Mahalanobis distance between the first point and each of the plurality of nearest neighbors in the test point cloud; and A first distance is determined by aggregating the set of distances associated with each point in the reference point cloud; and A second distance between the test point cloud and the reference point cloud is determined by aggregating each of the first distances associated with each corresponding point in the reference point cloud.

48. A method comprising: For each point in the first point cloud, access the local covariance; For each point in the first point cloud, visit multiple nearest neighbor points in the second point cloud; For each point in the first point cloud, determine the distance set. The distance set includes the Mahalanobis distance between a first point in the first point cloud and a second point among the plurality of nearest neighbors in the second point cloud; For each point in the first point cloud, a first distance is determined by aggregating the set of distances associated with each point in the first point cloud; as well as A second distance between the second point cloud and the first point cloud is determined by aggregating each of the first distances associated with each corresponding point in the first point cloud.

49. An apparatus comprising: processor; as well as A non-transitory computer-readable medium storing instructions that, when executed by the processor, operate to cause the device to: For each point in the first point cloud, access the local covariance; For each point in the first point cloud, visit multiple nearest neighbor points in the second point cloud; For each point in the first point cloud, determine the distance set. The distance set includes the Mahalanobis distance between a first point in the first point cloud and a second point among the plurality of nearest neighbors in the second point cloud; For each point in the first point cloud, a first distance is determined by aggregating the set of distances associated with each point in the first point cloud; as well as A second distance between the second point cloud and the first point cloud is determined by aggregating each of the first distances associated with each corresponding point in the first point cloud.

50. An apparatus comprising at least one processor configured to perform the method as claimed in any one of claims 1-31, 33-46 and 48.

51. An apparatus comprising a computer-readable medium storing instructions for causing one or more processors to perform the method as described in any one of claims 1-31, 33-46, and 48.

52. An apparatus comprising at least one processor and at least one non-transitory computer-readable medium storing instructions for causing the at least one processor to perform the method as claimed in any one of claims 1-31, 33-46 and 48.

53. A signal comprising a bit stream generated according to any one of claims 1-31, 33-46 and 48.

Images