Method, apparatus, and system for graph conditional autoencoder (GCAE) using topology-friendly representation

Abstract

Disclosed are methods, apparatus, and systems implemented by a neural network-based decoder (NNBD). In one method, the NNBD can obtain or receive a codeword as a descriptor of an input data representation. A first neural network module can determine a preliminary reconstruction of the input data representation based at least on the codeword and an initial graph. The NNBD can determine a modified graph based at least on the preliminary reconstruction and the codeword. The first neural network module can determine a refined reconstruction of the input data representation based at least on the codeword and the modified graph. The modified graph can indicate topological information associated with the input data representation.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of priority to U.S. Patent Application No. 63 / 047,446, filed June 1, 2020, and refiled July 2, 2020, the contents of which are incorporated by reference as if fully set forth herein.

[0002] The embodiments disclosed herein generally relate to autoencoders for processing and / or compressing and reconstructing data representations, e.g., methods, devices, and systems for processing, analyzing, interpolating, representing, and / or understanding data representations including, for example, point clouds (PCs), video, images, and audio using learned topology-friendly representations. DETAILED DESCRIPTION OF THE INVENTION

[0003] In particular embodiments, unsupervised learning processes, operations, methods, and / or functions may be implemented, for example, for 3D PCs and / or other implementations, using, among other things, a TearingNet or a Graph Conditional Autoencoder (GCAE). For example, unsupervised learning operations may include learning compact representations of, among other things, 3D PCs, videos, images, and / or audio without labeling information. In this manner, representative features may be extracted (e.g., automatically extracted) from 3D PCs and / or other data representations and applied to any subsequent tasks as auxiliary information and / or prior information. Because labeling large amounts of data (e.g., PC data or other data) can be time-consuming and / or expensive, unsupervised learning can be beneficial.

[0004] In particular embodiments, an autoencoder may be implemented to reconstruct PCs, e.g., based on their compact representations and / or semantic descriptors. For example, given a semantic descriptor corresponding to an object, a PC representing a particular object may be recovered. Such reconstruction may be implemented (e.g., fitted) as a decoder within a general unsupervised learning framework (e.g., an autoencoder), where the encoder may output feature descriptors with semantic interpretations.

[0005] In particular embodiments, an autoencoder may be implemented to take into account / use topology (e.g., via topology inference and / or topology information), for example. When dealing with PC reconstruction, a graph topology may be implemented to determine / take into account (e.g., explicitly determine / take into account) the relationships between points. A fully connected graph topology may be quite inaccurate in representing the PC topology because it does not follow the object surface, and may be less effective when dealing with objects with high genus and / or scenes with multiple objects. The N in the reconstructed PC may be 2 Since at a given point there are N graph parameters (graph weights) to learn, learning the complete graph may be costly and / or may use a lot of memory and / or computation.

[0006] In some embodiments, methods, apparatus, systems, and / or procedures may be implemented to learn (e.g., effectively learn) PC topological representations. The implementations may be useful in PC reconstruction for complex objects / scenes, but may also be applied to weakly supervised PC tasks in classification, segmentation, and / or recognition, among others. [Brief explanation of the drawings]

[0007] A more detailed understanding may be had from the following detailed description, taken by way of example in conjunction with the accompanying drawings, in which: The figures in the description are examples; therefore, the figures and detailed description should not be considered limiting, as other equally effective examples are possible and likely. Also, like reference numerals in the figures indicate like elements. [Figure 1A] FIG. 1 is a system diagram illustrating an example communication system in which one or more disclosed embodiments may be implemented. [Figure 1B] 1B is a system diagram illustrating an exemplary wireless transmit / receive unit (WTRU) that may be used within the communications system shown in FIG. 1A, according to one embodiment. [Figure 1C] 1B is a system diagram illustrating an example radio access network (RAN) and an example core network (CN) that may be used within the communication system shown in FIG. 1A, according to one embodiment. [Figure 1D] 1B is a system diagram illustrating a further exemplary RAN and a further exemplary CN that may be used within the communication system shown in FIG. 1A, according to one embodiment. [Figure 2] A diagram showing a typical autoencoder (e.g., FoldingNet). [Figure 3] FIG. 1 illustrates another representative autoencoder (e.g., AtlasNet). [Figure 4] FIG. 1 illustrates a further representative autoencoder (e.g., FoldingNet++). [Figure 5] FIG. 1 illustrates an additional representative autoencoder (e.g., TearingNet) with a Tearing Network (T-Net) module. [Figure 6] FIG. 1 illustrates a representative T-Net module. [Figure 7A] FIG. 1 shows an example of input PCs, the resulting torn 2D grid, and reconstructed PCs. [Figure 7B]FIG. 1 shows an example of input PCs, the resulting torn 2D grid, and reconstructed PCs. [Figure 7C] FIG. 1 shows an example of input PCs, the resulting torn 2D grid, and reconstructed PCs. [Figure 8] FIG. 1 illustrates a representative GCAE autoencoder using a T-Net module, e.g., for a PC. [Figure 9] FIG. 1 illustrates a representative GCAE using a T-Net module for use in generalized operations (e.g., for use with a PC, images, video, and / or audio, among others). [Figure 10] FIG. 1 is a block diagram illustrating a representative method (e.g., implemented by a neural network-based decoder (NNBD)). [Figure 11] FIG. 1 is a block diagram illustrating an exemplary training method using multi-stage training motions. [Figure 12] FIG. 10 is a block diagram illustrating another exemplary method (eg, implemented by an NNBD). [Figure 13] FIG. 10 is a block diagram illustrating a further representative method (e.g., implemented by a neural network-based autoencoder (NNBAE)) including, for example, an encoding network (E-Net) module and NNBD. [Figure 14] FIG. 10 is a block diagram illustrating additional exemplary methods (e.g., implemented by an NNBD). [Figure 15] FIG. 2 is a block diagram illustrating another exemplary training method (e.g., implemented by a neural network (NN)) that uses a multi-stage training operation. [Figure 16] FIG. 10 is a block diagram illustrating yet another exemplary method (e.g., implemented by an NNBAE including an E-Net module and an NNBD).

[0008] Exemplary Network for Implementing the Embodiments 1A illustrates an exemplary communication system 100 in which one or more disclosed embodiments may be implemented. Communication system 100 may be a multiple-access system that provides content, such as voice, data, video, messaging, broadcasts, etc., to multiple wireless users. Communication system 100 may enable multiple wireless users to access such content through sharing of system resources, including wireless bandwidth. For example, the communication system 100 may employ one or more channel access methods such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single-carrier FDMA (SC-FDMA), zero-tail unique-word DFT-Spread OFDM (ZT UW DTS-s OFDM), unique word OFDM (UW-OFDM), resource block filtered OFDM, filter bank multicarrier (FBMC), etc.

[0009] 1A, communications system 100 may include wireless transmit / receive units (WTRUs) 102a, 102b, 102c, 102d, RANs 104 / 113, CNs 106 / 115, public switched telephone network (PSTN) 108, the Internet 110, and other networks 112, although it will be understood that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and / or network elements. Each of WTRUs 102a, 102b, 102c, 102d may be any type of device configured to operate and / or communicate in a wireless environment. By way of example, the WTRUs 102a, 102b, 102c, 102d, any of which may be referred to as a “station” and / or “STA,” may be configured to transmit and / or receive wireless signals and may include user equipment (UE), mobile stations, fixed or mobile subscriber units, subscription-based units, pagers, cellular phones, personal digital assistants (PDAs), smartphones, laptops, netbooks, personal computers, wireless sensors, hotspots or Mi-Fi devices, Internet of Things (IoT) devices, watches or other wearables, head-mounted displays (HMDs), vehicles, drones, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices operating in industrial and / or automated processing chain contexts), consumer electronics devices, devices operating in commercial and / or industrial wireless networks, etc. Any of the WTRUs 102a, 102b, 102c, and 102d may be referred to interchangeably as a UE.

[0010] The communications system 100 may also include a base station 114a and / or a base station 114b. Each of the base stations 114a, 114b may be any type of device configured to wirelessly interface with at least one of the WTRUs 102a, 102b, 102c, 102d to facilitate access to one or more communications networks, such as the CN 106 / 115, the Internet 110, and / or other networks 112. By way of example, the base stations 114a, 114b may be a base transceiver station (BTS), a Node B, an eNodeB (eNB), a Home Node B (HNB), a Home eNodeB (HeNB), a gNB, an NR NodeB, a site controller, an access point (AP), a wireless router, etc. Although the base stations 114a, 114b are each shown as a single element, it will be understood that the base stations 114a, 114b may include any number of interconnected base stations and / or network elements.

[0011] The base station 114a may be part of the RAN 104 / 113, which may also include other base stations and / or network elements (not shown), such as a base station controller (BSC), a radio network controller (RNC), a relay node, etc. The base station 114a and / or base station 114b may be configured to transmit and / or receive radio signals on one or more carrier frequencies, which may be referred to as a cell (not shown). These frequencies may be licensed spectrum, unlicensed spectrum, or a combination of licensed and unlicensed spectrum. A cell may provide wireless service coverage for a particular geographic area, which may be relatively fixed or may change over time. A cell may be further divided into cell sectors. For example, the cell associated with the base station 114a may be divided into three sectors. Thus, in one embodiment, the base station 114a may include three transceivers, i.e., one transceiver for each sector of the cell. In one embodiment, the base station 114a may employ multiple-input multiple output (MIMO) technology and may utilize multiple transceivers per sector of the cell, for example, using beamforming to transmit and / or receive signals in desired spatial directions.

[0012] The base stations 114a, 114b may communicate with one or more of the WTRUs 102a, 102b, 102c, 102d over an air interface 116, which may be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). The air interface 116 may be established using any suitable radio access technology (RAT).

[0013] More specifically, as noted above, the communications system 100 may be a multiple-access system and may use one or more channel access schemes, such as, for example, CDMA, TDMA, FDMA, OFDMA, SC-FDMA, etc. For example, the base station 114 a and the WTRUs 102 a, 102 b, 102 c in the RAN 104 / 113 may implement a radio technology such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which may establish the air interface 115 / 116 / 117 using wideband CDMA (WCDMA). WCDMA may include communications protocols such as High-Speed Packet Access (HSPA) and / or Evolved HSPA (HSPA+). HSPA may include High-Speed Downlink (DL) Packet Access (HSDPA) and / or High-Speed Uplink Packet Access (HSUPA).

[0014] In one embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which may establish the air interface 116 using Long Term Evolution (LTE) and / or LTE-Advanced (LTE-A) and / or LTE-Advanced Pro (LTE-A Pro).

[0015] In one embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as NR radio access, which may establish the air interface 116 using New Radio (NR).

[0016] In one embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement multiple radio access technologies. For example, the base station 114a and the WTRUs 102a, 102b, 102c may jointly implement LTE radio access and NR radio access, e.g., using dual connectivity (DC) principles. Thus, the air interface utilized by the WTRUs 102a, 102b, 102c may be characterized by multiple types of radio access technologies and / or transmissions transmitted to / from multiple types of base stations (e.g., eNBs and gNBs).

[0017] In other embodiments, the base station 114a and the WTRUs 102a, 102b, 102c may implement a wireless technology such as IEEE 802.11 (i.e., Wireless Fidelity, WiFi), IEEE 802.16 (i.e., Worldwide Interoperability for Microwave Access, WiMAX), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Interim Standard 2000 (IS-2000), Interim Standard 95 (IS-95), Interim Standard 856 (IS-856), Global System for Mobile communications (GSM), Enhanced Data rates for GSM Evolution (EDGE), GSM EDGE (GERAN), or the like.

[0018] 1A may be, for example, a wireless router, a Home Node B, a Home eNode B, or an access point and may utilize any suitable RAT to facilitate wireless connectivity in a local area such as a location such as a business, a home, a vehicle, a campus, an industrial facility, an air corridor (e.g., for use by drones), a road, etc. In one embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology such as IEEE 802.11 to establish a wireless local area network (WLAN). In one embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology such as IEEE 802.15 to establish a wireless personal area network (WPAN). In yet another embodiment, the base station 114b and the WTRUs 102c, 102d may establish a picocell or a femtocell using a cellular-based RAT (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR, etc.). As shown in FIG. 1A, the base station 114b may have a direct connection to the Internet 110. Thus, the base station 114b may not need to access the Internet 110 through the CN 106 / 115.

[0019] The RAN 104 / 113 may communicate with the CN 106 / 115, which may be any type of network configured to provide voice, data, application, and / or voice over internet protocol (VoIP) services to one or more of the WTRUs 102a, 102b, 102c, 102d. The data may have various quality of service (QoS) requirements, such as different throughput, latency, error tolerance, reliability, data throughput, and mobility requirements. The CN 106 / 115 may provide call control, billing services, mobile location-based services, prepaid calls, Internet connectivity, video distribution, and / or perform high-level security functions such as user authentication. Although not shown in FIG. 1A , it will be understood that the RAN 104 / 113 and / or the CN 106 / 115 may communicate directly or indirectly with other RANs employing the same RAT as the RAN 104 / 113 or a different RAT. For example, in addition to being connected to the RAN 104 / 113, which may utilize NR radio technology, the CN 106 / 115 may also communicate with another RAN (not shown) employing GSM, UMTS, CDMA2000, WiMAX, E-UTRA, or WiFi radio technology.

[0020] The CN 106 / 115 may also serve as a gateway for the WTRUs 102a, 102b, 102c, 102d to access the PSTN 108, the Internet 110, and / or other networks 112. The PSTN 108 may include a public switched telephone network providing plain old telephone service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices that use common communication protocols, such as the transmission control protocol (TCP), user datagram protocol (UDP), and / or the internet protocol (IP) of the TCP / IP Internet protocol suite. The network 112 may include wired and / or wireless communication networks owned and / or operated by other service providers. For example, the network 112 may include another CN connected to one or more RANs, which may employ the same RAT as the RAN 104 / 113 or a different RAT.

[0021] Some or all of the WTRUs 102a, 102b, 102c, 102d in the communications system 100 may include multi-mode capabilities (e.g., the WTRUs 102a, 102b, 102c, 102d may include multiple transceivers for communicating with different wireless networks over different wireless links.) For example, the WTRU 102c shown in FIG. 1A may be configured to communicate with a base station 114a that may use a cellular-based wireless technology and a base station 114b that may use an IEEE 802 wireless technology.

[0022] 1B is a system diagram illustrating an example WTRU 102. As shown in FIG. 1B, the WTRU 102 may include, among other things, a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a keypad 126, a display / touchpad 128, non-removable memory 130, removable memory 132, a power source 134, a global positioning system (GPS) chipset 136, and / or other peripherals 138. It will be understood that the WTRU 102 may include any sub-combination of the foregoing elements while remaining consistent with an embodiment.

[0023] The processor 118 may be a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc. The processor 118 may perform signal coding, data processing, power control, input / output processing, and / or any other functionality that enables the WTRU 102 to operate in a wireless environment. The processor 118 may be coupled to the transceiver 120, which may be coupled to the transmit / receive element 122. While FIG. 1B depicts the processor 118 and the transceiver 120 as separate components, it will be understood that the processor 118 and the transceiver 120 may be integrated together in an electronic package or chip.

[0024] The transmit / receive element 122 may be configured to transmit signals to or receive signals from a base station (e.g., base station 114a) over the air interface 116. For example, in one embodiment, the transmit / receive element 122 may be an antenna configured to transmit and / or receive RF signals. In one embodiment, the transmit / receive element 122 may be an emitter / detector configured to transmit and / or receive IR, UV, or visible light signals, for example. In yet another embodiment, the transmit / receive element 122 may be configured to transmit and / or receive both RF and light signals. It will be understood that the transmit / receive element 122 may be configured to transmit and / or receive any combination of wireless signals.

[0025] 1B as a single element, the WTRU 102 may include any number of transmit / receive elements 122. More specifically, the WTRU 102 may use MIMO technology. Thus, in one embodiment, the WTRU 102 may include two or more transmit / receive elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface 116.

[0026] The transceiver 120 may be configured to modulate signals transmitted by the transmit / receive element 122 and demodulate signals received by the transmit / receive element 122. As mentioned above, the WTRU 102 may have multi-mode capabilities. Thus, the transceiver 120 may include multiple transceivers to enable the WTRU 102 to communicate via multiple RATs, such as NR and IEEE 802.11.

[0027] The processor 118 of the WTRU 102 may be coupled to and may receive user-entered data from a speaker / microphone 124, a keypad 126, and / or a display / touchpad 128 (e.g., a liquid crystal display (LCD) display unit or an organic light-emitting diode (OLED) display unit). The processor 118 may also output user data to the speaker / microphone 124, the keypad 126, and / or the display / touchpad 128. Furthermore, the processor 118 may access information from and store data in any type of suitable memory, such as non-removable memory 130 and / or removable memory 132. The non-removable memory 130 may include random-access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, etc. In other embodiments, the processor 118 may access information and store data in memory that is not physically located on the WTRU 102, such as on a server or home computer (not shown).

[0028] The processor 118 may receive power from the power source 134, but may be configured to distribute and / or control the power to other components in the WTRU 102. The power source 134 may be any suitable device for providing power to the WTRU 102. For example, the power source 134 may include one or more dry batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, etc.

[0029] The processor 118 may also be coupled to a GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) regarding the current location of the WTRU 102. In addition to or instead of information from the GPS chipset 136, the WTRU 102 may receive location information from a base station (e.g., base stations 114a, 114b) over the air interface 116 and / or determine its location based on the timing of signals being received from two or more nearby base stations. It will be appreciated that the WTRU 102 may obtain location information by way of any suitable location-determination method while remaining consistent with an embodiment.

[0030] The processor 118 may further be coupled to other peripherals 138, which may include one or more software and / or hardware modules that provide additional features, functionality, and / or wired or wireless connectivity. For example, the peripherals 138 may include an accelerometer, an electronic compass, a satellite transceiver, a digital camera (for photos and / or videos), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands-free headset, a Bluetooth module, a frequency modulated (FM) radio unit, a digital music player, a media player, a video game player module, an internet browser, a virtual reality and / or augmented reality (VR / AR) device, an activity tracker, etc. The peripheral device 138 may include one or more sensors, which may be one or more of a gyroscope, an accelerometer, a Hall effect sensor, a magnetometer, a direction sensor, a proximity sensor, a temperature sensor, a time sensor, a geolocation sensor, an altimeter, a light sensor, a touch sensor, a magnetometer, a barometer, a gesture sensor, a biometric sensor, and / or a humidity sensor.

[0031] The processor 118 of the WTRU 102 may be in operative communication with various peripherals 138, including, for example, one or more accelerometers, one or more gyroscopes, a USB port, other communication interfaces / ports, a display, and / or any other visual / audio indicators, to implement the exemplary embodiments disclosed herein.

[0032] The WTRU 102 may include a full-duplex radio for transmitting and receiving some or all of the signals (e.g., associated with a particular subframe on both the UL (e.g., for transmission) and DL (e.g., for reception)) simultaneously and / or together. The full-duplex radio may include an interference management unit for reducing and or substantially eliminating self-interference through hardware (e.g., chokes) or signal processing via a processor (e.g., via a separate processor (not shown) or processor 118). In one embodiment, the WTRU 102 may include a half-duplex radio for transmitting and receiving some or all of the signals (e.g., associated with a particular subframe on either the UL (e.g., for transmission) or DL (e.g., for reception)).

[0033] 1C is a system diagram illustrating the RAN 104 and the CN 106 according to one embodiment. As mentioned above, the RAN 104 may communicate with the WTRUs 102a, 102b, 102c over the air interface 116 using E-UTRA radio technology. The RAN 104 may also communicate with the CN 106.

[0034] The RAN 104 may include eNodeBs 160a, 160b, and 160c, although it will be understood that the RAN 104 may include any number of eNodeBs while remaining consistent with an embodiment. The eNodeBs 160a, 160b, and 160c may each include one or more transceivers for communicating with the WTRUs 102a, 102b, and 102c over the air interface 116. In one embodiment, the eNodeBs 160a, 160b, and 160c may implement MIMO technology. Thus, the eNodeB 160a may, for example, use multiple antennas to transmit wireless signals to and / or receive wireless signals from the WTRU 102a.

[0035] Each of the eNodeBs 160a, 160b, 160c may be associated with a particular cell (not shown) and may be configured to handle radio resource management decisions, handover decisions, user scheduling, etc. in the UL and / or DL. As shown in FIG. 1C, the eNodeBs 160a, 160b, 160c may communicate with one another via an X2 interface.

[0036] 1C may include a mobility management entity (MME) 162, a serving gateway (SGW) 164, and a packet data network (PDN) gateway (or PGW) 166. Although each of the foregoing elements is shown as part of the CN 106, it will be understood that any of these elements may be owned and / or operated by an entity other than the CN operator.

[0037] The MME 162 may be connected to each of the eNodeBs 160a, 160b, 160c in the RAN 104 via an S1 interface and may function as a control node. For example, the MME 162 may be responsible for authenticating users of the WTRUs 102a, 102b, 102c, activating / deactivating bearers, selecting a particular serving gateway during initial attach of the WTRUs 102a, 102b, 102c, etc. The MME 162 may provide a control plane function for switching between the RAN 104 and other RANs (not shown) that employ other radio technologies such as GSM and / or WCDMA.

[0038] The SGW 164 may be connected to each of the eNode-Bs 160a, 160b, 160c in the RAN 104 via an S1 interface. The SGW 164 may generally route and forward user data packets to and from the WTRUs 102a, 102b, 102c. The SGW 164 may perform other functions, such as anchoring the user plane during inter-eNode-B handovers, triggering paging when DL data is available to the WTRUs 102a, 102b, 102c, and managing and storing the context of the WTRUs 102a, 102b, 102c.

[0039] The SGW 164 may be connected to a PGW 166, which may provide the WTRUs 102a, 102b, 102c with access to packet-switched networks, such as the Internet 110, to facilitate communications between the WTRUs 102a, 102b, 102c and IP-enabled devices.

[0040] The CN 106 may facilitate communications with other networks. For example, the CN 106 may provide the WTRUs 102a, 102b, 102c with access to circuit-switched networks, such as the PSTN 108, to facilitate communications between the WTRUs 102a, 102b, 102c and traditional landline communications devices. For example, the CN 106 may include or communicate with an IP gateway (e.g., an IP multimedia subsystem (IMS) server) that serves as an interface between the CN 106 and the PSTN 108. Furthermore, the CN 106 may provide the WTRUs 102a, 102b, 102c with access to other networks 112, which may include other wired and / or wireless networks owned and / or operated by other service providers.

[0041] Although the WTRU is depicted in FIGS. 1A-1D as a wireless terminal, it is contemplated that in certain representative embodiments, such a terminal may use a wired communication interface (e.g., temporarily or permanently) with the communication network.

[0042] In a representative embodiment, the other network 112 may be a WLAN.

[0043] A WLAN in infrastructure Basic Service Set (BSS) mode may have an access point (AP) of the BSS and one or more stations (STAs) associated with the AP. The AP may have access or interface to a Distribution System (DS) or another type of wired / wireless network that carries traffic into and / or out of the BSS. Traffic originating from outside the BSS to a STA may arrive through the AP and be delivered to the STA. Traffic originating from a STA to a destination outside the BSS may be sent to the AP and transmitted to the respective destination. Traffic between STAs within the BSS may be transmitted, for example, through the AP; the source STA may send traffic to the AP, which may deliver the traffic to the destination STA. Traffic between STAs within the BSS may be considered and / or referred to as peer-to-peer traffic. Peer-to-peer traffic may be transmitted between a source STA and a destination STA (e.g., directly between them) in a direct link setup (DLS). In certain representative embodiments, the DLS may use 802.11e DLS or 802.11z tunneled DLS (TDLS). A WLAN using an Independent BSS (IBSS) mode may not have an AP, and STAs within or using the IBSS (e.g., all of the STAs) may communicate directly with each other. The IBSS mode of communication may be referred to herein as an "ad hoc" communication mode.

[0044] When using the 802.11ac infrastructure mode of operation or a similar mode of operation, an AP may transmit beacons on a fixed channel, such as a primary channel. The primary channel may be a fixed width (e.g., a 20 MHz wide bandwidth) or a width that is dynamically set via signaling. The primary channel may be the operating channel of the BSS and may be used by STAs to establish a connection with the AP. In certain representative embodiments, for example, in an 802.11 system, Carrier Sense Multiple Access / Collision Avoidance (CSMA / CA) with collision avoidance may be implemented. With CSMA / CA, STAs (e.g., all STAs), including the AP, may sense the primary channel. If the primary channel is sensed / detected and / or determined to be busy by a particular STA, the particular STA may back off. One STA (e.g., only one station) may transmit at any given time in a given BSS.

[0045] High Throughput (HT) STAs may use 40 MHz wide channels for communication, which may be formed, for example, through a combination of a primary 20 MHz channel and adjacent or non-adjacent 20 MHz channels.

[0046] A Very High Throughput (VHT) STA may support 20 MHz, 40 MHz, 80 MHz, and / or 160 MHz wide channels. The 40 MHz and / or 80 MHz wide channels may be formed by combining contiguous 20 MHz channels. A 160 MHz channel may be formed by combining eight contiguous 20 MHz channels or by combining two non-contiguous 80 MHz channels, which may be referred to as an 80+80 configuration. For the 80+80 configuration, after channel encoding, the data may pass through a segment parser that may split the data into two streams. Inverse Fast Fourier Transform (IFFT) processing and time-domain processing may be performed separately on each stream. The streams may be mapped to two 80 MHz channels, and the data may be transmitted by the transmitting STA. At the receiver of the receiving STA, the operations described above for the 80+80 configuration may be reversed and the combined data may be transmitted to the Medium Access Control (MAC).

[0047] Sub-1 GHz operating modes are supported by 802.11af and 802.11ah. Channel operating bandwidths and carriers are reduced in 802.11af and 802.11ah compared to those used in 802.11n and 802.11ac. 802.11af supports 5 MHz, 10 MHz, and 20 MHz bandwidths in the TV White Space (TVWS) spectrum, while 802.11ah supports 1 MHz, 2 MHz, 4 MHz, 8 MHz, and 16 MHz bandwidths using non-TVWS spectrum. According to representative embodiments, 802.11ah may support meter-type control / machine-type communications, such as MTC devices within macro coverage areas. MTC devices may have specific capabilities, including, for example, support for (e.g., only for) specific and / or limited bandwidths. MTC devices may include batteries with above-threshold battery life (e.g., to maintain very long battery life).

[0048] WLAN systems that can support multiple channels and channel bandwidths, such as 802.11n, 802.11ac, 802.11af, and 802.11ah, include a channel that can be designated as a primary channel. The primary channel can have a bandwidth equal to the maximum common operating bandwidth supported by all STAs in the BSS. The bandwidth of the primary channel can be configured and / or limited by the STAs among all STAs operating in the BSS that support the minimum bandwidth operating mode. In an 802.11ah example, the primary channel can be 1 MHz wide for STAs (e.g., MTC-type devices) that support (e.g., only) the 1 MHz mode, even if the AP and other STAs in the BSS support 2 MHz, 4 MHz, 8 MHz, 16 MHz, and / or other channel bandwidth operating modes. Carrier sensing and / or Network Allocation Vector (NAV) configuration can depend on the condition of the primary channel. For example, if the primary channel is busy due to a STA (that only supports 1 MHz mode of operation) transmitting to the AP, the entire available frequency band may be considered busy, even though most of the frequency band may remain idle and be available for use.

[0049] In the United States, the available frequency band that can be used by 802.11ah is 902MHz to 928MHz. In South Korea, the available frequency band is 917.5MHz to 923.5MHz. In Japan, the available frequency band is 916.5MHz to 927.5MHz. The total bandwidth available for 802.11ah is 6MHz to 26MHz depending on the country code.

[0050] 1D is a system diagram illustrating the RAN 113 and the CN 115 according to one embodiment. As mentioned above, the RAN 113 may communicate with the WTRUs 102a, 102b, 102c over the air interface 116 using NR radio technology. The RAN 113 may also communicate with the CN 115.

[0051] The RAN 113 may include gNBs 180a, 180b, and 180c, although it will be understood that the RAN 113 may include any number of gNBs while remaining consistent with an embodiment. The gNBs 180a, 180b, and 180c may each include one or more transceivers for communicating with the WTRUs 102a, 102b, and 102c over the air interface 116. In an embodiment, the gNBs 180a, 180b, and 180c may implement MIMO technology. For example, the gNBs 180a and 180b may utilize beamforming to transmit and / or receive signals to the gNBs 180a, 180b, and 180c. Thus, the gNB 180a may, for example, transmit wireless signals to and / or receive wireless signals from the WTRU 102a using multiple antennas. In one embodiment, the gNBs 180a, 180b, and 180c may implement carrier aggregation technology. For example, the gNB 180a may transmit multiple component carriers to the WTRU 102a (not shown). A subset of these component carriers may be on an unlicensed spectrum, and the remaining component carriers may be on a licensed spectrum. In one embodiment, the gNBs 180a, 180b, and 180c may implement coordinated multi-point (CoMP) technology. For example, the WTRU 102a may receive coordinated transmissions from the gNBs 180a and 180b (and / or 180c).

[0052] The WTRUs 102a, 102b, 102c may communicate with the gNBs 180a, 180b, 180c using transmissions associated with a scalable numerology. For example, the OFDM symbol spacing and / or OFDM subcarrier spacing may vary for different transmissions, different cells, and / or different portions of the wireless transmission spectrum. The WTRUs 102a, 102b, 102c may communicate with the gNBs 180a, 180b, 180c using subframes or transmission time intervals (TTIs) of different or scalable lengths (e.g., including different numbers of OFDM symbols and / or lasting different lengths of absolute time).

[0053] The gNBs 180a, 180b, 180c may be configured to communicate with the WTRUs 102a, 102b, 102c in a standalone configuration and / or a non-standalone configuration. In a standalone configuration, the WTRUs 102a, 102b, 102c may communicate with the gNBs 180a, 180b, 180c without accessing another RAN (e.g., eNodeBs 160a, 160b, 160c, etc.). In a standalone configuration, the WTRUs 102a, 102b, 102c may utilize one or more of the gNBs 180a, 180b, 180c as mobility anchor points. In a standalone configuration, the WTRUs 102a, 102b, 102c may communicate with the gNBs 180a, 180b, 180c using signals in unlicensed bands. In a non-standalone configuration, the WTRUs 102a, 102b, 102c may communicate with and connect to gNBs 180a, 180b, 180c while also communicating with and connecting to another RAN, such as eNodeBs 160a, 160b, 160c. For example, the WTRUs 102a, 102b, 102c may implement a DC principle to communicate with one or more gNBs 180a, 180b, 180c and one or more eNodeBs 160a, 160b, 160c substantially simultaneously. In a non-standalone configuration, the eNodeBs 160a, 160b, 160c may act as mobility anchors for the WTRUs 102a, 102b, 102c, while the gNBs 180a, 180b, 180c may provide additional coverage and / or throughput for serving the WTRUs 102a, 102b, 102c.

[0054] Each of the gNBs 180a, 180b, 180c may be associated with a particular cell (not shown) and may be configured to handle radio resource management decisions, handover decisions, scheduling of users in the UL and / or DL, support for network slicing, dual connectivity, interworking between NR and E-UTRA, routing of user plane data to User Plane Functions (UPFs) 184a, 184b, routing of control plane information to Access and Mobility Management Functions (AMFs) 182a, 182b, etc. As shown in FIG. 1D , the gNBs 180a, 180b, 180c may communicate with each other via an Xn interface.

[0055] 1D may include at least one AMF 182a, 182b, at least one UPF 184a, 184b, at least one Session Management Function (SMF) 183a, 183b, and possibly a Data Network (DN) 185a, 185b. While each of the foregoing elements is shown as part of the CN 115, it will be understood that any of these elements may be owned and / or operated by an entity other than the CN operator.

[0056] The AMF 182a, 182b may be connected to one or more of the gNBs 180a, 180b, 180c in the RAN 113 via an N2 interface and may function as a control node. For example, the AMF 182a, 182b may be responsible for user authentication of the WTRUs 102a, 102b, 102c, support for network slicing (e.g., handling different Protocol Data Unit (PDU) sessions with different requirements), selecting a particular SMF 183a, 183b, managing registration areas, terminating NAS signaling, mobility management, etc. Network slicing may be used by the AMF 182a, 182b to customize the CN support of the WTRUs 102a, 102b, 102c based on the type of service utilizing the WTRUs 102a, 102b, 102c. For example, different network slices may be established for different use cases, such as services relying on ultra-reliable low latency (URLLC) access, services relying on enhanced massive mobile broadband (eMBB) access, services for machine type communication (MTC) access, and / or the like. The AMF 162 may provide a control plane function for switching between the RAN 113 and other RANs (not shown) that employ other radio technologies, such as LTE, LTE-A, LTE-A Pro, and / or non-3GPP access technologies, such as WiFi.

[0057] The SMFs 183a and 183b may be connected to the AMFs 182a and 182b in the CN 115 via an N11 interface. The SMFs 183a and 183b may also be connected to the UPFs 184a and 184b in the CN 115 via an N4 interface. The SMFs 183a and 183b may select and control the UPFs 184a and 184b and configure the routing of traffic through the UPFs 184a and 184b. The SMFs 183a and 183b may perform other functions, such as managing and allocating UE IP addresses, managing PDU sessions, controlling policy enforcement and QoS, providing DL data notification, etc. The PDU session type may be IP-based, non-IP-based, Ethernet-based, etc.

[0058] The UPFs 184a, 184b may be connected to one or more of the gNBs 180a, 180b, 180c in the RAN 113 via an N3 interface, which may provide the WTRUs 102a, 102b, 102c with access to packet-switched networks such as the Internet 110 to facilitate communications between the WTRUs 102a, 102b, 102c and IP-enabled devices. The UPFs 184, 184b may perform other functions such as packet routing and forwarding, user plane policy enforcement, support for multi-homed PDU sessions, handling user plane QoS, DL packet buffering, mobility anchoring, etc.

[0059] The CN 115 may facilitate communication with other networks. For example, the CN 115 may include or communicate with an IP gateway (e.g., an IP multimedia subsystem (IMS) server) that acts as an interface between the CN 115 and the PSTN 108. Additionally, the CN 115 may provide the WTRUs 102a, 102b, 102c with access to other networks 112, which may include other wired and / or wireless networks owned and / or operated by other service providers. In one embodiment, the WTRUs 102a, 102b, 102c may be connected to local data networks (DNs) 185a, 185b through the UPFs 184a, 184b via an N3 interface to the UPFs 184a, 184b and an N6 interface between the UPFs 184a, 184b and the DNs 185a, 185b.

[0060] 1A-1D and the corresponding description thereof, one or more or all of the functions described herein with respect to one or more of the WTRUs 102a-d, base stations 114a-b, eNodeBs 160a-c, MME 162, SGW 164, PGW 166, gNBs 180a-c, AMFs 182a-ab, UPFs 184a-b, SMFs 183a-b, DNs 185a-b, and / or any other devices described herein may be performed by one or more emulation devices (not shown). The emulation devices may be one or more devices configured to emulate one or more or all of the functions described herein. For example, the emulation devices may be used to test other devices and / or simulate network and / or WTRU functions.

[0061] The emulation devices may be designed to implement one or more tests of other devices in a lab environment and / or an operator network environment. For example, one or more emulation devices may perform one or more or all functions while fully or partially implemented and / or deployed as part of a wired and / or wireless communication network to test other devices in the communication network. One or more emulation devices may perform one or more or all functions while temporarily implemented / deployed as part of a wired and / or wireless communication network. The emulation devices may be directly coupled to another device for testing purposes and / or may perform testing using terrestrial wireless communication.

[0062] One or more emulation devices may perform one or more functions, inclusive, while not being implemented / deployed as part of a wired and / or wireless communication network. For example, the emulation devices may be utilized in test scenarios in a test lab and / or in an undeployed (e.g., test) wired and / or wireless communication network to implement testing of one or more components. One or more emulation devices may be test equipment. Direct RF coupling and / or wireless communication via RF circuitry (which may include, e.g., one or more antennas) may be used by the emulation devices to transmit and / or receive data.

[0063] The WTRU 120 may include a decoder portion of an autoencoder or an entire autoencoder to enable various embodiments disclosed herein in the WTRU 102 .

[0064] Typical PC data formats The point cloud (PC) data format is a universal data format across many business domains, including autonomous driving, robotics, augmented reality / virtual reality (AR / VR), civil engineering, computer graphics, and / or animation / film. 3D LIDAR sensors can be deployed for self-driving cars. Emerging affordable LIDAR sensors can be implemented in numerous products, for example, the Apple iPad Pro 2020 and / or the Intel RealSense LIDAR Camera L515. With significant advances in sensing technology, 3D PC data can become more practical than ever and can become an enabler (e.g., an ultimate enabler) in the applications described herein.

[0065] It is believed that PC data may consume a large portion of network traffic (e.g., between cars connected via 5G networks and / or for immersive communications such as VR / AR). PC understanding and communication may lead to more efficient representation formats. For example, raw PC data may need to be appropriately organized or may be organized and processed for purposes of 3D world modeling and / or sensing.

[0066] A PC may represent sequential updates of the same scene, which may include one or more moving objects. Such a PC is called a dynamic PC (DPC), as opposed to a static PC (SPC), which may be captured from a static scene or object. DPCs are typically organized into frames, with different frames captured at different times.

[0067] Typical use cases for PC data The automotive industry and autonomous vehicles are also areas where PCs can be used. Autonomous vehicles can "probe" their environment to make better driving decisions based on their immediate surroundings (e.g., the reality of the environment immediately next to / close to the autonomous vehicle). Typical sensors such as LIDAR can generate DPCs that can be used by the decision engine. These PCs may not or are not intended for human viewing, may be small, not necessarily color-coded, and may be dynamic with a high capture frequency. PCs may have other attributes, such as reflectivity provided by LIDAR. Reflectivity can be good information about the material of the sensed object and can provide more information regarding the decision (e.g., which can be useful in making the decision).

[0068] VR and immersive worlds, which may use PCs, are foreseen by many as the future replacement for 2D flat video. With VR and immersive worlds, the viewer can be immersed in an environment (e.g., viewable all around the viewer). This is in contrast to standard TV, where the viewer can only see the virtual world in front of them. There are several levels of immersion depending on the viewer's degrees of freedom within the environment. PCs are a format (e.g., a good format candidate) for delivering VR worlds. PCs for use with VR and immersive worlds may be static or dynamic, and may have an average size ranging, for example, up to 100 million points at a time (e.g., up to a few million points at a time).

[0069] PCs may be used for various purposes, such as cultural heritage / architecture, where objects such as statues or buildings are scanned in 3D, for example, to share the spatial configuration of an object without transmitting and / or visiting the object, and / or to ensure preservation of knowledge about the object in case the object is destroyed (e.g., a temple destroyed by an earthquake). Such PCs are typically static, colored, and may be large in size (e.g., huge, e.g., above a threshold size).

[0070] PCs may be used in topography and / or cartography, where 3D representations and / or maps are not limited to flat surfaces and may include relief (such as showing bumps and depressions). Google Maps is a good example of a 3D map. PCs may be a suitable data format for 3D maps, and such PCs may be static, colored, and / or large (e.g., above a threshold size and / or enormous).

[0071] PC-mediated world modeling and sensing may be a technology (e.g., a useful and / or necessary technology) for enabling machines to gain knowledge about the 3D world around them for applications described herein, for example.

[0072] A typical PC data format. As a general discrete representation of a continuous surface in 3D space, PCs are divided into two categories: organized PCs (OPCs), which are collected by, for example, a camera-like 3D sensor or a 3D laser scanner and arranged on a grid, and unorganized PCs (UPCs). UPCs, for example, may have complex structures. UPCs may be scanned from multiple viewpoints and then fused together, leading to a loss of index ordering. OPCs are easier to process because the underlying grid implies natural spatial connectivity that can reflect the sensing order. Processing UPCs can be more challenging (e.g., due to the fact that UPCs are different from 1D audio data and / or 2D images) and are associated with a regular grid. UPCs can be sparse and irregularly scattered in 3D space, or are usually scattered, which can make traditional grid-based algorithms difficult to handle 3D PCs. For example, convolution operators are well-defined on regular grids and cannot be directly applied to 3D PCs.

[0073] In particular examples, discretized 3D PCs may be implemented to convert PCs (e.g., UPCs) into, among other things, either (1) 3D voxels and / or (2) multi-view images, which may introduce volumetric redundancy and / or one or more quantization artifacts. In one example, a deep neural network-based supervised process may use a point-wise multilayer perceptron (MLP) followed by pooling (e.g., max pooling) to provide / guarantee permutation invariance and achieve success for a range of supervised learning tasks, such as 3D PC recognition, segmentation, and semantic scene segmentation. Those skilled in the art will appreciate that similar techniques may be applied to many other tasks, such as 3D PC detection, classification, and / or upsampling.

[0074] In some representative embodiments, unsupervised learning processes, operations, methods, and / or functions may be implemented, for example, for 3D PCs and / or other implementations, using, among other things, a TearingNet or a Graph Conditional Autoencoder (GCAE). For example, unsupervised learning operations may include learning compact representations of, among other things, 3D PCs, videos, images, and / or audio without labeling information. In this manner, representative features may be extracted (e.g., automatically extracted) from 3D PCs and / or other data representations and applied to any subsequent tasks as auxiliary information and / or prior information. Because labeling large amounts of data (e.g., PC data or other data) can be time-consuming and / or expensive, unsupervised learning can be beneficial.

[0075] In some representative embodiments, an autoencoder may be implemented to reconstruct PCs, for example, based on their compact representations and / or semantic descriptors. For example, given a semantic descriptor corresponding to an object, a PC representing a particular object may be recovered. Such reconstruction may be implemented (e.g., fitted) as a decoder within a general unsupervised learning framework (e.g., an autoencoder), where the encoder may output feature descriptors with semantic interpretations.

[0076] In some representative embodiments, an autoencoder may be implemented to take into account / use topology (e.g., via topology inference and / or topology information). When dealing with PC reconstruction, a graph topology may be implemented to determine / take into account (e.g., explicitly determine / take into account) the relationships between points. A fully connected graph topology may be quite inaccurate in representing the PC topology because it does not follow the object surface, and may be less effective when dealing with objects with high genus and / or scenes with multiple objects. The N in the reconstructed PC may be 2 Since at a given point there are N graph parameters (graph weights) to learn, learning the complete graph may be costly and / or may use a lot of memory and / or computation.

[0077] In some representative embodiments, methods, apparatus, systems, and / or procedures may be implemented to learn (e.g., effectively learn) PC topology representations. The implementations may be useful in PC reconstruction for complex objects / scenes, but may also be applied to weakly supervised PC tasks in classification, segmentation, and / or recognition, among others.

[0078] While many of the examples disclosed herein relate to PC implementations, other implementations are possible as well, such as using graph topologies for image, video, audio, and other data representations that may have topologies associated with them.

[0079] A typical unsupervised learning procedure for PCs Unsupervised learning for PCs may employ an encoder-decoder framework. 3D points may be discretized into 3D voxels, and 3D convolution may be used to design and / or implement the encoder and / or decoder. Discretization may lead to unavoidable discretization errors, and the use of 3D convolution may be expensive. In a particular example, if PointNet is used as the encoder and a fully connected layer is used as the decoder, 3D points may be processed (e.g., directly) and may be effective. In some representative embodiments, methods, apparatus, systems, and / or procedures may be implemented for PC reconstruction that may, for example, use graph topology to improve PC reconstruction without using / requiring a huge amount of training parameters.

[0080] A typical procedure for using autoencoders such as FoldingNet and AtlasNet for PC A FoldingNet decoder is an efficient decoder design / implementation that allows for reduced training parameters compared to fully connected network implementations / designs. A FoldingNet decoder receives semantic descriptors as input (e.g., from an encoder) and learns a projection function that maps a set of 2D sample points into 3D space. The set of 2D points may be periodically sampled across a 2D grid. These operations are efficient (e.g., very efficient) for a single object with a simple topology, but are not good at handling objects with complex topologies or scenes with multiple objects.

[0081] Figure 2 illustrates the high-level structure / architecture of a representative autoencoder (e.g., FoldingNet architecture), which includes an encoder and a decoder, both of which contain neural networks that generate and store learned network node parameters / weights.

[0082] 2, a representative autoencoder 200 may include an encoder 220 and a decoder 260. The encoder 220 may have as input a set of points 210 (e.g., a set of 3D points and / or a point cloud) and as output a descriptor vector 230. The decoder 260 may have as input the descriptor vector 230 and as output a reconstructed point cloud 270. The decoder 260 may include a neural network (NN) and / or a folding module (FM) 250. The input to the NN / FM 250 may consist of and / or include the descriptor vector 230 and a set of points presampled on a grid 240 (e.g., a 2D grid).

[0083] Figure 3 illustrates another representative autoencoder structure / architecture (e.g., an AtlasNet-type architecture).

[0084] Referring to FIG. 3 , a representative autoencoder 300 may include an encoder 320 and a decoder 360. The encoder 320 may have a set of points 310 (e.g., a set of 3D points and / or a point cloud) as input and a descriptor vector 330 as output. The decoder 360 may have the descriptor vector 330 as input and a reconstructed point cloud 370 as output. The decoder 360 may include multiple NN / FMs 350-1, 350-2, . . . 350-K, e.g., in parallel. The input to each NN / FM may consist of and / or include the descriptor vector 330 and a set of points presampled on an N-dimensional grid 340 (e.g., each NN / FM may include a 2D grid 340-1, 340-2, or 340-K). In a particular example, the grids 340-1, 340-2, . . . 340-K may be the same. In other examples, each grid 340 may be different.

[0085] A typical autoencoder 300 (e.g., an AtlasNet-type autoencoder and / or an AtlasNet2-type autoencoder) provides a simple way to handle complex topologies by including multiple K FMs 350 in the decoder 360. In an AtlasNet-type encoder, each FM 350 maps an atlas patch (a 2D grid) to an object part. When the number of patches K is changed, the autoencoder / NN 300 may need to be retrained. As the number of FMs 350 increases (e.g., up to K FMs), the required network size and memory may scale linearly to store the network parameters / data. Setting the number of patches K in advance may make it difficult or impossible to adapt the network to cover PCs with a wide range of complexity. Reconstruction performance may be sensitive to the number of patches (e.g., visual quality may improve with the number of patches, but more artifacts may appear with more parameterization).

[0086] In certain representative embodiments, procedures may be implemented to use topology information (eg, a topology graph) to improve the folding procedure / operation.

[0087] A typical autoencoder for PC (e.g., FoldingNet++ with graph topology inference) FIG. 4 illustrates another representative autoencoder (e.g., FoldingNet++).

[0088] Referring to FIG. 4 , an exemplary autoencoder 400 with graph topology inference (e.g., a FoldingNet++-type autoencoder) can be implemented to enable representation of a topology (e.g., a point cloud PC topology). The autoencoder 400 can include an encoder 420 and a decoder 460. The encoder 420 can have as input a set of points 410 (e.g., a set of 3D points and / or a point cloud) and as output a descriptor vector 430. The decoder 460 can have as input the descriptor vector 430 and can have as output a reconstructed point cloud 470 and / or a fully connected graph 455 associated with the point cloud 410. The decoder 460 can include multiple modules, including a neural network / fractal model (NN / FM) 450 and / or a graph inference module 454. The input to the NN / FM 450 can consist of and / or include the descriptor vector 430 and a set of points presampled on a grid 440. The input to the graph inference module 454 may be an adjacency matrix 452 (e.g., a complete adjacency matrix) and / or a descriptor vector 430 that describes the grid-like graph topology. The output of the graph interference module 454 may be another adjacency matrix / connectivity graph 455 (e.g., a complete adjacency matrix of a learned fully connected graph). The adjacency matrix / connectivity graph 455 and / or the reconstructed point cloud 470 may be input to the graph filtering module 480. The graph filter module 480 may filter the reconstructed point cloud 470 with the graph 455 to generate a final (e.g., refined) reconstructed point cloud 490.

[0089] It is contemplated that the FM, the graph inference module and / or the graph filtering module may be or include one or more NNs.

[0090] A NN can be designed / implemented to capture a graph topology. For example, a fully connected graph 455 can be developed, in which any pair of points can be connected by a graph edge. However, a fully connected graph topology is not a good approximation of a PC topology (e.g., compared to a locally connected graph topology) because it allows connections between distant pairs of points and therefore does not follow the 2D manifold represented by the PCs.

[0091] Compared to the FoldingNet autoencoder structure, the FoldingNet++ autoencoder may include a graph inference module 454 and a graph filtering module 480. The input to the graph inference module 480 may be a complete adjacency matrix describing the grid-like graph topology, and the output of the graph interference module 454 is considered to be another complete adjacency matrix of the learned fully connected graph. The graph filtering module 454 can refine the coarse reconstruction from the folding module (e.g., the deformation module) and output a final reconstruction of the point cloud (PC) 410.

[0092] Compared to the AtlasNet autoencoder structure, the graph inference module 454 of the FoldingNet++ autoencoder may not scale up with complex topologies, and may still use / require large memory and computation due to the huge number of graph parameters (e.g., graph weights). If the number of points in the reconstructed PC is N, then the number of graph parameters is N 2 is.

[0093] In certain representative embodiments, methods, apparatus, systems, operations, and / or procedures may be implemented to enable an autoencoder architecture (e.g., having a TearingNet module) to learn topology-friendly representations (e.g., for PCs, images, video, and / or audio, among other data representations with topology).

[0094] In certain representative embodiments, methods, apparatus, systems, operations, and / or procedures may be implemented to provide a topology for data representation. For example, in one representative method, an explicit representation of PC topology may be implemented by dividing a 2D grid into multiple patches. Unlike the patches in the AtlasNet autoencoder, which are completely independent of each other, the patches in these embodiments may be contained in the same 2D plane and the same coordinate system, with or without overlapping.

[0095] In the case of a FoldingNet autoencoder, a set of points sampled from a 2D grid is provided as input to a folding process to reconstruct PCs from semantic descriptors, which is computationally efficient compared to fully connected networks. For the initial samples from a 2D grid in a FoldingNet autoencoder, the initial samples represent the simplest topology with genus 0. It is observed that the FoldingNet autoencoder cannot properly handle objects with complex topologies or scenes with multiple objects. It is believed that the overly simplified topology of the 2D grid may be the reason for its inability to handle such complex topologies.

[0096] Although graph topologies can be used to approximate PC topologies, two weaknesses have been observed: (1) there is a mismatch between fully connected graph topologies and PC topologies, and (2) graph filtering procedures can fail (e.g., often fail) to correct points that are incorrectly mapped outside the surface.

[0097] In certain representative embodiments, a TearingNet autoencoder (e.g., with a partitioning module and / or a topology-evolving grid representation) may be implemented to align a 2D topology (e.g., an n-1-dimensional grid topology) with a 3D topology (e.g., an n-dimensional PC topology or other n-dimensional topology related to the data representation). For example, a regular 2D grid may be partitioned into multiple patches to provide a 2D grid with patches (e.g., a topology-friendly 2D grid and / or topology-evolving grid representation).

[0098] In a particular representative embodiment, a TearingNet autoencoder can be implemented, promoting the local connectivity graph as a better approximation of the 3D PC topology.

[0099] In certain representative embodiments, a TearingNet autoencoder may be implemented, and a split 2D grid with a modified topology may be set / used as an input to a folding module so that the learned 2D topology can be directly counted / considered in 3D PC reconstruction. For example, a regular 2D grid may be first used as an input to the folding module, and then a modified and / or evolved 2D grid may be used as the next input to the folding module.

[0100] In certain representative embodiments, a T-Net module can be implemented that can generate a modified / evolving grid that can represent (e.g., explicitly represent) a topology (e.g., PC topology) by dividing a regular grid (e.g., a 2D grid) into divided grids (e.g., 2D grids, e.g., evolved 2D grids with one or more patches) that can serve as inputs for a subsequent folding network (F-Net) module or deformation module. For example, based on the divided 2D grid, a local connectivity graph can be constructed that can follow a 3D topology (e.g., a 3D PC topology or other 3D topology). The constructed local connectivity graph can be used to refine the output PCs.

[0101] In certain representative embodiments, an autoencoder (e.g., a TearingNet) can be implemented to enable PC reconstruction for PCs with various topological structures (e.g., PCs with objects having different genus and / or scenes with multiple objects). The autoencoder can generate representations (e.g., codewords) that reflect (e.g., closely reflect) the underlying topology of the input PCs.

[0102] In certain representative embodiments, a multi-stage (eg, two or more stages) training procedure may be implemented to resolve point collapse that may be caused, for example, by the use of chamfer distances.

[0103] In certain representative embodiments, a TearingNet autoencoder / graph conditional autoencoder (GCAE) with multiple iterations (e.g., more than two iterations) can be implemented to process PC scenes and / or other scenes (e.g., video and / or data representations, among others) with complex topologies.

[0104] A typical TearingNet autoencoder FIG. 5 illustrates an additional autoencoder (e.g., a TearingNet autoencoder) and an unsupervised training framework / procedure used with the TearingNet autoencoder.

[0105] Referring to FIG. 5, the TearingNet autoencoder 500 may include an encoder 520 and a decoder 560. The encoder 520 may have as input a set of points 510 (e.g., a set of 3D points and / or a point cloud) and as output a descriptor vector 530. The decoder 560 may have as input the description vector 530 and may have as output a reconstructed point cloud 570 and / or a local connectivity graph 558 associated with the point cloud 510. The decoder 560 may include multiple modules, including one or more neural networks and / or multiple FMs 550-1 and 550-2 and / or a segmentation module 556. The input to the first neural network / FM 550-1 may consist of and / or include the descriptor vector 530 and a set of points presampled on a grid 540. The input to the segmentation module 556 may include the set of points presampled on a grid 540, the descriptor vector 530, and / or the output of the first neural network / FM 550-1. The output of the segmentation module 556 may be combined and / or summed with a set of pre-sampled points on the grid 540 to generate a local connectivity graph 558. The input to the second NN / FM 550-2 may consist of and / or include the descriptor vector 530 and / or the local connectivity graph 558. The NN / FMs 550-1 and 550-2 of the decoder 560 may share the same neural network architecture and the same trained NN parameters. The output to the second NN / FM 550-2 may include a reconstructed point cloud 570. The local connectivity graph 558 and / or the reconstructed point cloud 570 may be input to a graph filtering module 580. The graph filter module 580 may filter the reconstructed point cloud 570 with the graph 558 to generate a final (e.g., refined) reconstructed point cloud 590.

[0106] It is contemplated that the FM, the segmentation module, and / or the graph filtering module may be or include one or more NNs.

[0107] For example, the encoder 520 may be an encoder such as PointNet (e.g., used in a FoldingNet or FoldingNet++ encoder) or any other neural network encoder capable of outputting a descriptor vector 530. The decoder 560 may include one or more F-Net / transformation modules 550 (e.g., one or more F-Net / transformation neural networks), one or more T-Net modules 556 (e.g., one or more T-Net neural networks), and a 2-D grid 540. Inputs to the first F-Net module 550-1 may include the descriptor vector 530 and the initial 2-D grid 540. Inputs to the T-Net module 556 may include the descriptor vector 530, the initial 2-D grid 540, and the output of the first F-Net module 550-1. The output of the T-Net module 556 may include a partitioned 2-D grid 558 (e.g., an evolved 2-D grid and / or a 2-D grid with patches representing the topology of the data representation that generates the descriptor vector via the encoder). Subsequent inputs to the first F-Net module 550-1 or to another F-Net module 550-2 with the same neural network architecture and the same learned NN parameters / weights may include the descriptor vector 540 and the partitioned 2D grid output from the first T-Net module 558. The output of the T-Net module 556 may include a local connectivity graph 558.

[0108] Similar to the F-Net module 550, the transformation module may transform the input to reconstruct the input data representation, such that the F-Net module and the transformation module may be used interchangeably.

[0109] The output of the final F-Net module 550-2 and the final evolving 2D grid 558 may be input to a graph filtering module 580. The output of the graph filtering module 580 may be a final reconstructed PC 590.

[0110] Although two F-Net modules and one T-Net module are shown in Figure 5, any number of F-Net modules (e.g., N F-Net modules) may be implemented in the decoder, and a corresponding number of T-Net modules (e.g., N or N-1 T-Net modules) may be implemented. In particular embodiments, a single F-Net module and a single T-Net module may be implemented in the decoder using an iterative process that generates a series of evolved partitioned 2D grids. Each partitioned 2D grid may be used as input to the F-Net module for one iteration of the reconstructed PC.

[0111] Comparing the TearingNet autoencoder with the FoldingNet autoencoder and FoldingNet++ autoencoder shown in Figures 2 and 4, respectively, several modules can be similarly implemented / designed, including the encoder (E-Net) module, the folding (F-Net) module, the 2D point set as input to the first run of the F-Net module, and the graph filtering (G-Filter) module.

[0112] In certain implementations, the E-Net module is k =(x k, y k ,z k ) as input and output a descriptor vector.

[0113] The descriptor vector may be sent to a decoder that includes an F-Net module and a T-Net module, both of which may be invoked for each 2D point with index k or i.

[0114] For the first run of the F-Net module, the inputs are a descriptor vector f and a 2D grid u using a predefined sampling behavior, e.g., equally spaced and uniformly sampled. (0) i =(u (0) i ,v (0) i ) and the 2D point i from the 2D point i. The F-Net module performs the first reconstruction x (1) i =(x (1) i ,y (1) i ,z (1) i ) can be output. Then, the T-Net module can be invoked. The inputs to the T-Net module are the descriptor vector f, the 2D points iu sampled from the 2D grid, and the (0) i =(u (0) i ,v (0) i ), and the first rebuild of the PC x (1) i =(x (1) i ,y (1) i ,z (1) i For example, the input may include u (0) i =(u (0) i ,v (0) i ), x (1) i =(x (1) i ,y (1) i ,z (1) i ), and the 6-dim gradient vector ∂x (1) i / ∂ (0) i It may also be a concatenated vector from

[0115]

number

[0116] The T-Net module is as follows: (0) i =(u (0) i ,v (0) i ), which can result in a corrected 2D point set as shown in Equation 2.

[0117]

number

[0118] A second run of the F-Net module can be invoked. It is assumed that the F-Net module in this run / run and the F-Net module from the previous run / run can use / share a common F-Net module. For this run, the inputs are a descriptor vector f and a modified 2D grid u (1) i =(u (1) i ,v (1) i ) (e.g., a set of modified 2D points or modified 2D samples). (2) i =(x (2) i ,y (2) i ,z (2) i ) can be output.

[0119] Similar to the F-Net module, the T-Net module may be implemented via a neural network whose parameters are achieved via training based on one or more PC datasets (e.g., a training dataset).

[0120] Corrected 2D sample u (1)i From the second reconstructed PC x, we can construct a nearest neighbor graph G (e.g., a local connectivity graph). (2) i =(x (2) i ,y (2) i ,z (2) i ), graph filtering can be performed using a graph filter that can be based on the nearest neighbor graph G. Graph filtering is used to find the final PC reconstruction.

[0121]

number

[0122] To train a TearingNet autoencoder (e.g., a TearingNet framework), in a particular implementation, the loss function shown in Equation 3 is given by: k} and N-point output PC

[0123]

number

[0124]

number

[0125] Although the loss function is shown as being based on the chamfer distance, other loss functions based on other distance-related measures (e.g., Hausdorff distance or Earthmover distance, among others) are possible.

[0126] Typical T-Net modules FIG. 6 is a diagram of a representative partitioning (T-Net) module.

[0127] Referring to FIG. 6, an exemplary segmentation / T-Net module 600 may include multiple sets (e.g., two or more sets) of N×N convolutional neural networks (CNNs) 610 and 620 (e.g., 3×3 CNNs) and / or one or more multi-layer perceptrons (MLPs) (e.g., fully connected neural networks), among other types of neural networks.

[0128] A codeword f (e.g., a descriptor vector 530) can be replicated N times in an N×512 matrix 630 (e.g., if the codeword f is 512-dim, other dimensions are possible, such as 128, 256, 1024, 2048, or 4096, among others). The replicated matrices 630 from f can be concatenated to generate a first concatenated matrix 640 (e.g., an N×523 matrix including an N×2 matrix 645 from the grid / point 540 (e.g., a 2D grid / point u), an N×3 matrix from the 3D point x, and an N×6 matrix from the gradient 650 (e.g., gradient ∂x / ∂u)). The 3D point x can be an output from the F-Net module 550-1. Each row of the first concatenated matrix 640 (e.g., an N×523 matrix) can be passed through a first neural network 610 (e.g., a shared 3×3 CNN or MLP) of the partitioning / T-Net module 556. The first neural network 610 (e.g., a first CNN) may include or consist of N layers (e.g., three layers). The first concatenation matrix 640 may be input to a first CNN (not shown) of a series of CNNs (not shown). The first series of CNNs may have output dimensions of 256, 128, and 64 for the first, second, and third layers, respectively.

[0129] The input matrix for the second neural network 620 (e.g., the second CNN) in the series of neural networks can be formed, generated, and / or constructed similarly to the previous operation and can include a second concatenation matrix 660 that includes the first concatenation matrix 645 and the 64-dimensional features output from the first CNN 610 (e.g., an N × 64 matrix 655) from the previous operation. The second concatenation matrix 660 (which may be an N × 587 matrix) can be the input matrix N × 587 for the second neural network 620 (e.g., the second CNN or MLP in the series). Each row of the input matrix can be passed through the second CNN 620 (e.g., a shared 3 × 3 CNN or MLP). The second series of CNNs can include or consist of three layers (not shown) with output dimensions of 256, 128, and 2 for the first, second, and third layers, respectively. The final output matrix N×2 665 of the Segmentation / T-Net module 556 may represent a modification / evolution of the 2D grid 540 (e.g., the 2D grid x).

[0130] In comparison with the complexity of FoldingNet++, for the same size of a 2D grid with N points, the input and output dimensions of FoldingNet++ are N + 512 and N, while the input and output dimensions of TearingNet are 11 + 512 and 2. Comparing the complexity of AtlasNet and TearingNet, in AtlasNet, the number of F-Net modules is equal to the preset size of the Atlas, which should or must be large for real scenes. TearingNet only requires / uses a total of one F-Net module and one T-Net module in the decoder, regardless of the complexity of the scene.

[0131] The T-Net module can use a neural network as a mapping function such as:

[0132]

number

[0133] The descriptor f can drive the T-Net module to divide the 2D grid / points into patches. For example, for a PC with three objects, the 2D grid / points may be divided or divided into three patches, and the T-Net module can generate modified / evolved 2D grid / points.

[0134] FIG. 7A illustrates an example input PC. FIG. 7B illustrates an example segmentation / evolution 2D grid associated with the input PC of FIG. 7A. FIG. 7C illustrates an example reconstructed PC associated with the input PC of FIG. 7A. The segmentation 2D grid of FIG. 7B may include patches A1, B1, C1, and D1. The segmentation / T-Net module 556 may generate the segmentation / evolution 2D grid. The input PC includes four objects (e.g., three vehicles (objects A, C, and D) and a cyclist (object B)), and the segmented 2D grid includes segments that roughly correspond to the areas around each object in the input PC.

[0135] Typical sculpture training procedure In certain representative embodiments, a training procedure (e.g., a two-stage sculpture training procedure) may be implemented using, for example, a distance measure (e.g., chamfer distance, earthmoving machine distance, or other distance metric) to train the TearingNet. Chamfer distance is less complex than earthmoving machine distance, but suffers from point collapse issues. The loss function using the chamfer distance in Equation 3 may be rewritten as set forth in Equations 5 and 6, as follows:

[0136]

number

[0137]

number

[0138]

number

[0139]

number

[0140]

number

[0141]

number

[0142]

number

[0143] Starting with training, the network parameters are randomly initialized, causing the reconstructed points to scatter around the space. Given a dataset with a sufficient number of points and sufficient topological structure, the subset distance is likely to be larger than the superset distance and dominate over the superset distance. This can be interpreted / determined by treating the reconstruction as learning the conditional probability of occurrence at each spatial location given a latent codeword. If the shape (e.g., PCs) used for training varies dramatically, the learned distribution may be more evenly spread across the space. Thus, there are more chances that the reconstructed points will fall outside the ground truth input PCs. The subset distance may be less favorable than the superset distance, which can cause the subset distance to dominate during training.

[0144] An unbalanced chamfer distance with a dominant subset distance can lead to point collapse even at the beginning of training. Considering that there is a single shared point between all objects in the dataset, the trivial solution to minimize (drive to zero) the subset distance is to collapse all points into the shared point. Even if there is no intersection between object shapes, the points may collapse into a single point estimator close to the surface, resulting in a trivial solution to minimize the subset distance.

[0145] A sculpting training procedure / strategy may be implemented and may include at least two training stages. In the first stage, a superset distance (e.g., only the superset distance) may be used as a training loss to coarsen a preliminary form. In the second stage, chamfer distances, including subset distances, may be used to refine (e.g., refine) the reconstruction. The sculpting training procedure for training a TearingNet may resemble a subtractive sculpting procedure / process. After a coarse form is constructed / generated from the first stage, the T-Net module may cut (e.g., specifically cut) unnecessary material for the final image in the second stage and generate a partitioned 2D grid (e.g., including patches as shown in FIG. 7B). A two-stage sculpting training procedure may include, for example, the following: (1) Training the F-Net module under the FoldingNet architecture with the superset distance loss function (in a specific embodiment, the learning rate is r = 10 -3 ) and (2) Load the pre-trained F-Net module into the TearingNet architecture and continue training the F-Net and T-Net modules using chamfer distance as the loss function (e.g., counting both superset and subset distances, and setting the learning rate to a smaller value, e.g., r = 10). -3 r1=10 -6 can be adjusted to be

[0146] Representative Iterative TearingNet Architectures / Implementations FIG. 8 illustrates a representative recurrent TearingNet architecture that supports multiple iterations. Referring to FIG. 8, the recurrent TearingNet 800 may include modules that are the same as or similar to those in FIG. 6. For example, the recurrent TearingNet 800 may include an encoder 820 and a decoder 860, which may include a T-Net module 856 and an F-Net module 850, and may use an evolutionary 2D grid 858. Using a loop structure, the F-Net module 850 and the T-Net module 856 can perform any number of iterations (e.g., several iterations). In each iteration, the F-Net module 850 may take the 2D grid 858 output from the T-Net module 850 from the previous iteration as one input to the F-Net module 850, and the T-Net module 856 may take the 3D points (and gradients) output from the F-Net module 856 from the current iteration as input to the T-Net module 856. A TearingNet 800 with multiple iterations can be used to handle difficult (e.g., even more difficult) object / scene topologies.

[0147] The input to the encoder 820 may be or include, for example, a point cloud 810.

[0148] The encoder 820 may output a descriptor vector 830. In a first operation / step of a first iteration of the iterative TearingNet 800, shown in FIG. 8 as the first step dashed line, the F-Net module 850 may receive inputs from the descriptor vector 830 and an initial 2D grid 858-1. The initial 2D grid 858-1 may be output as a local connectivity graph. In a second operation / step of a first iteration of the iterative TearingNet 800, shown in FIG. 8 as the second step dashed line, the T-Net 856 may receive as inputs the output of the F-Net 850 from the first operation, the descriptor vector 830, and the initial 2D grid 858-1. The output of the F-Net 850 in the second operation / step may be a reconstructed point cloud 870. Third Step: In the third operation / step of the first iteration of the iterative TearingNet 800, shown in FIG. 8 as the dashed line, the T-Net 856 can output a first modified 2D grid 858-2.

[0149] In the first operation / step of the second iteration of the iterative TearingNet 800, shown in FIG. 8 as the first-step dashed line, the F-Net module 850 may receive inputs from the descriptor vector 830 and the first modified 2D grid 858-2. The first modified 2D grid 858-2 may be output as a local connectivity graph. In the second operation / step of the second iteration of the iterative TearingNet 800, shown in FIG. 8 as the second-step dashed line, the T-Net 856 may receive as inputs the output of the F-Net 850 from the first operation in the second iteration, the descriptor vector 830, and the first modified 2D grid 858-2. The output of the F-Net 850 in the second operation / step of the second iteration may be the first modified reconstructed point cloud 870. Third Step: In the third operation / step of the second iteration of the iterative TearingNet 800, shown in FIG. 8 as the dashed line, the T-Net 856 can output a second modified 2D grid 858-3.

[0150] For each iteration, the output of the 2D grid / modified 2D grid (e.g., the current local connectivity graph 858-1, 858-2, or 858-3 and the reconstructed or modified reconstructed point cloud 870) may be input to a graph filtering module 880 to provide graph filtering and generate the final reconstructed point cloud.

[0151] Although two iterations are shown in Figure 8, any number of iterations of the TearingNet 800 are possible.

[0152] In certain representative embodiments, the initial set of points may be sampled periodically across a 2D grid (e.g., first / initial 2D grid 858). A spherical or cubic surface may be selected to replace the 2D grid, and / or the 2D grid may be replaced with an N-dimensional grid. In certain embodiments, another sampling operation may replace the uniform sampling on the surface.

[0153] TearingNet 800 can provide an unsupervised learning framework. A procedure for reconstructing such a data representation of PCs is disclosed herein and may include an initial training operation in which neural network weights / parameters are established for the E-Net, T-Net, and F-Net modules in an end-to-end operation. After the initial training operation, the encoder 820 and decoder 860 of the autoencoder 800 (e.g., with established neural network weights / parameters) can be operated separately. It is believed that the descriptor f can function as a topology-aware representation. TearingNet 800 can enable the encoder 820 to output a descriptor in a feature space that is more friendly to object / scene topology. Such a topology-aware representation can benefit many tasks, such as object classification, segmentation, detection, and scene completion, by alleviating the need for labeled data. TearingNet can be useful in PC compression because it provides a different method for reconstructing PCs.

[0154] In certain representative embodiments, a neural network may be implemented using a T-Net module to learn topology-friendly representations related to data representations such as PCs, videos, images, and / or audio, among others. For example, by using evolving 2D grids / points, the neural network can handle objects / scenes with complex topologies. The neural network may reside within the decoder portion of an end-to-end autoencoder for unsupervised learning. In other representative embodiments, a sculpted training procedure / strategy may enable, for example, better tuned neural network weights / parameters.

[0155] Representative design / architecture of an integrated T-Net and second F-Net module In certain embodiments, the functions associated with the first iteration of the T-Net module and the second iteration of the F-Net module may be implemented in a unified architecture / module (e.g., a combined partition-folding network (TF-Net) architecture / module). The input to the TF-Net module may be configured similarly to the input to the F-Net module, e.g., a latent codeword and a 2D point set from a 2D grid. The output of the TF-Net module may be a 3D point correction. For the final PC reconstruction, the 3D correction may be applied to the output from the first F-Net module. The TF-Net module can be viewed as a direct partition in 3D space instead of a partition of a 2D grid. For example, an advantage of implementing a TF-Net module may be a simplified overall architecture compared to the architecture of FIG. 8.

[0156] Representative GCAE FIG. 9 illustrates an exemplary GCAE 900. Referring to FIG. 9, GCAE highlights how it facilitates topology learning for general data types, such as in a TearingNet with multiple iterations. GCAE 900 may include the same or similar modules as TearingNet 800, e.g., an encoder E and a decoder D. Decoder D may include a folding module F and a segmentation module T. The output of encoder E may be a descriptor vector c, which may be input to decoder D. The output of decoder D is a reconstructed data representation

[0157]

number

[0158]

number

[0159] The GCAE 900 may include a folding module F in a loop structure with a division module T. The input to the folding module F may be modified for each iteration. Initially, a 2D grid u may be input to the folding module F. In the second and subsequent iterations, the output Δu may be combined (e.g., summed with the initial 2D grid u) to produce

[0160]

number

[0161] Instead of a two-module traditional autoencoder, the GCAE may include a three-module architecture / design that may include an encoder module (e.g., an E-Net module (E)), a folding module (e.g., an F-Net module (F)), and a segmentation module (e.g., a T-Net module (T)). As shown in various figures, graphs with specific initialization may also be implemented. The graph may explicitly represent the topology of the data representation in the decoding operation (e.g., the decoding computation).

[0162] In the decoder D of the autoencoder in FIG. 9, the F-Net module and the T-Net module are interfaced (e.g., they iteratively interact with each other). During the interaction, the F-Net module can embed a graph topology into the reconstructed signal. For example, if a signal (e.g., an image or PC) is sampled in the spatial domain, the topology can be implicitly represented by the relationship of the sampling points (pixels and / or points). The T-Net module can extract the implicit topology from the reconstructed signal and represent the topology in the graph domain. The output of the T-Net module (e.g., the direct output of the T-Net module) can be selected as a modification to the original graph to more easily converge training for an optimal configuration.

[0163] In a practical system, the number of iterations may be signaled, explicit, or predetermined, and the graph topology is thought to evolve with each iteration.

[0164] The TearingNet of the PC autoencoder disclosed herein is an example of a GCAE, and those skilled in the art will understand from the TearingNet how GCAE can be utilized to learn topology-friendly representations of signals (e.g., data representations) such as PCs. GCAE can provide benefits (e.g., clear benefits) when the PCs are for objects with high genus or scenes with multiple objects.

[0165] Typical Design / Architecture of T-Net Modules The T-Net module can be implemented in several different ways, including using an MLP network as a building block. In an MLP implementation, the gradient of the output of the F-Net module with respect to the graph can be useful because the gradient provides neighborhood information. In other embodiments, the T-Net module may be implemented using one or more CNNs (e.g., using convolutional neural network layers, e.g., using a 3x3 convolution kernel as the design / architecture). Such kernels may take context into account and may or may not skip the introduction / use of gradients as input to the T-Net module.

[0166] A representative GCAE procedure for human action recognition. Human skeletons can be detected in various ways, which is often used for human action recognition. Autoencoders can be considered for the task of human action recognition. The input signal may be a sequence of 2D (or 3D) coordinates of the human skeleton. A codeword from the E-Net module may be used for action recognition. The GCAE decoder (including the F-Net module) and the T-Net module are considered to be able to reconstruct the human skeleton from the codeword. For example, in a specific embodiment, for this task, an initial graph topology may be selected according to the joint connections of the human body. Graph weights on the connections may be updated from the output of the T-Net module. The F-Net module may be implemented / designed to take a graph as input and predict the coordinates of the skeletal joint positions. Because the skeletal graph contains a fairly small number of points (joints), the graph input to the F-Net module can be arranged as an adjacency matrix of the graph. It is considered that both the F-Net module and the T-Net module can also receive a codeword as input in addition to the graph. For brevity, codeword processing will not be discussed in detail. The focus will be on the context of topology. The loss function may be defined as the mean squared error between the input data representation for the skeleton and the output data representation for the skeleton. For example, the error at each joint may be calculated, and then the mean squared error may be calculated.

[0167] Representative GCAE Procedures for Image Search and Retrieval For image search and retrieval applications, it may be useful / necessary to identify communities within an image dataset. In image search and retrieval applications, the image dataset can be considered as a context. To apply GCAE, an image can be input to an E-Net module, which outputs a codeword. The decoder can initialize a graph representing the similarity of the input image to other images in the dataset. The F-Net module can predict a score of the similarity of the input image to each image in the image dataset. The T-Net module can take the predicted scores as input and update the graph so that it can better predict the similarity topology. Finally, a loss function can be defined as the image similarity between the input image and the image with the highest score. A graph topology across an image dataset is actually an asset (e.g., a critical asset) for search and retrieval applications. GCAE can be used to build and refine such a topology. Thus, the graph topology may be the output of a GCAE decoder after running a query within an image dataset.

[0168] Representative GCAE procedures for image analysis For image analysis applications, topology within an image is an asset (e.g., a key asset). Extracting image representation descriptions can be a target of the application. A GCAE design / architecture can be implemented to learn representations for image retrieval. An E-Net module can take an image as input and generate latent codewords for the image. The E-Net module can select a known image feature extractor, e.g., AlexNet, ResNet, etc. The decoder design / architecture can drive / modify the encoder output (e.g., via setting neural network weights during training) through end-to-end training. Because image pixels are organized in 2D, a graph can be initialized as a 2D grid. Graph edges can be constructed between adjacent pixels with constant weights (e.g., only between adjacent pixels). An F-Net module can take a graph as input in addition to a codeword and generate an image as output. A T-Net module can estimate graph modifications from the output image.

[0169] The loss function between the input and output images can be calculated based on the mean squared error (MSE) or another distance-based error function. Resampling is assumed to match the input and output resolutions to facilitate the calculation of the MSE.

[0170] A representative GCAE procedure for image coding. Similar to image search and retrieval applications, for image coding, identifying similar image patches to remove redundancy is useful / necessary. GCAE can be adapted to facilitate block-based image coding, where an image can be divided into blocks for encoding / compression (e.g., for encoding / compression purposes). In addition to embodiments similar to those for image analysis, different graph topologies may be selected to be learned. For example, a 1D graph (e.g., a line graph) may be applied as an image block for encoding a small picture. For example, imaging (e.g., image coding) of a small picture may be completed using a single stroke. The loss function can be defined in the same manner as described earlier in this specification.

[0171] A representative GCAE procedure for video coding Compared to image coding, video coding differs due to, for example, inter-frame prediction, which introduces a third dimension (e.g., temporal direction). In some embodiments, the evolutionary topology generated by iterations in the GCAE decoder can be used to encode motion fields between image frames. It is possible to handle groups of frames and / or groups of pictures (GOPs) within one framework. For example, the input to the video coding GCAE may be a GOP. Each iteration of the GCAE decoder can output a frame within the GOP. In this example, the graph may be initialized as an image with all pixels equal to 0. A T-Net module can decode the motion field, and an F-Net module can apply the motion field to the previous frame. In certain embodiments, the GOP may be modified into a smaller volume in the temporal direction, and this modified GOP may be referred to as a group of blocks (GOB).

[0172] A typical GCAE procedure for scene analysis GCAE and / or TearingNet can be used for scene analysis, including, for example, object counting and detection. The codeword obtained from the encoder (E-Net) module characterizes the topology of the input scene. For example, two scenes with similar topologies should have similar codewords. The codewords created / generated by GCAE can enable scene analysis tasks such as object counting and / or detection. For example, a classifier can be trained taking the codewords as input and output the number of objects in the scene. In addition to or instead of the classifier output, the partitioned 2D grid can also be used to perform object counting and / or detection, for example, based on the detected patches.

[0173] A representative GCAE procedure for PC coding With respect to PC coding, those skilled in the art will understand that the examples herein for image coding and / or video coding also apply (e.g., apply in principle). These procedures can be used to code static PCs and / or dynamic PCs.

[0174] FIG. 10 is a block diagram illustrating a representative method (eg, implemented by a neural network-based decoder (NNBD)).

[0175] 10 , a representative method 1000 may include, at block 1010, an NNBD obtaining or receiving a codeword as a descriptor of an input data representation. At block 1020, a first neural network (NN) module of the NNBD may determine a preliminary reconstruction of the input data representation based at least on the codeword and an initial graph. At block 1030, the NNBD may determine a modified graph based at least on the preliminary reconstruction and the codeword. At block 1040, the first NN module may determine a refined reconstruction of the input data representation based at least on the codeword and the modified graph. For example, the modified graph may indicate topological information associated with the input data representation.

[0176] In certain representative embodiments, the modified graph may be determined by combining the initial graph and the output of the second NN module.

[0177] In certain representative embodiments, the modified graph may be a local connectivity graph.

[0178] In certain representative embodiments, the NNBD may generate a concatenation matrix for processing by one or more convolutional neural networks (CNNs) by concatenating at least (1) the replicated codewords, (2) the initial graph or the modified graph, and (3) the reconstructed data representation. For example, the NNBD may use the generated concatenation matrix to perform a series of convolutional layer operations. The kernel size for each convolutional layer operation may be a (2n+1)×(2n+1) kernel size, where n is a non-negative integer.

[0179] In certain exemplary embodiments, the input data representation may be or include any of: (1) a point cloud, (2) an image, (3) video, and / or (4) audio.

[0180] In certain exemplary embodiments, the NNBD may be or include a graph-conditional NNBD.

[0181] In certain representative embodiments, the determination of the refined reconstruction of the input data representation may be performed through multiple iterative operations of at least the first NN module.

[0182] In certain representative embodiments, the NNBD may include either one or more convolutional neural networks (CNNs) or one or more multi-layer perceptrons (MLPs).

[0183] In certain representative embodiments, the NNBD may include one or more multi-layer perceptrons (MLPs). For example, the refined reconstruction of the modified graph and / or data representation may be based on or further based on gradient information generated by one or more MLPs.

[0184] In certain representative embodiments, the NNBD may identify any of the following according to the topology information indicated by the modified graph: (1) one or more objects represented in the input data representation, (2) the number of objects, (3) object surfaces represented in the input data representation, and / or (4) motion vectors associated with the objects represented in the input data representation.

[0185] In certain exemplary embodiments, the codeword may be a descriptor vector that represents an object or a scene having multiple objects.

[0186] In certain exemplary embodiments, the initial graph and the modified graph may be two-dimensional (2D) point sets, and the input data representation may be a point cloud.

[0187] In certain representative embodiments, determining a preliminary reconstruction of the input data representation may include the NNBD performing a deformation operation based on the descriptor vector and a 2D point set initialized with a predetermined sampling in the plane.

[0188] In certain representative embodiments, determining a preliminary reconstruction of the input data representation may include NNBD generating a preliminary reconstruction of the point cloud.

[0189] In certain representative embodiments, determining the modified graph may include the NNBD performing a partitioning operation based on the point cloud, the descriptor vector, and a preliminary reconstruction of the initial graph to generate the modified graph.

[0190] In certain representative embodiments, the NNBD may generate the modified graph as a local connectivity graph.

[0191] In certain representative embodiments, the NNBD may perform graph filtering on the refined reconstruction of the input data representation and / or may output the filtered refined reconstruction of the input data representation as a final reconstruction of the input data representation.

[0192] In certain representative embodiments, the local connectivity graph may be constructed based on: (1) generating graph edges for nearest neighbors in the initial graph or the modified graph, (2) assigning graph edge weights based on point distances in the modified graph, and / or (3) pruning graph edges with graph weights less than a threshold.

[0193] In certain representative embodiments, performing graph filtering on the refined reconstruction of the input data representation may include generating a smoothed and reconstructed input data representation such that the final reconstruction of the input data representation is smoothed in the graph domain.

[0194] In some representative embodiments, the NNBD may set neural network weights within the NNBD according to a two-stage training operation. For example, in a first stage of the two-stage training operation, a first NN module may be trained using a superset distance included in a first-stage loss function, and in a second stage of the two-stage training operation, the first NN module and the second NN module may be trained using a chamfer distance included in a second-stage loss function based on the subset distance and the superset distance.

[0195] In certain exemplary embodiments, the initial graph may be a 2D grid that includes a matrix of points, where each point represents a 2D location. For example, the 2D grid may be associated with a manifold, where each point represents a fixed location on the manifold, and / or the 2D grid may be a fixed set of sampled points from a 2D plane.

[0196] In some representative embodiments, determining the modified graph comprises: (1) K iterations of the received or obtained codeword to generate a KxD codeword matrix, where K is the number of nodes in the initial graph and D is the length of the codeword; and (2) KxD codeword matrix and , and the initial graph as a KxN matrix, The method may include any of (1) generating a Kx(D+N) connectivity matrix; (2) inputting the connectivity matrix to one or more CNNs and / or MLPs; (3) generating a modified graph by the one or more CNNs or MLPs from the connectivity matrix; and / or (4) updating the refined reconstruction of the input data representation based on the modified graph to generate a final reconstruction of the input data representation.

[0197] In certain representative embodiments, the NNBD may concatenate the codeword matrix as a concatenated intermediate matrix to the output of a first set of CNN layers or MLP layers, and / or may input the concatenated intermediate matrix to a next set of CNN layers or MLP layers following the first set of CNN layers or MLP layers.

[0198] FIG. 11 is a block diagram illustrating an exemplary training method using multi-stage training motions.

[0199] 11 , a representative method 1100 may include, at block 1110, in a first stage of a multi-stage training operation, training a first NN (e.g., a first NN module) using a first loss function. At block 1120, in a second stage of the multi-stage training operation, the first NN (e.g., the first NN module) and a second NN (e.g., a second NN module) interfaced to the first NN may be trained using a second loss function. For example, the first loss function may be based on a superset distance, and the second loss function may be based on a subset distance and a superset distance. In some examples, the first NN may include a folding module, and the second NN may include a splitting module.

[0200] In certain representative embodiments, in a first stage of a multi-stage training operation, the training may include iteratively determining values of parameters associated with nodes in a first NN that satisfy a first loss condition related to a difference between the input data representation and the reconstructed input data representation, and / or in a second stage of the multi-stage training operation, the training may include iteratively determining values of parameters associated with nodes in the first and second NNs that satisfy a second loss condition related to a difference between the input data representation and the reconstructed input data representation. For example, the determined values associated with nodes in a first NN in the first stage of the multi-stage training operation may be values originally used for nodes of the first NN in the second stage of the multi-stage training operation.

[0201] FIG. 12 is a block diagram illustrating another exemplary method (eg, implemented by an NNBD).

[0202] 12 , a representative method 1200 may include, at block 1210, an NNBD obtaining or receiving a codeword as a descriptor of an input data representation. At block 1220, the NNBD may determine a preliminary reconstruction of the input data representation based on the codeword. At block 1230, the NNBD may determine a modified graph based on (1) an initial graph associated with the input data representation, (2) the preliminary reconstruction of the input data representation, and (3) the codeword. The modified graph may indicate topological information associated with the input data representation.

[0203] In certain representative embodiments, the modified graph, evolved graph, and / or refined modified graph may be output and used to provide topological information related to the input data representation.

[0204] In certain representative embodiments, the NNBD may identify any of the following according to the topology information indicated by the modified graph: (1) one or more objects represented by the input data representation, (2) the number of objects, (3) the object surfaces represented by the input data representation, and / or (4) the motion vectors of the objects represented by the input data representation.

[0205] In certain representative embodiments, the NNBD may determine a refined reconstruction of the input data representation based on the codeword and the modified graph, and / or may determine a refined modified graph based on (1) the modified graph, (2) the refined reconstruction of the input data representation, and (3) the codeword, where the refined modified graph may indicate refined topological information associated with the input data representation.

[0206] FIG. 13 is a block diagram illustrating a further exemplary method (e.g., implemented by a neural network-based autoencoder (NNBAE)) including, for example, an encoding network (E-Net) module and a neural network-based decoder (NNBD).

[0207] Referring to FIG. 13 , a representative method 1300 may include, at block 1310, an E-Net module of the NNBAE determining a codeword as a descriptor of the input data representation based on the input data representation. At block 1320, an F-Net / folding module of the NNBAE may determine a preliminary reconstruction of the input data representation based on at least the codeword and an initial graph having K points. At block 1330, a T-Net / splitting module of the NNBD may determine a modified N-graph evolved from the initial graph based on at least the codeword and the initial graph. At block 1340, an F-Net module of the NNBD may determine a refined reconstruction of the input data representation based on at least the codeword and the modified graph. The modified graph may indicate topological information related to the input data representation, and the E-Net module may be trained jointly with the NNBD.

[0208] FIG. 14 is a block diagram illustrating an additional exemplary method (eg, implemented by an NNBD).

[0209] Referring to FIG. 14 , a representative method 1400 may include, at block 1410, an NNBD obtaining or receiving a codeword as a descriptor of an input data representation. At block 1420, a first NN and / or Folding Network (F-Net) module may determine a preliminary reconstruction of the input data representation based on at least the codeword and an N-dimensional point set having K points, where N is an integer. At block 1430, the NNBD may determine a modified N-dimensional point set evolved from the N-dimensional point set based on at least the codeword and the N-dimensional point set. At block 1440, the first NN and / or F-Net module may determine a refined reconstruction of the input data representation based on at least the codeword and the modified N-dimensional point set. The modified N-dimensional point set may indicate topology information associated with the input data representation.

[0210] In some representative embodiments, the second NN and / or partitioning network (T-Net) module may determine a modification to the N-dimensional point set based on at least the codeword and the N-dimensional point set. Determining the modified N-dimensional point set may include combining the M-dimensional point set with the modification to the N-dimensional point set to generate the modified N-dimensional point set.

[0211] In certain representative embodiments, determining the modifications to the N-dimensional point set may include any of: (1) concatenating the duplicated codeword with the N-dimensional point set as a concatenation matrix; (2) inputting the concatenation matrix into one or more CNNs; (3) generating a second set of points in the M-dimensional feature space by one or more CNNs from the concatenation matrix; (4) concatenating the duplicated codeword, the N-dimensional point set, and the second set of points as a second concatenation matrix; and / or (5) generating modifications to the N-dimensional point set by one or more CNNs from the second concatenation matrix.

[0212] In certain representative embodiments, the NNBD may use one or more NNs to perform a series of convolutional layer operations on the concatenated matrix to generate a modified N-dimensional point set, where the kernel size of each convolutional layer operation may be one of, among others: (1) a 1×1 kernel size; (2) a 3×3 kernel size; and / or (3) a 5×5 kernel size.

[0213] In certain exemplary embodiments, the input data representation may be or include any of the following: (1) a point cloud, (2) an image, (3) a video, or (4) audio.

[0214] In certain exemplary embodiments, N is equal to 2 and the input data representation may be or include a point cloud.

[0215] In certain exemplary embodiments, the NNBD may be or include a graph-conditional NNBD.

[0216] In some examples, the determination of the refined reconstruction of the input data representation may be performed through the iterative operation of at least an F-Net module.

[0217] In certain representative embodiments, an NNBD may include any of one or more CNNs and / or one or more MLPs.

[0218] In certain representative embodiments, the NNBD may include one or more MLPs. For example, the modified N-dimensional point set may be further based on gradient information generated by one or more MLPs.

[0219] In certain representative embodiments, the NNBD may identify one or more objects represented in the input data representation according to the topological information indicated by the modified N-dimensional point set. For example, the NNBD or another device may use the topological information to identify one or more objects in the input data representation and / or identify some objects represented in the input data representation according to the topological information indicated by the modified N-dimensional point set.

[0220] As another example, the NNBD or another device may identify an object surface represented in the input data representation according to the topological information indicated by the modified N-dimensional point set.

[0221] In certain representative embodiments, the NNBD may determine patches from the modified N-dimensional point set that identify different topological regions of the input data representation.

[0222] In certain exemplary embodiments, a codeword may be or include a descriptor vector that represents an object or a scene having multiple objects.

[0223] In certain representative embodiments, the N-dimensional point set may be or include a 2D point set, for example, the input data representation may be or include a point cloud, and / or determining a preliminary reconstruction of the input data representation may include performing a deformation operation based on the descriptor vector and a 2D point set initialized with a predetermined sampling in a plane.

[0224] In certain representative embodiments, determining a preliminary reconstruction of the input data representation may include generating a preliminary reconstruction of a point cloud.

[0225] In certain representative embodiments, determining the evolved modified N-dimensional point set from the 2D point set may include performing a segmentation operation based on a preliminary reconstruction of the point cloud, the descriptor vector, and the 2D point set, and / or generating the modified N-dimensional point set as a modified 2D point set from the 2D point set.

[0226] In certain representative embodiments, the NNBD can generate a local connectivity graph based on the 2D point set and the modified 2D point set.

[0227] In certain representative embodiments, the NNBD or another device (e.g., a graph filter) may construct / implement graph filtering (e.g., perform graph filtering using the graph filter generated on the refined reconstruction of the point cloud from the F-Net module and / or output the filtered refined reconstruction of the point cloud).

[0228] In certain representative embodiments, the local connectivity graph may be constructed based on: (1) generating graph edges for nearest neighbors in the 2D point set, (2) assigning graph edge weights based on point distances in the modified 2D point set, and / or pruning graph edges with graph weights less than a threshold.

[0229] In certain representative embodiments, performing graph filtering on the refined reconstruction of the point cloud may include generating a smoothed reconstructed refined point cloud such that the refined reconstructed point cloud may be smoothed in the graph domain.

[0230] In some representative embodiments, the NNBD may set neural network weights within the NNBD according to a two-stage training operation. For example, in a first stage of the two-stage training operation, the F-Net module may be trained using the superset distance as a loss function, and / or in a second stage of the two-stage training operation, the F-Net module and the T-Net module may be trained using the chamfer distance as a loss function based on the superset distance and the subset distance.

[0231] In certain representative embodiments, the N-dimensional point set may be or may include a 2D grid comprising a matrix of points, where each point may represent a 2D location. For example, the 2D grid may be associated with a manifold, where each point may represent a fixed location on the manifold, and / or the 2D grid may be a fixed set of points sampled from a 2D plane, sphere, or cubic box surface as the manifold.

[0232] In certain representative embodiments, the NNBD may replicate received or acquired codewords to generate a codeword matrix of replicated codewords, which may be the size of a 2D grid, and / or may concatenate the codeword matrix into a concatenation matrix.

[0233] In certain representative embodiments, determining the modified N-dimensional point set may include any of: concatenating a K×D matrix from the replicated codeword with a K×N matrix from the N-dimensional point set to generate a K×(D+N) concatenation matrix; inputting the concatenation matrix into one or more CNNs and / or MLPs; generating modifications to the N-dimensional point set by the one or more CNNs and / or MLPs from the concatenation matrix; and / or updating the N-dimensional point set based on the modifications to generate the modified N-dimensional point set.

[0234] In certain representative embodiments, the NNBD may either (1) concatenate a K×D matrix from the replicated codeword to the output of the first CNN or MLP layer, and / or (2) input the concatenation matrix to the next CNN or MLP layer following the first CNN or MLP layer.

[0235] FIG. 15 is a block diagram illustrating an exemplary training method (eg, implemented by a neural network (NN)) that uses a multi-stage training operation.

[0236] 15, a representative method 1500 may include a first neural network of a NN trained using a superset distance as a loss function in a first stage of a multi-stage training operation at block 1510. At block 1520, in a second stage of the multi-stage training operation, the first neural network and a second neural network interfaced to the first neural network may be trained using a chamfer distance as a loss function based on the superset distance and the subset distance.

[0237] FIG. 16 is a block diagram illustrating an exemplary training method (e.g., implemented by NNBAE, which includes an E-Net module and an NNBD).

[0238] Referring to FIG. 16 , a representative method 1600 may include, at block 1610, determining, by an E-Net module, a codeword as a descriptor of the input data representation based on the input data representation. At block 1620, an F-Net module of the NNBD may determine a preliminary reconstruction of the input data representation based on at least the codeword and an N-dimensional point set having K points, where N is an integer. At block 1630, the NNBD may determine a modified N-dimensional point set evolved from the N-dimensional point set based on at least the codeword and the N-dimensional point set. At block 1640, the F-Net module may determine a refined reconstruction of the input data representation based on at least the codeword and the modified N-dimensional point set. For example, the modified N-dimensional point set may indicate topological information associated with the input data representation, and / or the E-Net may be trained jointly with the NNBD.

[0239] In certain representative embodiments, the NNBD or another device may identify one or more objects represented in the input data representation according to the topology information embedded in the topology-friendly codeword.

[0240] In certain representative embodiments, the NNBD or another device may identify certain objects represented in the input data representation according to the topology information embedded in the topology-friendly codeword.

[0241] In certain representative embodiments, a partitioning network (T-Net) module can determine modifications to the N-dimensional point set based on at least the codeword and the N-dimensional point set. For example, determining the modified N-dimensional point set can include combining the M-dimensional point set with modifications to the N-dimensional point set to generate the modified N-dimensional point set.

[0242] Systems and methods for processing data according to exemplary embodiments may be performed by one or more processors executing sequences of instructions contained in memory devices. Such instructions may be read into the memory devices from other computer-readable media, such as secondary data storage devices. Execution of the sequences of instructions contained in the memory devices causes the processors to operate, for example, as described above. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions to implement the invention.

[0243] The hardware (e.g., processor, GPU, or other hardware) and appropriate software may include, among others, perceptual neural network architectures, feedforward neural network architectures, radial basis network architectures, deep feedforward neural network architectures, recurrent neural network architectures, long / short-term memory neural network architectures, gated recurrent unit neural network architectures, autoencoder (AE) neural network architectures, variational AE neural network architectures, denoising AE neural network architectures, sparse AE neural network architectures, Markov chain neural network architectures, Hopfield network neural network architectures, The neural network may implement one or more neural networks with various architectures, such as a Boltzmann machine (BM) neural network architecture, a restricted BM neural network architecture, a deep belief network neural network architecture, a deep convolutional network neural network architecture, a deconvolutional network architecture, a deep convolutional inverse graph network architecture, a generative adversarial network architecture, a liquid state machine neural network architecture, an extreme learning machine neural network architecture, an echo state network architecture, a deep residual network architecture, a Kohonen network architecture, a support vector machine neural network architecture, and a neural Turing machine neural network architecture. Each cell in the various architectures may be implemented as a backfeed cell, an input cell, a noisy input cell, a hidden cell, a stochastic hidden cell, a spiking hidden cell, an output cell, a match input output cell, a recurrent cell, a memory cell, a different memory cell, a kernel cell, or a convolution / pooling cell. Subsets of the cells of a neural network may form multiple layers.These neural networks can be trained manually or through an automated training process.

[0244] While features and elements are described above in particular combinations, those skilled in the art will understand that each feature or element may be used alone or in any combination with the other features and elements. Furthermore, the methods described herein may be implemented in a computer program, software, or firmware embodied in a computer-readable medium for execution by a computer or processor. Examples of non-transitory computer-readable storage media include, but are not limited to, read-only memory (ROM), random-access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, and optical media such as magneto-optical media and CD-ROM disks and digital versatile disks (DVDs). A processor in association with software may be used to implement a radio frequency transceiver for use in the WTRU 102, UE, terminal, base station, RNC, or any host computer.

[0245] Furthermore, in the above embodiments, processing platforms, computing systems, controllers, and other devices including processors are described. These devices may include at least one central processing unit ("CPU") and memory. In accordance with the practices of those skilled in the art of computer programming, references to acts and symbolic representations of operations or instructions may be performed by various CPUs and memories. Such acts and operations or instructions may be referred to as being "executed," "executed by a computer," or "executed by a CPU."

[0246] Those of ordinary skill in the art will understand that the operations and symbolically represented operations or instructions include the manipulation of electrical signals by a CPU. The electrical system represents data bits that can cause a resulting transformation or reduction of the electrical signals, and maintains the data bits in memory locations in a memory system, thereby reconfiguring or otherwise altering the operation of the CPU and the processing of other signals. The memory locations in which the data bits are maintained are physical locations that have particular electrical, magnetic, optical, or organic properties that correspond to or represent the data bits. It should be understood that exemplary embodiments are not limited to the above-mentioned platforms or CPUs, and that other platforms and CPUs may support the provided methods.

[0247] The data bits may also be maintained on computer-readable media, including magnetic disks, optical disks, and any other volatile (e.g., random access memory ("RAM")) or non-volatile (e.g., read-only memory ("ROM")) mass storage system readable by a CPU. The computer-readable media may include cooperative or interconnected computer-readable media that reside exclusively on a processing system or that are distributed among multiple interconnected processing systems, which may be local or remote to a processing system. Representative embodiments are not limited to the memories described above, and it will be understood that other platforms and memories may support the described methods.

[0248] In an exemplary embodiment, any of the operations, processes, etc. described herein may be implemented as computer-readable instructions stored on a computer-readable medium, which may be executed by a processor of a mobile, a network element, and / or any other computing device.

[0249] There is little distinction between hardware and software implementations of aspects of the system. The use of hardware or software is generally (though not always, the choice between hardware and software can be significant in certain situations) a design choice that implies a cost vs. efficiency trade-off. There may be a variety of vehicles (e.g., hardware, software, and / or firmware) in which the processes and / or systems and / or other technologies described herein may be affected, and the preferred vehicle may vary depending on the context in which the processes and / or systems and / or other technologies are deployed. For example, if an implementer determines that speed and accuracy are paramount, the implementer may select a primarily hardware and / or firmware vehicle. If flexibility is paramount, the implementer may select a primarily software implementation. Alternatively, the implementer may select some combination of hardware, software, and / or firmware.

[0250] The foregoing detailed description has illustrated various embodiments of devices and / or processes through the use of block diagrams, flowcharts, and / or examples. To the extent that such block diagrams, flowcharts, and / or examples include one or more functions and / or operations, those skilled in the art will appreciate that each function and / or operation in such block diagrams, flowcharts, or examples may be individually and / or collectively implemented by a wide range of hardware, software, firmware, or substantially any combination thereof. Suitable processors include, by way of example, general-purpose processors, special-purpose processors, conventional processors, digital signal processors (DSPs), multiple microprocessors, one or more microprocessors associated with a DSP core, controllers, microcontrollers, application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), field-programmable gate array (FPGA) circuits, any other type of integrated circuit (IC), and / or state machines.

[0251] While features and elements have been provided above in particular combinations, those of ordinary skill in the art will understand that each feature or element can be used alone or in any combination with other features and elements. The present disclosure is not limited in terms of the specific embodiments described herein, which are intended as examples of various aspects. It will be apparent to those skilled in the art that many modifications and variations can be made without departing from the spirit and scope of the invention. No element, act, or instruction used in the description of the present application should be construed as critical or essential to the invention unless explicitly stated as such. Functionally equivalent methods and apparatuses within the scope of the present disclosure, in addition to those enumerated herein, will be apparent to those skilled in the art from the foregoing description. Such modifications and variations are intended to fall within the scope of the appended claims. The present disclosure is limited only by the terms of the appended claims, along with the full scope of equivalents to which such claims are entitled. It is understood that the present disclosure is not limited to any particular method or system.

[0252] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used herein, "station" and its abbreviation "STA," "user equipment" and its abbreviation "UE," when referred to herein, may mean or include (i) a wireless transmit and / or receive unit (WTRU), such as a described infrastructure; (ii) any of several embodiments of a WTRU, such as a described infrastructure; (iii) a wireless-enabled and / or wired-enabled (e.g., tethered) device configured with some or all of the structure and functionality of an illustrated WTRU (e.g., a described infrastructure); (iii) a wireless-enabled and / or wired-enabled device configured with less than all of the structure and functionality of a WTRU (e.g., a described infrastructure); or (iv) others. Details of an exemplary WTRU that may represent any of the UEs enumerated herein are provided below with respect to FIGS. 1A-1D.

[0253] In certain exemplary embodiments, portions of the subject matter described herein may be implemented via application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), and / or other integrated formats. However, those skilled in the art will recognize that certain aspects of the embodiments disclosed herein may be equivalently implemented in an integrated circuit, in whole or in part, as one or more computer programs running on one or more computers (e.g., as one or more programs running on one or more computer systems), as one or more programs running on one or more processors (e.g., as one or more programs running on one or more microprocessors), as firmware, or as substantially any combination thereof, and that designing circuitry and / or writing software and / or firmware code is within the skill of those skilled in the art in light of this disclosure. Furthermore, those skilled in the art will recognize that the mechanisms of the subject matter described herein may be distributed as program products in various forms, and that exemplary embodiments of the subject matter described herein apply regardless of the particular type of signal-bearing medium used to actually effect the distribution. Examples of signal bearing media include, but are not limited to, recordable-type media such as floppy disks, hard disk drives, CDs, DVDs, digital tape, computer memory, and transmission-type media such as digital and / or analog communications media (e.g., fiber optic cables, wave guides, wired communications links, wireless communications links, etc.).

[0254] The subject matter described herein may, in some cases, depict different components that are contained within or connected to different other components. It should be understood that such illustrated architectures are merely examples, and that in fact many other architectures that achieve the same functionality may be implemented. Conceptually, any arrangement of components to achieve the same functionality is effectively “associated” such that the desired functionality may be achieved. Thus, any two components combined herein to achieve a particular function can be viewed as “associated” with each other such that the desired functionality is achieved, regardless of the architecture or intermediate components. Similarly, any two components so associated may also be considered to be “operably connected” or “operably coupled” to each other to achieve the desired functionality, and any two components so associated may also be considered to be “operably coupleable” to each other to achieve the desired functionality. Examples of operably coupleable include, but are not limited to, physically matable and / or physically interacting components, wirelessly interacting and / or wirelessly interacting components, and / or logically interacting and / or logically interacting components.

[0255] With respect to the use of virtually any plural and / or singular term herein, those skilled in the art can convert from plural to singular and / or from singular to plural as appropriate to the context and / or application. Various singular / plural permutations may be expressly set forth herein for purposes of clarity.

[0256] In general, those skilled in the art will understand that terms used in this specification, and particularly in the appended claims (e.g., the body of the appended claims), are generally intended as "open" terms (e.g., the term "including" should be interpreted as "including, but not limited to," the term "having" should be interpreted as "having at least," and the term "comprises" should be interpreted as "including, but not limited to"). Furthermore, where a specific number of recitations of an introduced claim are intended, such intention will be explicitly set forth in the claim; in the absence of such recitation, those skilled in the art will understand that no such intention exists. For example, where only one item is intended, the term "single" or similar language may be used. To assist in understanding, the following appended claims and / or description of this specification may include the use of the introductory phrases "at least one" and "one or more" to introduce claim recitations. However, the use of such phrases should not be construed as meaning that the introduction of a claim recitation by the indefinite article "a" or "an" limits any particular claim containing such an introduced claim recitation to embodiments containing only one such recitation, even if the same claim contains the introductory phrase "one or more" or "at least one" and an indefinite article such as "a" or "an" (e.g., "a" and / or "an" should be interpreted to mean "at least one" or "one or more"). The same applies to the use of definite articles used to introduce claim recitations. Furthermore, those skilled in the art will recognize that even when a specific number of recitations of an introduced claim are explicitly recited, such recitation should be interpreted to mean at least the recited number (e.g., the simple recitation "two recitations" without other modifiers means at least two recitations, or more than two recitations).Furthermore, when notation similar to "at least one of A, B, and C" is used, such structure is generally intended as the meaning that one of ordinary skill in the art would understand the notation (e.g., "a system having at least one of A, B, and C" includes, but is not limited to, a system having A only, B only, C only, A and B together, A and C together, B and C together, and / or A, B, and C together). When notation similar to "at least one of A, B, or C" is used, such structure is generally intended as the meaning that one of ordinary skill in the art would understand the notation (e.g., "a system having at least one of A, B, or C" includes, but is not limited to, a system having A only, B only, C only, A and B together, A and C together, B and C together, and / or A, B, and C together). Those skilled in the art will further appreciate that virtually any disjunctive word and / or phrase presenting two or more alternative terms, whether in the description, claims, or drawings, should be understood to contemplate the possibility of including one of the terms, either of the terms, or both terms. For example, the phrase "A or B" should be understood to include the possibilities of "A" or "B" or "A and B." Furthermore, as used herein, the term "any of," followed by a list of items and / or a list of categories of items, is intended to include "any of," "any combination of," "any plurality of," and / or "any combination of" the items and / or categories of items, individually or in combination with other items and / or other categories of items. Furthermore, as used herein, the terms "set" or "group" are intended to include any number of items, including zero. Furthermore, as used herein, the term "number" is intended to include any number, including zero.

[0257] Furthermore, where features or aspects of the disclosure are described in terms of a Markush group, those skilled in the art will recognize that the disclosure is also thereby described in terms of any individual member or subgroup of members of the Markush group.

[0258] As will be understood by those skilled in the art, for all purposes, including in terms of providing a written description, all ranges disclosed herein encompass any possible subranges and combinations of subranges. Any recited range can be readily recognized as fully descriptive and allowing the same range to be broken down into at least equal halves, thirds, quarters, fifths, tenths, etc. As a non-limiting example, each range described herein can be easily broken down into a lower third, middle third, upper third, etc. Additionally, as will be understood by those skilled in the art, all terms such as "up to," "at least," "greater than," "less than," etc., refer to ranges that are inclusive of the recited number and that can be further broken down into subranges as described above. Finally, as will be understood by those skilled in the art, ranges include individual elements. Thus, for example, a group having 1 to 3 cells refers to a group having 1, 2, or 3 cells. Similarly, a group having 1 to 5 cells refers to a group having 1, 2, 3, 4, or 5 cells, and so on.

[0259] Furthermore, the claims should not be read as limited to the provided order or to the provided elements unless specifically so stated. Furthermore, the use of the term "means for" in any claim is intended to invoke 35 U.S.C. 112, paragraph 6, or means-plus-function claim format, and any claim without the term "means for" is not so intended.

[0260] A processor in association with software may be used to implement a radio frequency transceiver for use in a wireless transmit / receive unit (WTRU), user equipment (UE), terminal, base station, mobility management entity (MME), or evolved packet core (EPC), or any host computer. The WTRU may be used in conjunction with modules implemented in hardware and / or software, such as, for example, a software defined radio (SDR), and may also be implemented in other components, such as a camera, a video camera module, a video phone, a speaker phone, a vibration device, a speaker, a microphone, a television transceiver, a hands-free headset, a keyboard, a Bluetooth module, a frequency modulation (FM) radio unit, a near field communication (NFC) module, an LCD display unit, an organic light emitting diode (OLED) display unit, a digital music player, a media player, a video game player module, an internet browser, and / or a wireless local area network (WLAN) or ultra wide band (UWB) module.

[0261] Although the present invention has been described with respect to a communications system, it is contemplated that the system may be implemented in software on a microprocessor / general purpose computer (not shown). In particular embodiments, one or more of the functions of the various components may be implemented in software controlling a general purpose computer.

[0262] Moreover, although the invention is illustrated and described herein with reference to specific embodiments, the invention is not intended to be limited to the details shown. Rather, various modifications of the details can be made within the scope of the claims and their equivalents without departing from the invention.

[0263] Throughout this disclosure, those skilled in the art will understand that certain exemplary embodiments may be used alternatively or in combination with other exemplary embodiments.

[0264] Although features and elements are described above in particular combinations, those skilled in the art will understand that each feature or element can be used alone or in any combination with the other features and elements. Furthermore, the methods described herein may be implemented in a computer program, software, or firmware embodied in a computer-readable medium for execution by a computer or processor. Examples of non-transitory computer-readable storage media include, but are not limited to, read-only memory (ROM), random-access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, and optical media such as magneto-optical media and CD-ROM disks and digital versatile disks (DVDs). A processor in association with software may be used to implement a radio frequency transceiver for use in a WTRU, UE, terminal, base station, RNC, or any host computer.

[0265] Furthermore, in the above embodiments, processing platforms, computing systems, controllers, and other devices including processors are described. These devices may include at least one central processing unit ("CPU") and memory. In accordance with the practices of those skilled in the art of computer programming, references to acts and symbolic representations of operations or instructions may be performed by various CPUs and memories. Such acts and operations or instructions may be referred to as being "executed," "executed by a computer," or "executed by a CPU."

[0266] Those of ordinary skill in the art will understand that the operations and symbolically represented operations or instructions involve the manipulation of electrical signals by a CPU. An electrical system represents data bits that can cause a resulting transformation or reduction of the electrical signals, and maintains the data bits in memory locations in a memory system, thereby reconfiguring or otherwise altering the operation of the CPU and the processing of other signals. The memory locations in which the data bits are maintained are physical locations that have particular electrical, magnetic, optical, or organic properties that correspond to or represent the data bits.

[0267] The data bits may also be maintained on computer-readable media, including magnetic disks, optical disks, and any other volatile (e.g., random access memory ("RAM")) or non-volatile (e.g., read-only memory ("ROM")) mass storage system readable by a CPU. The computer-readable media may include cooperative or interconnected computer-readable media that reside exclusively on a processing system or that are distributed among multiple interconnected processing systems, which may be local or remote to a processing system. Representative embodiments are not limited to the memories described above, and it will be understood that other platforms and memories may support the described methods.

[0268] Suitable processors include, by way of example, a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application specific integrated circuit (ASIC), an application specific standard product (ASSP), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), and / or a state machine.

[0269] Although the present invention has been described with respect to a communications system, it is contemplated that the system may be implemented in software on a microprocessor / general purpose computer (not shown). In particular embodiments, one or more of the functions of the various components may be implemented in software controlling a general purpose computer.

[0270] Moreover, although the invention is illustrated and described herein with reference to specific embodiments, the invention is not intended to be limited to the details shown. Rather, various modifications of the details can be made within the scope of the claims and their equivalents without departing from the invention.

Claims

1. 1. A method performed by a neural network based decoder (NNBD), comprising: obtaining or receiving, by said NNBD, a codeword as a descriptor of an input data representation; determining, by a first neural network module, a preliminary reconstruction of the input data representation based on at least the codewords and an initial 2D grid; determining, by a second neural network module, a modified 2D grid based on at least the preliminary reconstruction and the codeword; determining, by the first neural network module, a refined reconstruction of the input data representation based on at least the codewords and the modified 2D grid, the modified 2D grid indicating topological information associated with the input data representation; A method comprising:

2. The method of claim 1 , wherein the modified 2D grid is determined by combining the initial 2D grid and the output of the second neural network module.

3. The method of claim 1 , wherein the modified 2D grid is a local connectivity graph.

4. 10. The method of claim 1, further comprising generating a concatenation matrix for processing by one or more convolutional neural networks (CNNs) by concatenating at least the replicated codewords, the initial 2D grid or the modified 2D grid, and the reconstructed data representation.

5. The NNBD is a graph-conditional NNBD, determining the refined reconstruction of the input data representation is performed via multiple iterations of at least the first neural network module; 10. The method of claim 1.

6. the NNBD includes one or more multi-layer perceptrons (MLPs); the modified 2D grid and the refined reconstruction of the input data representation are further based on gradient information generated by the one or more MLPs.

10. The method of claim 1.

7. the initial 2D grid and the modified 2D grid are two-dimensional (2D) point sets; the input data representation is a point cloud; determining the preliminary reconstruction of the input data representation includes performing a deformation operation based on the codeword and the 2D point set initialized with a predetermined sampling in a plane; 10. The method of claim 1.

8. The determining of the modified 2D grid may include: The method of claim 1 , comprising performing a partitioning operation based on the input data representation, the codewords, and the preliminary reconstruction of the initial 2D grid to generate the modified 2D grid.

9. The determining of the modified 2D grid may include: replicating the received or acquired codeword K times to generate a K×D codeword matrix, where K is the number of nodes in the initial 2D grid and D is the length of the codeword; concatenating the K×D codeword matrix with the initial 2D grid as a K×N matrix to generate a K×(D+N) concatenation matrix; inputting the connectivity matrix into one or more convolutional neural networks (CNNs) or multi-layer perceptrons (MLPs); generating the modified 2D grid by the one or more CNNs or MLPs from the connectivity matrix; updating the refined reconstruction of the input data representation based on the modified 2D grid to generate a final reconstruction of the input data representation; 2. The method of claim 1, comprising:

10. Concatenating the codeword matrix as a concatenated intermediate matrix to outputs of a first set of CNN layers or MLP layers; inputting the concatenated intermediate matrix into a next set of CNN layers or MLP layers following the first set of CNN layers or MLP layers; 10. The method of claim 9, further comprising:

11. A neural network based decoder (NNBD), comprising: a receiver unit configured to receive or obtain codewords as descriptors of the input data representation; a first neural network (NN) module configured to determine a preliminary reconstruction of the input data representation based on at least the codewords and an initial 2D grid; a second NN module configured to determine a modified 2D grid based on at least the preliminary reconstruction and the codeword; Including, the first NN module is further configured to determine a refined reconstruction of the input data representation based on at least the codeword and the modified 2D grid, the modified 2D grid indicating topological information associated with the input data representation. NNBD.

12. The NNBD of claim 11 , wherein the modified 2D grid is a locally connected graph.

13. the second NN module includes one or more convolutional neural networks (CNNs); the NNBD is configured to generate a concatenation matrix using at least (1) the replicated codewords, (2) the initial 2D grid or the modified 2D grid, and (3) the reconstructed data representation; the one or more CNNs are configured to process the connectivity matrix and generate the modified 2D grid or a refined modified 2D grid. NNBD of claim 11.

14. The NNBD is a graph-conditional NNBD, the first NN module is configured to perform a plurality of iterations; NNBD of claim 11.

15. the first NN module includes one or more multi-layer perceptrons (MLPs) configured to generate gradient information; the second NN module is configured to output the modified 2D grid based on the gradient information generated by the one or more MLPs. NNBD of claim 11.

16. the initial 2D grid and the modified 2D grid are two-dimensional (2D) point sets and the input data representation is a point cloud; the first NN module is configured to perform a deformation operation based on the codeword and the 2D point set initialized with a predetermined sampling in a plane. NNBD of claim 11.

17. 17. The NNBD of claim 16, wherein the second NN module is configured to perform a partitioning operation based on the input data representation, the codewords, and the preliminary reconstruction of the initial 2D grid to generate the modified 2D grid.

18. the initial 2D grid comprises a matrix of points, each point indicating a 2D position; the 2D grid is associated with a manifold, each point representing a fixed position on the manifold; The 2D grid is a fixed set of points sampled from a 2D plane. NNBD of claim 11.

19. The NNBD is replicating the received or acquired codeword K times to generate a K×D codeword matrix, where K is the number of nodes in the initial 2D grid and D is the length of the codeword; concatenating the K×D codeword matrix with the initial 2D grid as a K×N matrix to generate a K×(D+N) concatenation matrix; inputting the connectivity matrix into one or more convolutional neural networks (CNNs) or multi-layer perceptrons (MLPs) of the NNBD; generating the modified 2D grid by the one or more CNNs or MLPs of the NNBD from the connection matrix; updating the refined reconstruction of the input data representation based on the modified 2D grid to generate a final reconstruction of the input data representation; 19. The NNBD of claim 18, configured to perform the following.

20. The NNBD is Concatenating the codeword matrix as a concatenated intermediate matrix to outputs of a first set of CNN layers or MLP layers; inputting the concatenated intermediate matrix into a next set of CNN layers or MLP layers following the first set of CNN layers or MLP layers; 20. The NNBD of claim 19, configured to perform the following.

Images