System and method for optimizing dynamic point clouds based on prioritized transformations

Abstract

A system and method for optimizing dynamic point clouds based on prioritized transformations, the disclosed example method comprising (i) transmitting a first point cloud to a client, where the first point cloud corresponds to a reference point cloud, (ii) receiving a second point cloud, and (iii) hierarchically determining a change in the second point cloud from the reference point cloud, wherein hierarchically determining the variation comprises: (a) identifying a first region in the second point cloud that has changed from the reference point cloud, and (b) for a first region having a highest priority, determining a first rigid 3D transformation that approximates a first variation from the reference point cloud, and if the first rigid 3D transformation cannot be determined, determining a second rigid 3D transformation that approximates a second variation from the reference point cloud. If so, a first point to be used to modify the reference point cloud is further determined, where the first point represents the first change.

Description

This application is a divisional application of Chinese Patent Application No. 201980031106.9, filed on March 19, 2019, entitled “System and Method for Optimizing Dynamic Point Clouds Based on Prioritized Sorting Transformations”, the contents of which are incorporated herein by reference in their entirety. Cross-reference section

[0001] This application is a non-provisional application of and claims the benefit of 35 U.S.C. §119(e) of U.S. Provisional Patent Application No. 62 / 645,603, filed on March 20, 2018, entitled “System and Method for Optimizing Dynamic Point Clouds Based on Prioritized Transformations,” which is incorporated herein by reference in its entirety. Background Art

[0002] As new systems and sensors become available for capturing high-resolution three-dimensional (3D) data, the challenge of transmitting large data sizes of dynamic point cloud sequences arises. For example, testing of the data throughput required to publish a full 3D appearance of a person has revealed several network bottlenecks that limit the practicality of high-resolution data capture and publication. Summary of the invention

[0003] According to some embodiments, a method performed at a server includes: sending a first point cloud to a client, wherein the first point cloud corresponds to a reference point cloud; receiving a second point cloud; and hierarchically determining changes in the second point cloud from the reference point cloud, wherein hierarchically determining the changes includes: identifying a first area in the second point cloud that has changed from the reference point cloud; prioritizing the first area; and for the first area with the highest priority, determining whether there is a first rigid 3D transformation that approximates the first change from the reference point cloud; in response to determining that the first rigid 3D transformation exists, determining the first rigid 3D transformation that approximates the first change from the reference point cloud; and in response to determining that the first rigid 3D transformation does not exist, further determining a first point to be used to modify the reference point cloud, wherein the first point represents the first change.

[0004] According to some embodiments, a method performed at a server includes: sending a first point cloud to a client, wherein the first point cloud corresponds to a reference point cloud; receiving a second point cloud; and hierarchically determining changes in the second point cloud from the reference point cloud, wherein hierarchically determining the changes includes: identifying a first area in the second point cloud that has changed from the reference point cloud; prioritizing the first area; and for the first area with the highest priority, determining a first rigid 3D transformation that approximates the first change from the reference point cloud, and if the first rigid 3D transformation cannot be determined, further determining a first point to be used to modify the reference point cloud, wherein the first point represents the first change.

[0005] In some embodiments, the first point for modifying the reference point cloud includes at least one of: (i) a point to be removed from the reference point cloud or (ii) a point to be added to the reference point cloud. In some embodiments, identifying the first region includes comparing the second point cloud with the reference point cloud to identify a region in the second point cloud that deviates from the reference point cloud. In some embodiments, prioritizing the first regions includes assigning a corresponding first priority to each of the first regions based at least in part on a current viewpoint of the client. In some embodiments, assigning the corresponding first priority to each of the first regions includes determining the corresponding first priority using a size of a primary region, an amount of deviation of the first region from the reference point cloud, and a distance of the first region from the current viewpoint of the client. Further, in some embodiments, determining whether the first rigid 3D transform exists includes determining whether a shape correspondence has been found between the reference point cloud and the second point cloud.

[0006] In some embodiments, the method further comprises: receiving the current viewpoint from the client. In some embodiments, the method further comprises storing the reference point cloud, and if the first 3D rigid transformation has been determined, updating the reference point cloud stored at the server by applying the first rigid 3D transformation to the reference point cloud. In some embodiments, the method further comprises sending an indication of a first region with the highest priority and a 3D transformation to the client to update the reference point cloud at the client. In some embodiments, the indication of the first primary region with the highest priority comprises bounding volume coordinates.

[0007] In some embodiments, the method further comprises storing the reference point cloud and, if the first point has been determined, updating the reference point cloud stored at the server by modifying the reference point cloud using the first point. In some embodiments, the method further comprises sending an indication of the first point to a client to update the reference point cloud at the client.

[0008] In some embodiments, hierarchically determining the change further comprises: identifying one or more first sub-regions within the first region having the highest priority, prioritizing the first sub-regions, and adding the prioritized first sub-regions to the remaining prioritized first regions. In some embodiments, the method further comprises, for a region having a second highest priority among the remaining prioritized first regions and the prioritized first sub-regions, determining (i) a second rigid 3D transformation to approximate a second change from the updated reference point cloud or (ii) a second point to be used to modify the updated reference point cloud, wherein the second point represents the second change. In this regard, in some embodiments, the method further comprises negotiating a processing budget with a client, wherein the processing budget provides at least an amount of time and bandwidth available for point cloud updates, and wherein determining a second rigid 3D transformation or a second point for the region having the second highest priority is performed only when the processing budget is available.

[0009] In some embodiments, identifying the one or more first sub-regions comprises: comparing the second point cloud with the updated reference point cloud to identify a finer granularity region that deviates from the updated reference point cloud. In some embodiments, prioritizing the first sub-regions comprises assigning a respective second priority to each of the first sub-regions. In some embodiments, the method comprises negotiating a processing budget with the client, wherein the processing budget provides at least an amount of time and bandwidth available for point cloud updates.

[0010] In some embodiments, at least one of the reference point cloud or the second point cloud comprises sensor data. In some embodiments, the reference point cloud and the second point cloud are received from a storage medium as a pre-captured dynamic sequence of point cloud data. In addition, in some embodiments, the method further comprises communicating with a plurality of clients and performing the method for each of the plurality of clients.

[0011] According to some embodiments, a method performed at a server includes: sending an initial point cloud to a client; and hierarchically determining changes in a current point cloud from the initial point cloud, wherein hierarchically determining the changes includes: identifying major change regions in the current point cloud; using first major change regions to determine whether there is a first rigid 3D transformation that approximates a first change from the initial point cloud; in response to determining that the first rigid 3D transformation exists, determining the first rigid 3D transformation that approximates the first change from the initial point cloud; in response to determining that the first rigid 3D transformation does not exist, further determining a first point to be used to modify the initial point cloud, wherein the first point represents the first change; identifying one or more finer-grained residual change regions within the major change region; using the one or more finer-grained residual change regions and the remaining major change regions to determine whether there is a second rigid 3D transformation that approximates a second change from the initial point cloud; in response to determining that the second rigid 3D transformation exists, determining the second rigid 3D transformation that approximates the second change from the initial point cloud; and in response to determining that the second rigid 3D transformation does not exist, further determining a second point to be used to further modify the initial point cloud, wherein the second point represents the second change.

[0012] According to some embodiments, a method for transmitting time-varying point cloud data performed at a server includes: sending an initial point cloud to a client; and hierarchically determining changes in a current point cloud from the initial point cloud, wherein hierarchically determining the changes includes: identifying major change regions in the current point cloud; using a first major change region to determine a first rigid 3D transformation that approximates a first change from the initial point cloud, and if the first rigid 3D transformation cannot be determined, further determining a first point to be used to modify the initial point cloud, wherein the first point represents the first change; identifying one or more finer-grained residual change regions within the major change region; and using the one or more finer-grained residual change regions and the remaining major change regions to determine a second rigid 3D transformation that approximates a second change from the initial point cloud, and if the second rigid 3D transformation cannot be determined, further determining a second point to be used to further modify the initial point cloud, wherein the second point represents the second change.

[0013] In some embodiments, the second rigid 3D transform is determined for one of the one or more finer-grained residual change regions. In some embodiments, the second rigid 3D transform is determined for one of the remaining major change regions. In some embodiments, the second rigid 3D transform is determined for one of the remaining major change regions. In addition, in some embodiments, the method further includes transmitting to the client at least one of: (i) the first rigid 3D transform and first bounding volume coordinates indicating the first change region, (ii) the second rigid 3D transform and second bounding volume coordinates indicating the second change region, (iii) the first point, and (iv) the second point.

[0014] According to some embodiments, a method performed at a server includes: receiving first 3D data as an initial point cloud; in response to a request from the client for streaming point cloud data, transmitting the initial point cloud to the client, and storing the initial point cloud as a reference point cloud; and for each current point cloud, iteratively performing a process including the following steps: receiving second 3D data as the current point cloud; receiving second 3D data as the current point cloud; performing a hierarchical check of the current point cloud relative to the reference point cloud, including: identifying an area that deviates from the reference point cloud; separating the deviated area into a first cluster; calculating an indication of the first cluster; The first clusters are prioritized based on corresponding first scores of the importance of each cluster, each first score being calculated at least in part based on the current viewpoint of the client; and for the first cluster with the highest priority, determining whether there is a transformation that approximates the first deviation from the reference point cloud; if the transformation exists, applying the transformation to the stored reference point cloud to update the reference point cloud, and sending an indication of the area in the first cluster and the transformation to the client; if not, performing at least one of adding points and removing points on the stored reference point cloud to update the reference point cloud, and sending the points to the client.

[0015] In some embodiments, performing a hierarchical check of the current point cloud relative to the reference point cloud also includes: after processing the first cluster with the highest priority, identifying a sub-region within the first cluster that still deviates from the updated reference point cloud; separating the deviated sub-region into a second cluster; prioritizing the second clusters by calculating a corresponding second score indicating the importance of each cluster in the second cluster, each second score being calculated at least in part based on the current viewpoint of the client; adding the prioritized second cluster to the remaining clusters of the prioritized first cluster, and for the next cluster with the next highest priority, determining whether there is another transformation that approximates the second deviation from the updated reference point cloud; if the other transformation exists, applying the other transformation to the updated reference point cloud stored in the update to further update the reference point cloud, and sending another indication of the sub-region in the processed first cluster and the other transformation to the client; and if the other transformation does not exist, performing at least one of adding additional points or removing additional points from the updated reference point cloud to update the reference point cloud, and transmitting the additional points to the client.

[0016] In some embodiments, the process is terminated upon receipt of a request from the client or when a processing budget negotiated between the server and the client is no longer available for further processing.

[0017] Other embodiments include systems and servers (eg, having a processor and a non-transitory computer-readable medium storing a plurality of instructions for execution by the processor) configured to perform the methods described herein. In some embodiments, the system further includes at least one 3D sensor. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] A more detailed understanding can be obtained from the following description presented by way of example in conjunction with the accompanying drawings.In addition, the same reference numerals in the figures represent the same elements.

[0019] Figure 1A is a system diagram illustrating an exemplary communication system in which one or more disclosed embodiments may be implemented.

[0020] Figure 1B is a diagram showing that according to an embodiment, Figure 1A A system diagram of an exemplary wireless transmit / receive unit (WTRU) for use within a communication system is shown.

[0021] Figure 1C is a diagram showing that according to an embodiment, Figure 1AA system diagram of an exemplary radio access network (RAN) and an exemplary core network (CN) used within a communication system is shown.

[0022] Figure 1D is a diagram showing that according to an embodiment, Figure 1A A system diagram of another exemplary RAN and another exemplary CN used within the communication system shown.

[0023] Figure 2 An example system arrangement is shown in which various embodiments of the present disclosure may be employed, according to some embodiments.

[0024] Figure 3A An example scenario is shown that may be captured and processed in accordance with some embodiments.

[0025] Figure 3B An example point cloud is shown that may be processed in accordance with some embodiments.

[0026] Figure 4 A message sequence chart according to some embodiments is shown.

[0027] Figure 5 is a flow chart of an example method of processing point cloud data according to some embodiments.

[0028] Figure 6 is a flow chart of another example method of processing point cloud data according to some embodiments.

[0029] Fig. 7A and 7B is a corresponding flow chart of yet another example method of processing point cloud data according to some embodiments.

[0030] Figure 8 is a flow chart of an example method of processing clustering according to some embodiments.

[0031] Fig. 9 is a flow chart illustrating an example method according to some embodiments.

[0032] Fig. 10A and 10B is a corresponding flow chart illustrating another example method according to some embodiments.

[0033] The entities, connections, arrangements, etc. depicted in and described in connection with the various figures are presented as examples and not as limitations. Therefore, any and all statements or other indications as to what a particular figure "depicts," "a particular element or entity in a particular figure" is "or "has," and any and all similar statements (which might be isolated and read out of context as absolute and therefore limiting) may be properly read only if they are constructively preceded by a clause such as "In at least one embodiment, ..." For the sake of brevity and clarity of presentation, this implicit introductory clause is not repeated in the detailed description of the drawings. DETAILED DESCRIPTION

[0034] A detailed description of illustrative embodiments will now be described with reference to the various drawings. Although this specification provides detailed examples of possible implementations, it should be noted that these details are intended to be exemplary and in no way limit the scope of the present application.

[0035] An example network for implementing the example embodiments will now be described.

[0036] Figure 1A 1 is a schematic diagram of an exemplary communication system 100 in which one or more disclosed embodiments may be implemented. The communication system 100 may be a multiple access system that provides content such as voice, data, video, messaging, broadcast, etc. to multiple wireless users. The communication system 100 may enable multiple wireless users to access such content by sharing system resources including wireless bandwidth. For example, the communication system 100 may use one or more channel access methods such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single carrier FDMA (SC-FDMA), zero tail unique word DFT-spread OFDM (ZT UW DTS-s OFDM), unique word OFDM (UW-OFDM), resource block filtered OFDM, and filter bank multi-carrier (FBMC), etc.

[0037] like Figure 1AAs shown, the communication system 100 may include wireless transmit / receive units (WTRUs) 102a, 102b, 102c, 102d, RAN 104 / 113, CN 106 / 115, public switched telephone network (PSTN) 108, Internet 110, and other networks 112, although it should be appreciated that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and / or network components. Each of the WTRUs 102a, 102b, 102c, 102d may be any type of device configured to operate and / or communicate in a wireless environment. By way of example, any of the WTRUs 102a, 102b, 102c, 102d may be referred to as a “station” and / or “STA”, which may be configured to transmit and / or receive wireless signals, and may include a user equipment (UE), a mobile station, a fixed or mobile subscriber unit, a subscription-based unit, a pager, a cellular phone, a personal digital assistant (PDA), a smart phone, a laptop, a netbook, a personal computer, a wireless sensor, a hotspot or Mi-Fi device, an Internet of Things (IoT) device, a watch or other wearable device, a head-mounted display (HMD), a vehicle, a drone, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices working in an industrial and / or automated process chain environment), consumer electronic devices, and devices working on a commercial and / or industrial wireless network, etc. Any of the WTRUs 102a, 102b, 102c, 102d may be interchangeably referred to as a UE.

[0038] The communication system 100 may also include a base station 114a and / or a base station 114b. Each of the base stations 114a, 114b may be any type of device configured to facilitate access to one or more communication networks (e.g., CN 106 / 115, Internet 110, and / or other networks 112) by wirelessly interfacing with at least one of the WTRUs 102a, 102b, 102c, 102d. For example, the base stations 114a, 114b may be a base transceiver station (BTS), a Node B, an eNode B, a Home Node B, a Home eNode B, a gNB, a New Radio (NR) Node B, a site controller, an access point (AP), a wireless router, and the like. Although each of the base stations 114a, 114b is depicted as a single component, it should be appreciated that the base stations 114a, 114b may include any number of interconnected base stations and / or network components.

[0039] The base station 114a may be part of the RAN 104 / 113, and the RAN may also include other base stations and / or network components (not shown), such as a base station controller (BSC), a radio network controller (RNC), a relay node, and the like. The base station 114a and / or the base station 114b may be configured to transmit and / or receive wireless signals on one or more carrier frequencies called cells (not shown). These frequencies may be in a licensed spectrum, an unlicensed spectrum, or a combination of licensed and unlicensed spectrum. A cell may provide wireless service coverage for a specific geographic area that is relatively fixed or may change over time. A cell may be further divided into cell sectors. For example, a cell associated with the base station 114a may be divided into three sectors. Thus, in one embodiment, the base station 114a may include three transceivers, i.e., each transceiver corresponds to a sector of the cell. In an embodiment, the base station 114a may use multiple-input multiple-output (MIMO) technology, and may use multiple transceivers for each sector of the cell. For example, by using beamforming, signals may be transmitted and / or received in a desired spatial direction.

[0040] The base stations 114a, 114b may communicate with one or more of the WTRUs 102a, 102b, 102c, 102d over an air interface 116, which may be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, millimeter wave, infrared (IR), ultraviolet (UV), visible light, etc.). The air interface 116 may be established using any suitable radio access technology (RAT).

[0041] More specifically, as described above, the communication system 100 may be a multiple access system and may use one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, and SC-FDMA, among others. For example, the base station 114a in the RAN 104 / 113 and the WTRUs 102a, 102b, 102c may implement a radio technology such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which may use Wideband CDMA (WCDMA) to establish the air interface 115 / 116 / 117. WCDMA may include communication protocols such as High Speed ​​Packet Access (HSPA) and / or Evolved HSPA (HSPA+). HSPA may include High Speed ​​Downlink (DL) Packet Access (HSDPA) and / or High Speed ​​UL Packet Access (HSUPA).

[0042] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which may establish the air interface 116 using Long Term Evolution (LTE) and / or Advanced LTE (LTE-A) and / or Advanced LTE Pro (LTE-A Pro).

[0043] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology that may use new radio (NR) to establish the air interface 116, such as NR radio access.

[0044] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement multiple radio access technologies. For example, the base station 114a and the WTRUs 102a, 102b, 102c may jointly implement LTE radio access and NR radio access (e.g., using dual connectivity (DC) principles). Thus, the air interface used by the WTRUs 102a, 102b, 102c may be characterized by multiple types of radio access technologies and / or transmissions sent to / from multiple types of base stations (e.g., eNBs and gNBs).

[0045] In other embodiments, the base station 114a and the WTRUs 102a, 102b, 102c may implement radio technologies such as IEEE 802.11 (i.e., Wireless Fidelity (WiFi)), IEEE 802.16 (i.e., Worldwide Interoperability for Microwave Access (WiMAX)), CDMA2000, CDMA2000 1X, CDMA2000EV-DO, Interim Standard 2000 (IS-2000), Interim Standard 95 (IS-95), Interim Standard 856 (IS-856), Global System for Mobile Communications (GSM), Enhanced Data Rates for GSM Evolution (EDGE), and GSM EDGE (GERAN), among others.

[0046] Figure 1AThe base station 114b in the example may be, for example, a wireless router, a Home NodeB, a Home eNodeB, or an access point, and may use any appropriate RAT to facilitate wireless connectivity in a local area, such as a business location, a residence, a vehicle, a campus, an industrial facility, an air corridor (e.g., for use by drones), a road, and the like. In one embodiment, the base station 114b and the WTRUs 102c, 102d may establish a wireless local area network (WLAN) by implementing a radio technology such as IEEE 802.11. In an embodiment, the base station 114b and the WTRUs 102c, 102d may establish a wireless personal area network (WPAN) by implementing a radio technology such as IEEE 802.15. In yet another embodiment, the base station 114b and the WTRUs 102c, 102d may establish a picocell or a femtocell by using a cellular-based RAT (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR, and the like). As Figure 1A As shown, the base station 114b may be directly connected to the Internet 110. Thus, the base station 114b does not need to access the Internet 110 via the CN 106 / 115.

[0047] The RAN 104 / 113 may be in communication with the CN 106 / 115, which may be any type of network configured to provide voice, data, applications, and / or Voice over Internet Protocol (VoIP) services to one or more of the WTRUs 102a, 102b, 102c, 102d. The data may have different quality of service (QoS) requirements, such as different throughput requirements, latency requirements, fault tolerance requirements, reliability requirements, data throughput requirements, mobility requirements, etc. The CN 106 / 115 may provide call control, billing services, mobile location-based services, prepaid calls, Internet connectivity, video distribution, etc., and / or may perform advanced security functions such as user authentication. Although in Figure 1A Although not shown, it will be appreciated that the RAN 104 / 113 and / or the CN 106 / 115 may be in direct or indirect communication with other RANs that employ the same RAT or a different RAT as the RAN 104 / 113. For example, in addition to being connected to the RAN 104 / 113 employing NR radio technology, the CN 106 / 115 may also be in communication with another RAN (not shown) employing GSM, UMTS, CDMA 2000, WiMAX, E-UTRA, or WiFi radio technology.

[0048] The CN 106 / 115 may also act as a gateway for the WTRUs 102a, 102b, 102c, 102d to access the PSTN 108, the Internet 110, and / or other networks 112. The PSTN 108 may include a circuit-switched telephone network that provides plain old telephone service (POTS). The Internet 110 may include a global system of interconnected computer network devices that use common communication protocols, such as TCP, User Datagram Protocol (UDP), and / or IP in the Transmission Control Protocol / Internet Protocol (TCP / IP) Internet protocol suite. The networks 112 may include wired or wireless communication networks owned and / or operated by other service providers. For example, the networks 112 may include another CN connected to one or more RANs, where the one or more RANs may use the same RAT or a different RAT as the RAN 104 / 113.

[0049] Some or all of the WTRUs 102a, 102b, 102c, 102d in the communication system 100 may include multi-mode capabilities (e.g., the WTRUs 102a, 102b, 102c, 102d may include multiple transceivers for communicating with different wireless networks over different wireless links). Figure 1A The illustrated WTRU 102c may be configured to communicate with the base station 114a, which may employ a cellular-based radio technology, and with the base station 114b, which may employ an IEEE 802 radio technology.

[0050] Figure 1B is a system diagram illustrating an exemplary WTRU 102. Figure 1B As shown, the WTRU 102 may include a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a numeric keypad 126, a display / touchpad 128, a non-removable memory 130, a removable memory 132, a power source 134, a global positioning system (GPS) chipset 136, and / or peripherals 138. It should be appreciated that the WTRU 102 may include any sub-combination of the foregoing elements while remaining consistent with an embodiment.

[0051] The processor 118 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, and the like. The processor 118 may perform signal coding, data processing, power control, input / output processing, and / or any other functionality that enables the WTRU 102 to operate in a wireless environment. The processor 118 may be coupled to the transceiver 120, which may be coupled to the transmit / receive element 122. Although Figure 1B The processor 118 and the transceiver 120 are depicted as separate components, however it should be appreciated that the processor 118 and the transceiver 120 may also be integrated together in an electronic package or chip.

[0052] The transmit / receive component 122 may be configured to transmit or receive signals to or from a base station (e.g., base station 114a) via the air interface 116. For example, in one embodiment, the transmit / receive component 122 may be an antenna configured to transmit and / or receive RF signals. As an example, in another embodiment, the transmit / receive component 122 may be an emitter / detector configured to transmit and / or receive IR, UV, or visible light signals. In yet another embodiment, the transmit / receive component 122 may be configured to transmit and / or receive RF and light signals. It should be appreciated that the transmit / receive component 122 may be configured to transmit and / or receive any combination of wireless signals.

[0053] Although in Figure 1B 102 as a single component, the WTRU 102 may include any number of transmit / receive components 122. More specifically, the WTRU 102 may employ MIMO technology. Thus, in one embodiment, the WTRU 102 may include two or more transmit / receive components 122 (e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface 116.

[0054] The transceiver 120 may be configured to modulate signals to be transmitted by the transmit / receive component 122 and to demodulate signals received by the transmit / receive component 122. As described above, the WTRU 102 may have multi-mode capabilities. Thus, the transceiver 120 may include multiple transceivers that allow the WTRU 102 to communicate via multiple RATs, such as NR and IEEE 802.11.

[0055] The processor 118 of the WTRU 102 may be coupled to the speaker / microphone 124, the keypad 126, and / or the display / touchpad 128 (e.g., a liquid crystal display (LCD) display unit or an organic light emitting diode (OLED) display unit), and may receive user input data from these components. The processor 118 may also output user data to the speaker / microphone 124, the keypad 126, and / or the display / touchpad 128. In addition, the processor 118 may access information from and store information in any suitable memory such as the non-removable memory 130 and / or the removable memory 132. The non-removable memory 130 may include a random access memory (RAM), a read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, and the like. In other embodiments, the processor 118 may access information from and store data in memories that are not physically located in the WTRU 102, such as, for example, a server or a home computer (not shown).

[0056] The processor 118 may receive power from the power source 134, and may be configured to distribute and / or control power for use by other components in the WTRU 102. The power source 134 may be any suitable device for powering the WTRU 102. For example, the power source 134 may include one or more dry cell batteries (e.g., nickel-cadmium (Ni-Cd), nickel-zinc (Ni-Zn), nickel-metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, and the like.

[0057] The processor 118 may also be coupled to the GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) related to the current location of the WTRU 102. In addition to or in lieu of the information from the GPS chipset 136, the WTRU 102 may receive location information from a base station (e.g., base stations 114a, 114b) via the air interface 116 and / or determine its location based on the timing of signals received from two or more nearby base stations. It should be appreciated that the WTRU 102 may acquire location information via any suitable positioning method while remaining consistent with an embodiment.

[0058] The processor 118 may also be coupled to other peripheral devices 138, wherein the peripheral devices may include one or more software and / or hardware modules that provide additional features, functions, and / or wired or wireless connectivity. For example, the peripheral devices 138 may include an accelerometer, an electronic compass, a satellite transceiver, a digital camera (for photos and / or video), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands-free headset, Modules, frequency modulation (FM) radio units, digital music players, media players, video game console modules, Internet browsers, virtual reality and / or augmented reality (VR / AR) devices, and activity trackers, etc. The peripheral device 138 may include one or more sensors, which may be one or more of the following: gyroscopes, accelerometers, Hall effect sensors, magnetometers, orientation sensors, proximity sensors, temperature sensors, time sensors, geographic location sensors, altimeters, light sensors, touch sensors, magnetometers, barometers, gesture sensors, biometric sensors, and / or humidity sensors, etc.

[0059] The WTRU 102 may include a full-duplex radio in which reception or transmission of some or all signals (e.g., associated with specific subframes for UL (e.g., for transmission) and downlink (e.g., for reception)) may be concurrent and / or simultaneous for the radio. A full-duplex radio may include an interference management unit that reduces and / or substantially eliminates self-interference by means of hardware (e.g., chokes) or by signal processing by a processor (e.g., a separate processor (not shown) or by the processor 118). In an embodiment, the WTRU 102 may include a half-duplex radio that transmits and receives some or all signals (e.g., associated with specific subframes for UL (e.g., for transmission) or downlink (e.g., for reception)).

[0060] Figure 1C 1 is a system diagram showing the RAN 104 and the CN 106 according to an embodiment. As described above, the RAN 104 may communicate with the WTRUs 102a, 102b, 102c using an E-UTRA radio technology over the air interface 116. The RAN 104 may also communicate with the CN 106.

[0061] The RAN 104 may include eNode-Bs 160a, 160b, 160c, however it will be appreciated that the RAN 104 may include any number of eNode-Bs while remaining consistent with an embodiment. The eNode-Bs 160a, 160b, 160c may each include one or more transceivers for communicating with the WTRUs 102a, 102b, 102c over the air interface 116. In one embodiment, the eNode-Bs 160a, 160b, 160c may implement MIMO technology. Thus, for example, the eNode-B 160a may use multiple antennas to transmit wireless signals to, and / or receive wireless signals from, the WTRU 102a.

[0062] Each of the eNodeBs 160a, 160b, 160c may be associated with a particular cell (not shown) and may be configured to handle radio resource management decisions, handover decisions, user scheduling in the UL and / or DL, etc. Figure 1C As shown, the eNode-Bs 160a, 160b, 160c may communicate with each other via an X2 interface.

[0063] Figure 1C The illustrated CN 106 may include a mobility management entity (MME) 162, a serving gateway (SGW) 164, and a packet data network (PDN) gateway (or PGW) 166. Although each of the aforementioned components is described as being part of the CN 106, it should be appreciated that any of these components may be owned and / or operated by an entity other than the CN operator.

[0064] The MME 162 may be connected to each of the eNode-Bs 162a, 162b, 162c in the RAN 104 via an S1 interface and may serve as a control node. For example, the MME 162 may be responsible for authenticating users of the WTRUs 102a, 102b, 102c, performing bearer activation / deactivation processing, selecting a particular serving gateway during an initial attach of the WTRUs 102a, 102b, 102c, and the like. The MME 162 may provide a control plane function for switching between the RAN 104 and other RANs (not shown) that employ other radio technologies, such as GSM and / or WCDMA.

[0065] The SGW 164 may be connected to each of the eNode-Bs 160a, 160b, 160c in the RAN 104 via an S1 interface. The SGW 164 may generally route and forward user data packets to / from the WTRUs 102a, 102b, 102c. The SGW 164 may also perform other functions, such as anchoring the user plane during an inter-eNB handover, triggering paging processing when DL data is available for the WTRUs 102a, 102b, 102c, and managing and storing the contexts of the WTRUs 102a, 102b, 102c, and the like.

[0066] The SGW 164 may be connected to the PGW 146, which may provide the WTRUs 102a, 102b, 102c with access to packet-switched networks, such as the Internet 110, to facilitate communications between the WTRUs 102a, 102b, 102c and IP-enabled devices.

[0067] The CN 106 may facilitate communications with other networks. For example, the CN 106 may provide the WTRUs 102a, 102b, 102c with access to circuit-switched networks, such as the PSTN 108, to facilitate communications between the WTRUs 102a, 102b, 102c and traditional land-line communications devices. For example, the CN 106 may include or communicate with an IP gateway, such as an IP Multimedia Subsystem (IMS) server, and the IP gateway may serve as an interface between the CN 106 and the PSTN 108. In addition, the CN 106 may provide the WTRUs 102a, 102b, 102c with access to other networks 112, which may include other wired and / or wireless networks owned and / or operated by other service providers.

[0068] Although in Figures 1A-1D Although the WTRU is described as a wireless terminal, it should be appreciated that in certain representative embodiments, such a terminal may use a (eg, temporary or permanent) wired communication interface with a communication network.

[0069] In a representative embodiment, the other network 112 may be a WLAN.

[0070] A WLAN using an infrastructure basic service set (BSS) mode may have an access point (AP) for the BSS and one or more stations (STAs) associated with the AP. The AP may access or be connected to a distributed system (DS) or other types of wired / wireless networks that send traffic into and / or out of the BSS. Traffic originating from outside the BSS and destined for the STA may arrive through the AP and be delivered to the STA. Traffic originating from the STA and destined for a destination outside the BSS may be sent to the AP for delivery to the corresponding destination. Traffic between STAs within the BSS may be sent through the AP, for example, when the source STA may send traffic to the AP and the AP may deliver traffic to the destination STA. Traffic between STAs within the BSS may be considered and / or referred to as point-to-point traffic. The point-to-point traffic may be sent using a direct link establishment (DLS) between the source and destination STAs (e.g., directly therebetween). In certain representative embodiments, the DLS may use 802.11e DLS or 802.11z channelized DLS (TDLS)). For example, a WLAN using an independent BSS (IBSS) mode does not have an AP, and STAs (eg, all STAs) within or using the IBSS can communicate directly with each other. Here, the IBSS communication mode may sometimes be referred to as an "ad-hoc" communication mode.

[0071] When using the 802.11ac infrastructure mode of operation or a similar mode of operation, the AP may transmit a beacon on a fixed channel (e.g., a primary channel). The primary channel may have a fixed width (e.g., a bandwidth of 20 MHz) or a width that is dynamically set via signaling. The primary channel may be a working channel of the BSS and may be used by STAs to establish a connection with the AP. In certain representative embodiments, carrier sense multiple access (CSMA / CA) with collision avoidance (e.g., in an 802.11 system) may be implemented. For CSMA / CA, STAs (e.g., each STA) including the AP may sense the primary channel. If a particular STA senses / detects and / or determines that the primary channel is busy, the particular STA may back off. In a specified BSS, there is one STA (e.g., only one station) transmitting at any given time.

[0072] A high throughput (HT) STA may communicate using a 40 MHz wide channel (eg, by combining a 20 MHz wide primary channel with a 20 MHz wide adjacent or non-adjacent channel to form a 40 MHz wide channel).

[0073] Very high throughput (VHT) STA can support channels with widths of 20MHz, 40MHz, 80MHz and / or 160MHz. 40MHz and / or 80MHz channels can be formed by combining continuous 20MHz channels. A 160MHz channel can be formed by combining 8 continuous 20MHz channels or by combining two discontinuous 80MHz channels (this combination can be referred to as an 80+80 configuration). For the 80+80 configuration, after channel coding, the data can be transmitted and passed through a segment parser, which can separate the data into two streams. Inverse Fast Fourier Transform (IFFT) processing and time domain processing can be performed separately on each stream. The stream can be mapped on two 80MHz channels, and the data can be transmitted by the STA performing the transmission. At the receiver of the STA performing the reception, the above operations for the 80+80 configuration can be reversed, and the combined data can be sent to the medium access control (MAC).

[0074] 802.11af and 802.11ah support working modes below 1 GHz. Compared with 802.11n and 802.11ac, the channel working bandwidth and carrier used in 802.11af and 802.11ah are reduced. 802.11af supports 5MHz, 10MHz and 20MHz bandwidths in the TV White Space (TVWS) spectrum, and 802.11ah supports 1MHz, 2MHz, 4MHz, 8MHz and 16MHz bandwidths using non-TVWS spectrum. According to a representative embodiment, 802.11ah can support meter type control / machine type communication (MTC) (e.g., MTC devices in macro coverage areas). MTC devices can have certain capabilities, such as limited capabilities including support (e.g., only support) certain and / or limited bandwidths. MTC devices can include a battery, and the battery life of the battery is higher than a threshold (e.g., for maintaining a very long battery life).

[0075] For WLAN systems that can support multiple channels and channel bandwidths (e.g., 802.11n, 802.11ac, 802.11af, and 802.11ah), these systems include channels that can be designated as primary channels. The bandwidth of the primary channel can be equal to the maximum common operating bandwidth supported by all STAs in the BSS. The bandwidth of the primary channel can be set and / or limited by a certain STA, where the STA originates from all STAs operating in the BSS that supports the minimum bandwidth operating mode. In the example of 802.11ah, even if the AP and other STAs in the BSS support 2MHz, 4MHz, 8MHz, 16MHz, and / or other channel bandwidth operating modes, for STAs (e.g., MTC-type devices) that support (e.g., only support) 1MHz mode, the width of the primary channel can be 1MHz. Carrier sensing and / or network allocation vector (NAV) settings can depend on the status of the primary channel. If the primary channel is busy (e.g., because a STA (which only supports the 1MHz operating mode) transmits to the AP), then even if most of the available frequency bands remain idle and available for use, the entire available frequency band can be considered busy.

[0076] In the United States, the available frequency band for 802.11ah is 902MHz to 928MHz. In South Korea, the available frequency band is 917.5MHz to 923.5MHz. In Japan, the available frequency band is 916.5MHz to 927.5MHz. Depending on the country code, the total bandwidth available for 802.11ah is 6MHz to 26MHz.

[0077] Figure 1D 1 is a system diagram showing the RAN 113 and the CN 115 according to an embodiment. As described above, the RAN 113 may communicate with the WTRUs 102a, 102b, 102c using NR radio technology over the air interface 116. The RAN 113 may also communicate with the CN 115.

[0078] The RAN 113 may include gNBs 180a, 180b, 180c, though it will be appreciated that the RAN 113 may include any number of gNBs while remaining consistent with an embodiment. The gNBs 180a, 180b, 180c may each include one or more transceivers to communicate with the WTRUs 102a, 102b, 102c over the air interface 116. In one embodiment, the gNBs 180a, 180b, 180c may implement MIMO technology. For example, the gNBs 180a, 180b may use beamforming processing to transmit and / or receive signals to and / or from the gNBs 180a, 180b, 180c. Thus, for example, the gNB 180a may use multiple antennas to transmit wireless signals to the WTRU 102a and receive wireless signals from the WTRU 102a. In an embodiment, the gNBs 180a, 180b, 180c may implement carrier aggregation technology. For example, gNB 180a may transmit multiple component carriers to WTRU 102a (not shown). A subset of these component carriers may be on an unlicensed spectrum, while the remaining component carriers may be on a licensed spectrum. In an embodiment, gNBs 180a, 180b, 180c may implement coordinated multi-point (CoMP) technology. For example, WTRU 102a may receive coordinated transmissions from gNB 180a and gNB 180b (and / or gNB 180c).

[0079] The WTRUs 102a, 102b, 102c may communicate with the gNBs 180a, 180b, 180c using transmissions associated with a scalable digital configuration. For example, the OFDM symbol spacing and / or OFDM subcarrier spacing may be different for different transmissions, different cells, and / or different portions of the radio transmission spectrum. The WTRUs 102a, 102b, 102c may communicate with the gNBs 180a, 180b, 180c using subframes or transmission time intervals (TTIs) of different or scalable lengths (e.g., containing different numbers of OFDM symbols and / or lasting different absolute time lengths).

[0080] The gNBs 180a, 180b, 180c may be configured to communicate with the WTRUs 102a, 102b, 102c in a standalone configuration and / or a non-standalone configuration. In a standalone configuration, the WTRUs 102a, 102b, 102c may communicate with the gNBs 180a, 180b, 180c without accessing other RANs (e.g., the eNodeBs 160a, 160b, 160c). In a standalone configuration, the WTRUs 102a, 102b, 102c may use one or more of the gNBs 180a, 180b, 180c as mobility anchor points. In a standalone configuration, the WTRUs 102a, 102b, 102c may communicate with the gNBs 180a, 180b, 180c using signals in an unlicensed band. In a non-standalone configuration, the WTRU 102a, 102b, 102c may communicate / connect to the gNB 180a, 180b, 180c while communicating / connecting to another RAN (e.g., the eNode-B 160a, 160b, 160c). For example, the WTRU 102a, 102b, 102c may communicate with one or more gNBs 180a, 180b, 180c and one or more eNode-Bs 160a, 160b, 160c in a substantially simultaneous manner by implementing the DC principle. In a non-standalone configuration, the eNode-B 160a, 160b, 160c may act as a mobility anchor point for the WTRU 102a, 102b, 102c, and the gNB 180a, 180b, 180c may provide additional coverage and / or throughput to serve the WTRU 102a, 102b, 102c.

[0081] Each of the gNBs 180a, 180b, 180c may be associated with a particular cell (not shown) and may be configured to handle radio resource management decisions, handover decisions, user scheduling in UL and / or DL, support network slicing, dual connectivity, implement interworking between NR and E-UTRA, route user plane data to a user plane function (UPF) 184a, 184b, and route control plane information to an access and mobility management function (AMF) 182a, 182b, and the like. Figure 1D As shown, gNBs 180a, 180b, and 180c can communicate with each other through the Xn interface.

[0082] Figure 1DThe CN 115 shown may include at least one AMF 182a, 182b, at least one UPF 184a, 184b, at least one session management function (SMF) 183a, 183b, and may include a data network (DN) 185a, 185b. Although each of the aforementioned components is described as part of the CN 115, it should be understood that any of these components may be owned and / or operated by an entity other than the CN operator.

[0083] The AMF 182a, 182b may be connected to one or more of the gNBs 180a, 180b, 180c in the RAN 113 via the N2 interface and may act as a control node. For example, the AMF 182a, 182b may be responsible for authenticating users of the WTRU 102a, 102b, 102c, supporting network slicing (e.g., handling different PDU sessions with different requirements), selecting a specific SMF 183a, 183b, managing registration areas, terminating NAS signaling, and mobility management, etc. The AMF 182a, 182b may use network slicing processing to customize the CN support provided to the WTRU 102a, 102b, 102c based on the type of service used by the WTRU 102a, 102b, 102c. As an example, different network slices may be established for different use cases, such as services relying on Ultra Reliable Low Latency (URLLC) access, services relying on enhanced massive mobile broadband (eMBB) access, and / or services for machine type communication (MTC) access, etc. The AMF 182 may provide a control plane function for switching between the RAN 113 and other RANs (not shown) using other radio technologies (e.g., LTE, LTE-A, LTE-A Pro, and / or non-3GPP access technologies such as WiFi).

[0084] The SMF 183a, 183b may be connected to the AMF 182a, 182b in the CN 115 via the N11 interface. The SMF 183a, 183b may also be connected to the UPF 184a, 184b in the CN 115 via the N4 interface. The SMF 183a, 183b may select and control the UPF 184a, 184b, and may configure traffic routing through the UPF 184a, 184b. The SMF 183a, 183b may perform other functions such as managing and allocating WTRU or UE IP addresses, managing PDU sessions, controlling policy enforcement and QoS, and providing downlink data notifications, etc. The PDU session type may be IP-based, non-IP-based, Ethernet-based, etc.

[0085] The UPF 184a, 184b can be connected to one or more of the gNBs 180a, 180b, 180c in the RAN 113 via the N3 interface, which can provide the WTRUs 102a, 102b, 102c with access to packet-switched networks (e.g., the Internet 110) to facilitate communication between the WTRUs 102a, 102b, 102c and IP-enabled devices. The UPF 184, 184b can perform other functions, such as routing and forwarding packets, implementing user plane policies, supporting multi-host PDU sessions, processing user plane QoS, buffering downlink packets, and providing mobility anchor processing, etc.

[0086] The CN 115 may facilitate communications with other networks. For example, the CN 115 may include or may communicate with an IP gateway (e.g., an IP Multimedia Subsystem (IMS) server) that acts as an interface between the CN 115 and the PSTN 108. In addition, the CN 115 may provide the WTRUs 102a, 102b, 102c with access to other networks 112, which may include other wired and / or wireless networks owned and / or operated by other service providers. In one embodiment, the WTRUs 102a, 102b, 102c may be connected to the local data network (DN) 185a, 185b via an N3 interface connected to the UPF 184a, 184b and an N6 interface between the UPF 184a, 184b and the DN 185a, 185b and through the UPF 184a, 184b.

[0087] In view of Figures 1A-1D and about Figures 1A-1D In accordance with the corresponding description of the present invention, one or more or all of the functions described herein in comparison with one or more of the following may be performed by one or more simulation devices (not shown): WTRU 102a-d, base station 114a-b, eNodeB 160a-c, MME 162, SGW 164, PGW 166, gNB 180a-c, AMF 182a-b, UPF 184a-b, SMF 183a-b, DN 185a-b and / or one or more other devices described herein. These simulation devices may be one or more devices configured to simulate one or more or all of the functions described herein. For example, these simulation devices may be used to test other devices and / or simulate network and / or WTRU functions.

[0088] The simulation device can be designed to implement one or more tests about other devices in a laboratory environment and / or an operator network environment. For example, the one or more simulation devices can perform one or more or all functions while being implemented and / or deployed as a part of a wired and / or wireless communication network in whole or in part to test other devices inside the communication network. The one or more simulation devices can perform one or more or all functions while being temporarily implemented or deployed as a part of a wired and / or wireless communication network. The simulation device can be directly coupled to other equipment to perform the test, and / or can use over-the-air wireless communication to perform the test.

[0089] The one or more simulation devices can perform one or more functions, including all functions, while not being implemented or deployed as part of a wired and / or wireless communication network. For example, the simulation device can be used in a test lab and / or a test scenario of a wired and / or wireless communication network that is not deployed (e.g., testing) to implement tests on one or more components. The one or more simulation devices can be test devices. The simulation device can transmit and / or receive data using direct RF coupling and / or wireless communication with the aid of RF circuits (e.g., which can include one or more antennas).

[0090] As described above, systems and methods for optimizing streaming of dynamic point cloud sequences are disclosed according to some embodiments.

[0091] More specifically, systems and methods for optimizing streaming of dynamic point cloud sequences are disclosed in accordance with some embodiments. In some embodiments, rather than delivering, for example, all captured point cloud data in sequence as a complete snapshot, the point cloud can be dynamically processed as a constantly changing system with hierarchical rigid body transformations. In some embodiments, the transformations are continuously prioritized based on the importance of the visual impact each of these transformations has on the appearance of the point cloud. In some embodiments, the transformations are then transmitted from the server that is processing the captured sensor data to the client as a continuous stream of transformations (e.g., pairs of region and rigid body transformations). This eliminates the need to continuously send updated raw point cloud data. Such an example method can optimize data distribution for multiple scenarios.

[0092] Example point cloud collection and viewing according to some embodiments will now be described.

[0093] Figure 2An example system arrangement 200 is shown in which an example embodiment of the present disclosure may be employed. As shown, in some embodiments, the system arrangement 200 includes a server (e.g., a virtual reality (VR) server) 202 that can capture and process point clouds. The VR server 202 may include a processor 204 and a non-transitory computer-readable memory 206, which contains a plurality of instructions that can be executed by the processor 204 to implement embodiments of the various methods disclosed herein. Although not explicitly shown, the processor 204 and the memory 206 may be interconnected via a bus or similar mechanism. Coupled to the VR server 204 are a plurality of sensors 208, examples of which include cameras (e.g., camera arrays), which may be 3D stereo cameras, RGB-D cameras, and light detection and ranging (LIDAR) data collectors. The VR server 202 is also coupled to a database 210, which may, for example, maintain point cloud data (e.g., previously stored dynamic point cloud data, newly captured point cloud data, etc.). Although database 210 is shown as being external to VR server 202, in some embodiments, database 210 and VR server 202 may be co-located (eg, server 202 may include database 210).

[0094] In addition, the server 202 can be coupled to the Internet 220 and other networks 222 to communicate with the client 212. The client 212 may include a processor 214 and a non-transitory computer-readable memory 216 that may contain a plurality of instructions that can be executed by the processor 204 to implement embodiments of the various methods disclosed herein. In addition, in some embodiments, the memory 216 can be configured to store point cloud data. However, a separate storage medium may be provided for this purpose instead. In addition, as shown, the client 212 includes a VR viewing device 218. For example, either or both of the server 202 and the client 212 (e.g., the VR viewing device 218 itself) may be the above-mentioned Figures 1A-1D Note that Figure 2 The various entities shown in the drawings may be coupled to each other via any suitable wired and / or wireless links and / or other intervening elements not shown.

[0095] Example point cloud processing according to some embodiments will now be described.

[0096] Figure 3AAn example scene 300 is shown that may be received and processed in accordance with some embodiments. The scene includes a number of buildings at a distance and some closer objects that are imaged from an observer viewpoint having some apparent height. Additionally, the captured scene may have a "fixed" background and dynamic elements that move through it. However, if the viewer's position is dynamically changed, the "fixed" background may no longer be fixed. Moreover, the "stationary" scene changes when, for example, the camera moves. As the observer viewpoint changes, perhaps by moving lower or closer relative to buildings or other scene elements, the relative angles to points within the point cloud may change. The point cloud may be detected in a real world scene, generated with virtual objects, or any combination of these or other applicable techniques. However, changing point cloud data may require considerable bandwidth to transmit.

[0097] Figure 3B An example point cloud (image) 400 is shown that may be processed according to some embodiments. Figure 3B Although not explicitly shown in the figure, the point cloud image 400 will typically contain many points within the point cloud. These points represent three-dimensional (3D) coordinates where a portion of the object 402 has been detected to be present. Such detection can be performed using 3D sensors, such as light detection and ranging (LIDAR), stereo video, and RGB-D cameras. In addition, the point cloud data may include 3D position and radiometric image data or volume pixels (voxels).

[0098] As VR and augmented reality (AR) platforms move toward mass adoption by a wide range of consumers, the demand for high-resolution 3D content will increase. The traditional de facto standard for such full 3D content is polygonal 3D graphics, which are manually produced through modeling and rendered using tools and techniques used to create real-time 3D games. However, emerging mixed reality (MR) display and content capture technologies (such as VR head-mounted displays (HMDs), AR HMDs, RGB-D sensors, and light field cameras) have set new requirements for fully immersive 3D content production and distribution.

[0099] In addition to the development of MR-driven reality capture technology, in the near future, it can be expected that human-computer interaction systems will become more spatially aware. Environmental sensing technologies that enable structural understanding of the environment are increasingly embedded in smart home environments, automotive systems, and mobile devices. These spatially aware systems collect point cloud data from the environment, and therefore for certain applications, efficient tools for transmitting the environmental data in scenarios where such spatial data is to be transmitted across multiple systems and services may be required.

[0100] Traditional manual 3D modeling of 3D elements is often insufficient to capture the nuances of the real-world appearance of environments and humans, which has led to the development of newer techniques for capturing fully 3D content. These recent techniques include, for example, the use of multi-view camera stages (see, e.g., Collet, Alvaro, et al., "High-quality streamable free-viewpoint video", ACM Transactions on Graphics (TOG), 34.4:69, 2015), multiple RGB-D sensors (see, e.g., Orts-Escolano, Sergio, et al., "Holoportation: Virtual 3d teleportation in real-time", Proceedings of the 29th Annual Symposium on User Interface Software and Technology, 34.4:69, 2015), and the use of 3D-based image processing techniques. and Technology), ACM, pp. 741-754, 2016), and the use of light field cameras. With many of these methods, the capture system produces a dynamic point cloud sequence with a time-varying structure. As these emerging methods (which enable dynamic capture of real-world scenes as point clouds) become widely used, methods for efficiently distributing sequences of dynamic point cloud data will become increasingly important (see, for example, Thanou, Dorina et al., "Graph-based motion estimation and compensation for dynamic 3D point cloud compression", Image Processing (ICIP), 2015 IEEE International Conference on, pp. 3235-3239, 2015).

[0101] Raw capture of real-world scenes in a manner that preserves both image quality and depth / structure information places significant demands on capture and distribution systems due to the typically large size of the captured data. Example methods for distributing dynamic point cloud data sequences generated using dynamic sensors according to some embodiments can adapt to varying throughput of distribution channels and can render performance in a manner that prioritizes content updates based on relative visual importance. Visual importance can vary between clients; and consideration of this during data distribution can improve the overall quality of experience for users.

[0102] In some embodiments, possible optimizations may include treating the point cloud data sequence not as just a snapshot of the complete environment geometry at a moment in time, but rather as a continuum of changes to the environment geometry that can be streamed to the client, thereby prioritizing those changes with more significant visual changes over more subtle changes. Rather than streaming a "full frame" of the environment geometry, transformations of sub-regions of the geometry can be streamed. In some embodiments, the prioritization of the streamed changes depends on the amount of visual impact that the transformation has on a particular client, which can be a factor of the size of the area of ​​the geometry affected by the transformation and the amount of deformation (e.g., the amount of transformation) resulting. In addition, the visual impact is subjective to each client because the visual impact may depend on the client's observation of the geometry transformation (e.g., how far away from the client the transformation occurs).

[0103] Consider a video capture scene with a fixed background and dynamic elements moving through the video capture scene, the fixed background is no longer fixed in position relative to the observer as the observer's viewpoint changes.

[0104] In some embodiments, methods for optimizing the amount of data that may be required for a point cloud sequence are based on transforming the point cloud into a voxel grid using, for example, octree spatial partitioning (see, for example, De Gueiroz, Ricardo et al., "Compression of 3d point clouds using a region-adaptive hierarchical transform," IEEE Transactions on Image Processing, 25.8:3947-3956, 2016), or extended offline processing of pre-captured data to extract a full point cloud that is then reduced to a set of full keyframe point clouds with delta frames in between (see, for example, Collet, Alvaro et al., "High-quality streamable free-viewpoint video," ACM Transactions on Graphics (TOG), 34.4:69, 2015).

[0105] However, none of these approaches have been construed to take into account that in data captured from normal everyday environments, most objects are static or in motion that can be modeled as rigid body transformations. Furthermore, point cloud data can be considered hierarchical.

[0106] In accordance with some embodiments, the spatially hierarchical nature of a point cloud (e.g., an environment) includes multiple sub-objects that can be examined with different levels of detail. Similarly, in some embodiments, motion can be hierarchically decomposed from the motion of an entire object, down to the motion of the parts that make up the object, and then down to the motion of individual points that may actually be caused by sensor noise (rather than, for example, being actual motion). In some embodiments, hierarchical considerations regarding the point cloud also relate to how large areas and / or changes with a lot of movement are typically more visible to a viewer than subtle and smaller changes (e.g., changes related to sensor noise). Thus, in accordance with some embodiments, point cloud updates can be considered hierarchical in terms of their priority: for example, it is more important to visualize larger visible changes than smaller, more subtle changes.

[0107] According to some embodiments, disclosed herein is a method and system for consolidating 3D point cloud data by representing the 3D point cloud data as a continuously changing (e.g., time-varying) system with hierarchical rigid body transformations prioritized based on visual impact on the overall point cloud appearance.

[0108] As described above, systems and methods for optimizing streaming of dynamic point cloud sequences are disclosed in accordance with some embodiments. In some embodiments, instead of delivering, for example, all captured point cloud data in sequence as a complete snapshot, the point cloud can be dynamically processed as a constantly changing system with hierarchical rigid body transformations. In some embodiments, the transformations are continuously prioritized based on the importance of the visual impact each of these transformations has on the appearance of the point cloud. In some embodiments, the transformations can then be transmitted from the server that is processing the captured sensor data to the client as a continuous stream of transformations (e.g., pairs of region and rigid body transformations). These example techniques in accordance with some embodiments can exclude the continuous sending of updated raw point cloud data. Such an approach can optimize data distribution for a variety of scenarios.

[0109] Figure 4 An example message sequence chart 500 according to some embodiments is shown. Figure 4 In one example method shown in , a server captures point cloud data from a sensor and streams the data to a client in a (e.g., optimized) format, where instead of sending the raw point cloud data, a stream of transformations at different levels of point cloud sub-regions that approximate dynamic changes in the captured point cloud is sent to the client. Figure 4 An example sequence of operations performed by a client and a server according to some embodiments and an overview of the messaging that occurs between the server and the client in an example typical usage session are shown.

[0110] That is, reference Figure 4 At 508, the server 504 may send any (e.g., suitable) (one or more) messages to initialize the one or more sensors 506 to begin capturing point cloud data. At 510, the (one or more) sensors 506 may transmit a first (e.g., initial) point cloud (data) to the server 504. The point cloud data includes three-dimensional (3D) data. At 512, the server 504 may wait for a request from the client 502 for streaming of the point cloud data. Although Figure 4 504, but while waiting for the client request, the server 504 can continue to receive new point cloud data captured by the (one or more) sensors 506. At 514, the server 504 can receive a client request for point cloud data streaming. In response to the client request, at 516, the server 504 can store the current (e.g., most recent) point cloud received from the (one or more) sensors 506 as a reference point cloud. In addition, at 518, the server 504 can transmit the reference point cloud to the client 502.

[0111] At 520, the client 502 may provide to the server 504 (e.g., as a continuation of the point cloud data streaming request) a current client position relative to the reference point cloud. As will be described in more detail, in some embodiments, the current client position will be used to determine a distance that is used, for example, to prioritize areas within a given point cloud relative to the client viewpoint. Note that in some embodiments, the current client position may be provided as part of or concurrently with the client request for point cloud data streaming performed at 514. Additionally, in some embodiments, the client 502 (e.g., when the client 502 already knows the initial viewpoint to use without first receiving, for example, a complete reference point cloud) may signal its viewpoint along with the point cloud streaming request so that the server 504 may be able to adjust the resolution of the reference point cloud accordingly.

[0112] At 522, the server 504 receives a second (e.g., current, e.g., newly captured) point cloud (data) from the sensor(s) 506. At 524, the server 504 is configured to process the reference point cloud and the second point cloud accordingly to determine: (i) (one or more) 3D transformations (e.g., (one or more) rigid body transformations that approximate (one or more) changes in the second point cloud from the reference point cloud), whether such 3D transformations exist, and / or (ii) points representing (one or more) changes to be removed and / or added to modify the reference point cloud, such as, for example, to substantially reproduce (e.g., replicate) the deviation (one or more) between the reference point cloud and the second point cloud. Once the transformation(s) and / or points are determined, the server 504 may update the reference point cloud stored at the server 504.

[0113] In addition, at 526, the server 504 can send to the client 502 an indication of the region(s) in the reference point cloud to be transformed using, for example, the determined transformations (if they can be determined), or, for example, points to be added and / or removed from the reference point cloud, so that the client 502 can update the local copy of the reference point cloud on the client side. In this way, the client 502 can make the update(s) to reflect the changes that have occurred in the newly captured point cloud, rather than continuously receiving new complete sets of raw updated point cloud data from the server.

[0114] Now, some embodiments will be described in more detail. Figure 4 An exemplary overall process is shown in .

[0115] Figure 5 600 is a flowchart illustrating an example method of processing point cloud data according to some embodiments. In some embodiments, Figure 5 An example method may be executed by a server, such as Figure 4 Server 504 or Figure 2 Server 202.

[0116] like Figure 5 As shown, at step 602, the server may (e.g., continuously) receive point cloud data captured by a sensor. At 604, the server waits to see a client request a point cloud data stream. At step 606, the server determines whether a client request has been received. If a client request has been received at step 610, the server stores the current point cloud as a reference point cloud, and at step 612, sends the current point cloud to the client. At step 614, the server begins (or, e.g., continues) a process for streaming point cloud data to the client. In some embodiments, this process may be performed for each new point cloud instance captured from the sensor and received at the server.

[0117] The process begins (or, e.g., continues) at step 616, where the server receives from a viewing client a current viewpoint for rendering a given point cloud relative to the reference point cloud. In some embodiments, the client viewpoint is used to determine what changes in the given point cloud are important to the client, where a measure of importance is determined, e.g., by the amount of visual impact caused by a given change from the perspective of the client viewpoint.

[0118] At step 618, the server receives the newly captured point cloud. Then, at step 620, the server hierarchically examines the newly captured point cloud compared to the reference point cloud to isolate areas that have changed. Figure 5 As shown, in some embodiments, step 620 may include a sub-process 622 having several further sub-steps.

[0119] More specifically, at sub-step 624, the server identifies areas that have changed. In some embodiments, the goal is to identify which areas of the captured point cloud have changed since the most recent update in order to approximate a transformation that best estimates the changed point locations within the cluster corresponding to the changed areas. Thus, sub-step 626 involves clustering areas that deviate between the reference point cloud and the captured point cloud. In some embodiments, clustering includes isolating areas of the most recently captured point cloud that differ from the reference point cloud.

[0120] In sub-step 628, the server may determine (e.g., Figure 5 In some embodiments, the importance of the visual impact of the deviation is calculated as in , wherein the corresponding score for each cluster indicates the importance of the visual impact of the deviation. In some embodiments, such importance can be calculated, for example, as a weighted sum of the size of the deviation area, the deviation amount, and the distance of the deviation area from the current (virtual) viewpoint used by the client to inspect the point cloud on the client side, but of course this is an example and other formulas can be used.

[0121] At sub-step 630, for the first cluster with the highest score (and therefore the highest priority example), the server determines the shape correspondence between the two point clouds (i.e., the captured point cloud and the reference point cloud) near an area where the deviation has been clustered. As a general case, according to some embodiments, when an area with a deviation is first detected, it can generally be assumed that the deviation is caused by, for example, a fragment that has moved. In some embodiments, in order to find where the fragment has moved, it may be necessary to examine a larger area from the point cloud representing the second time step. If the deviation is caused by a rigid transformation of an element, the element may be relatively close to its original position. Therefore, in some embodiments, shape correspondence (e.g., matching) is performed to an area that may be slightly larger than the deviation area. In some embodiments, the size of the larger area can be controlled by checking the time length between frames and the expected movement speed of the element. Then, in sub-step 632, if, for example, a similar shape is found, in sub-step 634, the server may determine (e.g., solve) a transformation that approximates the deviation between the reference point cloud and the captured point cloud. At sub-step 636, the server may then add (eg, apply) the transformation to the reference point cloud and, at sub-step 638, send an indication of the region having the deviation and the transformation to the client.

[0122] In some embodiments, the deviated region may be sent as bounding volume coordinates that encapsulate the region. In some embodiments, the transformation may be sent as a transformation matrix, such as a 3 by 4 transformation matrix. As a general case, a transformation matrix is ​​a standard way to describe transformations in linear algebra and computer graphics, where a single matrix defines a 3D rigid body transformation including rotation and translation. Then, by multiplying the point coordinates (to be transformed) with the transformation matrix, the new point position can be obtained. See, for example, the article at the following address: en.wikipedia.org / wiki / Transformation_matrix.

[0123] If no similar shape is found in sub-step 632, sub-process 622 moves to sub-step 640, where the server determines points to be removed from and / or added to the reference point cloud based on the captured point cloud, and then in sub-step 642, the added and / or removed points are streamed to the client.

[0124] At sub-step 644, in some embodiments, the server may isolate additional finer-grained regions (also referred to herein as, for example, "sub-regions") that have not yet been addressed from the just processed cluster. These unaddressed regions may still have, for example, a significant amount of deviation. In some embodiments, the server may cluster those sub-regions into new deviation clusters and calculate an importance score (e.g., or other metric) for each of those new clusters as described above.

[0125] In sub-step 646, in response to a determination of time, processing, and communication bandwidth resource availability at the server and / or client side, the server may continue to process and partition the remaining clusters. For example, the server may return to sub-step 628 and continue processing the next still unprocessed cluster (e.g., which is a combination of previously formed clusters and new clusters corresponding to the sub-regions just processed) having the next highest importance score and thus (e.g., the next highest priority). However, according to this example, if the server determines that no more processing budget remains, then sub-process 622 ends and processing may return to, for example, step 614.

[0126] On the other hand, if at step 648, it is determined that, for example, the server or client has signaled a request to terminate the streaming, the process can end. Otherwise, the process can return to step 604.

[0127] As implemented in some embodiments and in Figure 4 and Figure 5 As generally shown in the examples of , the example methods and systems presented herein can enable efficient streaming of dynamic point cloud data sequences according to some embodiments. In some embodiments, the amount of data that may need to be transmitted can be optimized, for example, by considering changes in the point cloud data as sequential hierarchical rigid transformations that are prioritized, for example, according to the visual impact of the client viewing the data. In some embodiments, this can, for example, ensure that available bandwidth and processing power are consumed for content updates that may have the greatest impact on the quality of the user experience.

[0128] Compared to some other approaches for point cloud optimization, the example methods and systems disclosed herein according to some embodiments may generally operate well, for example, in typical uncontrolled capture environments, such as self-organized capture of motion in indoor or outdoor environments, where much of the geometry is static and transformations can generally be approximated with rigid body animations, such as individual objects in motion.

[0129] As spatial sensing is increasingly embedded in smart environments, automotive systems, and mobile devices, the example methods presented herein, according to some embodiments, may provide an efficient method for streaming data between systems and services that share environmental data. In most example cases, it may be expected that the environmental data includes both static environmental geometry and rigid objects moving in the environment. For such cases, in some embodiments, the example compression methods described herein, according to some embodiments, may be well suited.

[0130] According to some embodiments, changes can be prioritized according to the user's current viewpoint relative to the point cloud data, and delivered hierarchically, so that, for example, each region with changes can have additional sub-regions describing changes in the data at different levels. In the case of the point cloud data, each frame may have constant changes due to sensor noise. Advantageously, using the methods proposed herein, according to some embodiments, only the most important changes are communicated within the limits of the available bandwidth.

[0131] Figure 6 Flowchart of another example method 700 for processing point cloud data according to some embodiments is shown. In some embodiments, Figure 6 An example method may be executed by a server, such as Figure 4 Server 504 or Figure 2 Server 202. In some embodiments, Figure 6 The example method may be applied to situations where, for example, the server may examine the area of ​​the point cloud visible to the client at a given moment (e.g., a snapshot point cloud, as in step 720), rather than examining the entire point cloud for changes over time, or changes between time steps.

[0132] like Figure 6 As shown, at step 702, the server may (e.g., continuously) receive 3D data representing, for example, a user environment. The 3D data may be collected by a sensor, examples of which include RGB-D, LIDAR, and / or a combination of these and / or other sensors.

[0133] At 704, the server waits (e.g., here) for a client joint request for point cloud data and radiation image data, and at step 706, the server determines whether the client request has been received. If the client request has been received, at step 708, the server stores the current point cloud (3D data captured by the sensor) and the radiation image data as a reference point cloud, and at step 710, the server also sends the current point cloud and radiation image data to the client.

[0134] At step 712, the server begins (or, for example, continues) a process for streaming point cloud data to the client. In some embodiments, this process may be performed for each new point cloud instance captured from the sensor and received at the server.

[0135] The process begins (or, e.g., continues) at step 714, where the server receives and stores captured (e.g., sensor-collected) updates to the point cloud data (e.g., due to changes in the captured image of the real-world scene) and, e.g., updates to the radiometric image data (if any). Typically, in some embodiments, the point cloud data may be in the form of, e.g., primitive geometric shapes, and the colors of the geometric shapes may be derived from, e.g., a radiometric image that may be, e.g., distributed separately. However, in some embodiments, the geometric shapes and colors associated therewith may instead be included in the point cloud data.

[0136] The server receives the client's current viewpoint (e.g., within the immersive scene) at step 716. In response to receiving the client's current viewpoint, the server may determine a change in viewpoint relative to a viewpoint associated with the reference point cloud at steps 718 and 720, and receive a snapshot point cloud (of the captured scene) from the new perspective / viewpoint. Figure 6 Examples of may provide optimized processing, for example by limiting the checks on temporal changes of the point cloud performed only for the region(s) currently seen by the client.

[0137] At step 722, the server hierarchically checks the snapshot point cloud or the currently newly captured point cloud relative to the reference point cloud to isolate those areas that have changed. Figure 6 As shown, in some embodiments, step 722 may include a sub-process 724 having several further sub-steps.

[0138] More specifically, in sub-step 726, the server identifies the region that has changed and is similar in some respects to Figure 5 In the sub-process shown in , in sub-step 726, the server may cluster the areas that deviate between the reference point cloud and the snapshot point cloud. In addition, the server may determine (e.g., calculate) a corresponding score for each cluster, wherein in some embodiments, the corresponding score indicates the importance of the visual impact of the deviation. In some embodiments, such importance may be calculated, for example, as a weighted sum of the size of the deviated area, the amount of deviation, and the distance of the deviated area from the current viewpoint of the client.

[0139] At sub-step 728, for the first cluster with the highest score, and therefore for example for the highest priority example, the server determines shape correspondences between the two point clouds (i.e., the snapshot point cloud and the reference point cloud) that deviate from the clustered ones. Then, if a similar shape is found (e.g., detected) at sub-step 730, the server may determine (e.g., solve) a transformation that approximates the deviation between the reference point cloud and the snapshot point cloud at sub-step 732. At sub-step 734, the server then adds (e.g., applies) the transformation to the reference point cloud at sub-step 736 and sends an indication of the area having the deviation and the transformation to the viewing client. These sub-steps are described in detail in detail in detail in detail. Figure 5 Described in more detail in the description of .

[0140] If no similar shape is found at sub-step 730, the sub-process moves to sub-step 738, where the server determines points to be removed and / or added from the reference point cloud based on the snapshot point cloud, and at sub-step 740, the added and / or removed points are streamed to the client rather than, for example, the entire set of point cloud data.

[0141] At sub-step 742, in some embodiments, the server may isolate additional, finer-grained regions (also referred to herein as, for example, “sub-regions”) that have not yet been addressed from the cluster just processed. In some embodiments, the server may cluster these sub-regions into new divergent clusters and calculate an importance score (e.g., or other metric) for each of these new clusters, such as in combination with Figure 5 As described.

[0142] At sub-step 744, if the processing budget still allows (e.g., the processing time reserved for processing one instance of the captured point cloud, the bandwidth available for sending the transforms, or the estimated processing power on the client and / or server side), the server can continue to process and divide the remaining clusters. For example, the server can return to, for example, sub-step 728 and continue processing the next still unprocessed cluster with the next highest importance score (and therefore, for example, the next highest priority) (e.g., which is a combination of previously formed clusters and the new cluster corresponding to the sub-region just processed). However, if the server determines that no more processing budget remains, then sub-process 724 ends and processing can return to step 712.

[0143] On the other hand, if at step 746, it is determined that, for example, the server or the client has issued a request to terminate the streaming, the process can end. Otherwise, the process can return to step 704.

[0144] Fig. 7A and 7Bis a corresponding flow chart of another example method for processing point cloud data according to some embodiments. Note that Figure 7B yes Fig. 7A A continuation of the method steps shown in . Fig. 7A and 7B An example process performed by, for example, a server that may be responsible for capturing a point cloud obtained by collecting sensor data from sensors that may be used for environmental sensing is shown together. The sensor may be, for example, any of the following: (one or more) RGB-D sensors, (one or more) camera arrays, (one or more) stereo cameras that produce depth information, (one or more) LIDARs, etc. At the beginning of the process, at step 802, the server initializes the sensor to be ready to produce raw sensor data, and at step 804, waits for a client to request a (e.g., optimized) point cloud stream.

[0145] In some embodiments, when a client requests to receive a (e.g., optimized) dynamic point cloud stream from a server, in step 806, first the server and client can negotiate an appropriate processing budget. In some embodiments, the processing budget defines how much time and bandwidth can be used to stream data for each single dynamic point cloud update step from the server to the client. In some embodiments, in order to negotiate the budget to be used, the bandwidth of the network connection between the client and the server can be tested. In this regard, in some embodiments, the client can define a (e.g., required) update frame rate, which, for example, defines how much time the server has to process each update step. In addition, in some embodiments, the performance of the client can be tested to define the approximate number of transformations that the client can handle during each update step. In some embodiments, the client can also roughly estimate how many point cloud points the client can process and render during each display update cycle.

[0146] When a budget for processing has been defined in step 808, in step 810, the server may use the raw sensor data to capture the latest point cloud data and store the latest point cloud data as a reference point cloud. In step 812, the server may also send the reference point cloud to the client. In some embodiments, the server may reduce the complexity of the reference point cloud (e.g., using a suitable compression technique) when storing the reference point cloud, for example, if the data size is too high for the server or client to process in real time. Note that the initial reference frame may not depend on the client viewpoint.

[0147] In addition to sending a complete reference point cloud at the start of a session, in some embodiments, a new reference point cloud may be sent periodically, for example, to avoid data degradation caused by cumulatively approximating changes in the point cloud as a hierarchical transformation. A new reference point cloud may be sent using, for example, a fixed interval, such as sending a new complete reference point cloud every Nth update step. In some embodiments, other methods may be used to determine the magnitude of cumulative change that warrants sending a new reference point cloud. Such methods may, for example, ascertain the overall change between a currently captured scene and a scene sent earlier as a reference point cloud, and then send a new reference point cloud when that change exceeds a threshold.

[0148] When the reference point cloud has been sent to the client, the process proceeds to a continuous sub-process that is performed for each time step of the point cloud streaming. In this continuous sub-process, at step 812, the server first receives or predicts the latest client position relative to the reference point cloud. Then, at point 816, the server captures the raw sensor data into the latest point cloud, which is then processed to convert the changes that have occurred since the last capture into a prioritized transformation to the client.

[0149] Since point cloud data is a format for describing virtual content in full 3D format, different viewpoints can be selected to examine scene data. Therefore, in some embodiments, when optimizing the content, the viewpoint used by the client is also considered in order to more accurately meet the quality requirements of a specific client. In various embodiments herein, since changes in the dynamic point cloud are prioritized according to the magnitude of the visual impact of a particular change perceived by the viewer, the viewpoint of the content used by the viewer has an impact on the weighting of the visual impact of the particular change. In some embodiments, changes close to the viewer are typically more visible than similar changes on objects far away from the user (or viewer), and can therefore be prioritized accordingly. Therefore, in various embodiments described herein, there is a client that continuously reports to the server the value of the viewpoint used to examine the content.

[0150] At step 818, streaming of changes in the captured point cloud since the previously processed time step begins, which can be done by first isolating regions of the point cloud that deviate from the previously captured point cloud. The newly captured point cloud (and therefore the current point cloud to be processed) is compared to the reference point cloud. In some embodiments, all regions of captured points characterized by, for example, significantly deviated overall geometry are isolated into separate clusters.

[0151] In some embodiments, a cluster is, for example, an isolated (e.g.,) continuous region of a point cloud that is different from a reference point cloud. In some embodiments, a continuous region may refer to, for example, a region consisting of a collection (e.g., a group) of points that are relatively close to each other (e.g., located below a given threshold distance from each other). As will be described in more detail, for example, once the original cluster region has been processed, the cluster may be re-examined (e.g., re-examined in a subsequent process) to identify any finer-grained regions or sub-regions that are still deviating. In some embodiments, identification of clusters enables identification of which regions of the point cloud have changed since the last update step so that a transformation can then be approximated that best estimates the changed point positions within the cluster. In some embodiments, the clustering operation may be accomplished by first subtracting the newly captured point cloud from the reference point cloud, and thus comparing the point positions between the reference point cloud and the captured point cloud to a certain tolerance of the Euclidean distance, so that if there are points within the tolerance distance, the corresponding points in the newly captured point cloud may be marked as not deviating. According to an example, once the captured point cloud has been cleaned of non-deviating points, the result is the deviating regions. These regions can then be clustered into isolated deviating regions.

[0152] Point cloud registration and difference comparison can be done by iterative closest point method or other similar methods. See, for example, HUANG, Jing et al., "Point cloud matching based on 3D self-similarity", Computer Vision and Pattern Recognition Workshops (CVPRW), 2012 IEEE Computer Society Conference IEEE, 2012. p. 41-48; BOUAZIZ, Sofien et al., "Sparse iterative closest point", Computer Graphics Forum, Blackwell Publishing Limited, 2013. p. 113-123; and the article at the following URL: www.pointclouds.org / documentation / tutorials / cluster_extraction.php.

[0153] Reference now Figure 7B, when the deviation areas are divided into multiple clusters (step 818), in step 820, the multiple clusters are prioritized according to, for example, the visual impact of the changes contained in a given cluster on the client's viewing. As discussed above, the client viewpoint is important because, in some embodiments, it is used to determine what changes in the point cloud data are important to the client. In some embodiments, the visual impact can be determined so that relatively large changes near the viewing user and near the current focus of the viewing direction are more important than smaller changes away from the viewing user and more toward the back of the viewing user. In some embodiments, the magnitude of the change can depend on both: the area of ​​the change in the point cloud and the distance of the area from the viewer. In some cases, changes away from the viewpoint may not have a significant visual impact on the viewer.

[0154] In some embodiments, when calculating the priority of each cluster (in Figure 7B The distance used when calculating the distance of a point from a virtual viewpoint (also represented as "importance" in step 814) can be the distance of the point to a virtual viewpoint (for example, the client is using the virtual viewpoint to examine the point cloud on the client side). This distance can be calculated from the current client position provided in step 814. In some embodiments, for clusters, the distance can be the average distance of each point in the cluster to the virtual viewpoint that the client is using. In some embodiments, all points in the point cloud can be defined using a global point cloud coordinate system, and the client viewpoint can use this same coordinate system. For example, if the current viewpoint is at coordinates (x, y, z) in the point cloud, the viewpoint can be given as a combination of coordinates (x2, y2, z2) and a vector (x3, y3, z3). In some cases, the viewpoint vector can be a unit vector. In addition, in some embodiments, additional information such as the viewing frustum used by the client can be signaled to the server. For example, returning to reference Figure 6 In an example process of, only a portion of the point cloud visible to the client may be processed. In some embodiments, in order for the server to determine the visible area, the server may need to know the field of view (e.g., cone) in addition to, for example, the viewpoint and direction.

[0155] At step 822, the cluster with the highest priority (e.g., Figure 7B The highest calculated importance metric (e.g., score) in . As described above in conjunction with Figure 5 and 6 Said and will be combined Figure 8Described in more detail, processing for a given cluster (e.g., the cluster with the highest priority) in some embodiments involves determining a 3D transformation to be applied to the reference point cloud, or if such a transformation cannot be determined, determining points to be added and / or removed from the reference point cloud based on the changes.

[0156] In step 824, the clustering area in the reference point cloud is updated by the server according to the processing result. Figure 7B As shown, in step 824, information about the processing results (e.g., information related to transformation or point removal and / or addition) can also be sent by the server to the client to update the reference point cloud on the client side. In step 826, the server again checks the just processed cluster to identify other areas that are still deviated from the processed cluster (also referred to as "sub-regions" or "finer-grained residual change regions"). Once identified, new clusters are formed, and the importance of each new cluster is calculated as described above.

[0157] In step 828, the server determines whether there is still enough processing budget remaining to continue processing. If the determination result is positive, the process returns to step 822, where the newly formed cluster and its corresponding importance value are added to the previously formed cluster to again determine the cluster with the second highest priority, and continue to the processing box.

[0158] On the other hand, if the determination result is negative, the process moves to step 830, where a determination is made as to whether a request has been made, for example, by the client, to end the current session. If the determination result is negative, the process returns to step 814. If the determination result is positive, the process moves to step 832, where a determination is made as to whether a process termination has been requested. If so, the process ends at 834. If not, the process returns to step 804, where the process ends. Fig. 7A As shown, at this step, the server waits for another request for point cloud streaming from the client.

[0159]

[0046] An example clustering process according to some embodiments will now be described in more detail.

[0160] In some embodiments, in the processing of each cluster, the goal can be to find a rigid transformation that accurately describes the deviation between the reference point cloud and the captured point cloud. If no transformation is found that describes the deviation, then in some embodiments, points can be added and / or removed from the reference point cloud in order to make the reference point cloud, for example, similar to the captured point cloud.

[0161] Figure 8is a flow chart of an example method of processing a cluster according to some embodiments. In some embodiments, the processing can be performed by a server.

[0162] Processing of the clusters may involve assigning a respective priority to each cluster such that the cluster with the highest priority is selected for processing first.

[0163] Thus, at step 902, the server may select the cluster with the highest priority for processing. As described above, in some embodiments, the determination of the priority of a given cluster may be based on determining (e.g., calculating) the importance of the cluster (e.g., a score indicating the importance, or, for example, a priority), where the importance indicates the visual impact of the changes contained in the cluster relative to the viewer's current viewpoint. After selecting the cluster to be processed, at step 904, the server compares the reference point cloud and the newly captured point cloud to find shape correspondences of the cluster regions between the two point clouds. In some embodiments, the goal is to determine whether the same overall geometric shape exists in the two point clouds, which may mean that the cluster regions have been transformed between two points in time. In some embodiments, depending on whether a shape correspondence is found (at step 906), the processing branch may proceed to solving the transformation (at and after step 908) or (e.g., directly) move to the process of point addition and / or removal (at step 912).

[0164] In general, it will be appreciated that there are many example techniques for how shape correspondence between two point clouds may be performed. The process may involve, for example, a registration process of two point cloud datasets. The point cloud registration process may generally involve estimating correspondences, for example based on similarities between features and positions, from which a transformation may be estimated (such as in the form of a rigid transformation matrix that, when applied to one of the datasets, may align it with the other dataset). See, for example, the article at pointclouds.org / documentation / tutorials / registration_api.php#registration-api.

[0165] As an illustrative example, in some embodiments, the server can be configured to run a fairly rough shape estimate between the deviation clusters that are located relatively close to each other in the reference point cloud and the newly captured point cloud. If the result appears to be a match, the server can then perform an iterative closest point (ICP) algorithm to find a (e.g., optimal) transformation. In some embodiments, if necessary, the server can perform additional checks to check whether the error between the registered clusters is below, for example, a given threshold to ensure that there is a correspondence.

[0166] Note that, to take advantage of the example embodiments described herein, the accuracy of shape correspondence does not need to be relatively high, since the layered point cloud data processing or cluster checking as described herein according to some embodiments will correct for residual errors in the data. Thus, in some embodiments, performing ICP on the clusters may be sufficient, since even for poorly matched clusters, the subsequent sub-region clustering process (as described previously and described in more detail below) may correct for residual errors.

[0167] Furthermore, in some embodiments, finding shape correspondence (e.g., matching) may include generating a bounding volume of the clustered regions and matching it to a bounding volume of approximately the same size / shape. In this case, if the accuracy of the match needs to be increased, the bounding volume may be created with a more complex geometry (e.g., more complex than just a box).

[0168] If, in step 906, a similar geometric area is detected from the second point cloud to the deviating area detected as deviating from one point cloud, the server can determine that the deviation is caused by a transformation of the deviating area. In this case, in some embodiments, in step 908, a transformation is solved that can best describe the transformation. Typically, according to some embodiments, rigid transformations (e.g., translations and rotations) are considered and solved by forming a linear system B=R*A+t from two corresponding point cloud areas (A in the reference point cloud and B in the captured point cloud). From the linear system, R and t are solved by minimizing the least square error calculated based on the deviation of the points between the two corresponding point cloud areas. In order to solve R and t that give the least square error, a plurality of different analytical or iterative linear solvers can be used.

[0169] Transformations can include known methods for linear algebra and computer graphics (e.g., transformation matrices). A transformation matrix can combine rotation and translation into a single matrix. Using a transformation matrix, a point can be transformed by multiplying the point coordinates by the transformation matrix. A 3D transformation matrix is ​​typically 3 by 4 in shape. Using a rigid transformation, the first 3 by 3 portion can be interpreted as describing a rotation, while the last 3 by 1 portion can describe the translation. See, for example, the article at the following URL: en.wikipedia.org / wiki / Transformation_matrix.

[0170] At step 908, a transformation (e.g., a rigid transformation) is solved that approximates the deviation between the two point clouds. In some embodiments, the solved transformation can (e.g., optimally) transform the reference point cloud clusters to the captured point clusters. At step 910, the server can modify the reference point cloud by applying the transformation to the deviating area (also referred to herein as the "changed area") in the reference point cloud. At step 910, the transformation and an indication of the deviating area can also be sent by the server to the client so that the client can update the local copy of the reference point cloud on the client side accordingly. In some embodiments, the indication can be in the form of bounding volume coordinates indicating the changed area. In this way, the client can locally track updates of the reference point cloud by the server.

[0171] In the event that no shape correspondence is found (step 906), in step 912, the clustered reference point cloud area and the captured point cloud area may be examined by the server to define point additions (e.g., points to be added) and / or removals of points from the reference point cloud (e.g., points to be removed), which, for example, replicate the appearance of the deviated area in the captured point cloud. To illustrate, if there are holes in the geometry in the reference point cloud whose data now becomes visible, portions of the new data filling the holes in the captured point cloud are added to the reference point cloud by adding points. Alternatively, if points that appeared in the reference point cloud no longer exist in the captured point cloud, those points may be removed from the reference point cloud. Such point removal may, for example, be caused by an object moving outside the captured area. In step 912, when the server adds points to the reference point cloud and / or removes points from the reference point cloud, in some embodiments, such as Figure 8 As shown, the point information is also sent (e.g., streamed) to the client so that the client can perform the same point addition / removal operations on the client's local copy of the reference point cloud. In some embodiments, the added points can be downsampled from the full resolution data present in the captured point cloud. This may occur, for example, if the resolution of the captured point cloud is significantly higher relative to the resolution used in the reference point cloud.

[0172] When the transformation has been performed on the reference point cloud (step 910), or when points have been added and / or removed from the reference point cloud (step 912), the server may recheck the clustering regions (also referred to herein as "major change regions" or "major clustering regions") in step 914 to cluster possible sub-regions within the just processed clusters that may still deviate or show (one or more) residual changes (also referred to herein as "finer granularity regions" or "finer granularity residual change regions"). In some embodiments, in order to identify one or more sub-regions within the processed clusters with the highest priority, the server compares the captured point cloud with the updated reference point cloud to identify finer granularity regions (if any) that may still deviate from the updated reference point cloud.

[0173] As previously described, the updated reference point cloud may be the original reference point cloud to which transformations have been applied or which has been modified by point additions and / or removals. In some embodiments, new clusters may be defined for the identified sub-areas, and in step 916, the new clusters may be prioritized (e.g., assigned corresponding priorities) by determining (e.g., calculating) the corresponding importance of each new cluster. This calculation may be performed in a manner similar to that described above for the clusters corresponding to the primary cluster areas. Then, if in step 918 the server determines that there is still processing budget left, the server may add these new clusters to the list of clusters to be processed. Thus, the list of clusters will now include the previously prioritized primary cluster areas (based on their importance) and the new clusters with their respective importance values ​​or priorities. In some embodiments, the processing budget includes the time available for the server to calculate the required transformations and / or point additions / removals between each point cloud update cycle. If the processing budget is no longer available, then in step 920 the server may return to the processing budget as described above. Fig. 7A and 7B The main treatments shown (see e.g. Figure 7B and its corresponding description (eg, steps 830 and 832)).

[0174] If processing budget still remains, when the cluster with the highest priority has been processed and new clusters have been isolated for sub-regions within that cluster that still have deviations, the process returns to step 902, where the server selects the next cluster with the next highest priority to be processed from the prioritized cluster list. In this regard, in some embodiments, the next cluster with the next highest priority can be one of the remaining clusters previously identified for the major change region, or a cluster corresponding to one of the sub-regions (e.g., the sub-region within the just processed cluster with the previous highest priority). In some embodiments, Figure 8The illustrated process continuously prioritizes the identified clusters with the highest visual impact in order to ensure that as many visible changes as possible between point cloud update cycles are reproduced in the reference point cloud and sent to the client.

[0175] In some embodiments, the update process may be performed continuously until a user on the client side or the server side requests to terminate the process (e.g., Figure 5-8 Then, for example, depending on the request, the server-side processing can return to waiting for the client connection request or terminate the process entirely.

[0176] Exemplary variations according to some embodiments will now be described.

[0177] In some embodiments, the server may handle connections with multiple clients simultaneously. In this case, the server may, for example, continuously listen for client connection requests, and, for example, whenever a client signals an intent to connect to the server, the server may send a current reference point cloud to the connecting client. Thus, according to some embodiments, the various methods described herein may be performed for each of the clients actively connected to the server. In some embodiments, the server may also receive a given location from each of all connected clients so as to take the location into account, for example, when determining (e.g., calculating) a corresponding importance value (e.g., score) for a cluster processed for a particular client. As described above, in some embodiments, the client location may be used to determine the distance of an area within a given point cloud relative to, for example, a client viewpoint. In this regard, in some embodiments, as previously described, the importance calculation (and therefore, for example, cluster priority) is based at least in part on the distance of the changing area from the client's current viewpoint.

[0178] In some embodiments, various embodiments of the disclosed method may be performed by a server based on a pre-captured dynamic point cloud sequence. That is, in accordance with some embodiments, in this particular variation, the point cloud data may be stored on a storage medium and retrieved from the storage medium rather than from a sensor. Furthermore, in some embodiments, before any client requests the data, as a pre-processing step, the data may be transformed into a prioritized transformed (e.g., optimized) stream (e.g., using various techniques described herein in accordance with some embodiments).

[0179] In some embodiments, the server may additionally examine the point cloud data to estimate regional correspondence with predefined object models. The server may then indicate some natural markers to the viewer, which may be used by the client for contextual adjustment of viewing applications. In this regard, in some embodiments, when processing the point cloud (data), a separate shape classification may also be performed on the point cloud to detect, for example, whether a known object type can be identified from the point cloud.

[0180] In some embodiments, the server may analyze the point cloud to provide contextual information to the client along with the reference point cloud. For example, the server may analyze the point cloud to estimate the floor plane in the point cloud and provide that information to the client. The client may then use the floor plane information to align the point cloud display so that the floor plane represented within the point cloud data is consistent with the floor plane of the viewer's physical environment. In some embodiments, this is an example of how object recognition may be used. More specifically, separate object recognition may be performed by the server on the point cloud, and if something "contextually" important (e.g., the floor plane in this case) is detected, the server may transmit that information to the client.

[0181] In some embodiments, the server may use contextual understanding of the point cloud data to filter portions of the content. In this regard, in some embodiments, rather than signaling only contextually meaningful information to the client (e.g., as described above), the server may use this information to process the point cloud data on the server side. In some embodiments, filtering may be driven by a client request or some predefined setting (e.g., on the server side). For example, a client may request that the server only stream updates to dynamic elements in a scene, or filter out the floor and ceiling of the captured content. Thus, in some embodiments, the server may only stream updates to elements that are (e.g., at least partially) dynamic (and therefore potentially changing over time) and not static elements.

[0182] Example operations according to some embodiments will now be described. In one possible example operation, a camera array may be aimed at walls and doors. From the camera array data, a 3D point cloud may be generated, which may then be distributed to (one or more) clients. On the client side, MR glasses may be used to view the 3D point cloud in a manner that achieves motion parallax. Using the disclosed methods and systems according to some embodiments, a server may generate a point cloud from the camera array data. When a client connects to the server, the server may begin streaming a dynamic point cloud sequence to the client, which may begin with the first point cloud to be set as a reference point cloud. Local device tracking may be performed on the client device, and once the client has received the reference point cloud and selected an initial viewpoint, the client may begin sending viewpoint positions relative to the reference point cloud. In some embodiments, the client may select an initial viewpoint for the point cloud data based on local user preferences.

[0183] When a client first connects to a server and requests the (e.g., optimized) dynamic point cloud stream, the server sends a complete point cloud that is used as a reference point cloud. In some embodiments, after the reference point cloud has been sent to the client, only updates relative to the reference point cloud are sent (e.g., within a certain time period).

[0184] If there are no changes in the scene for a certain period of time, in some embodiments, the main data exchange involves the client sending its position relative to the point cloud to the server. During this time, the server uses the updated client viewpoint position to prioritize the changes that occur, continuously estimating the changes that occur in the dynamic point cloud data so that those changes with the greatest visual impact are transmitted to the client. In some embodiments, when capturing a static scene in which elements do not move or change, the sensor used to capture the point cloud may have measurement noise or inconsistent measurement density, which can cause the captured point cloud sequence to characterize dynamic behavior even if the captured point cloud sequence was recorded from a static scene. In some embodiments, the server will continuously detect changes in the point cloud data, use the latest client position to prioritize the changes, and stream the changes to the client as transformations based on the priorities associated with those changes.

[0185] Additionally, in this example operation (capturing walls and doors), the door may begin to open. The server may capture point cloud data from the time step when the door begins to open. The process may isolate the door as a cluster with deviating regions, and for each time step, find a transformation that approximates the motion of the region of the point cloud containing the door. When the sensor captures an image of a person opening the door, in some embodiments, the process may also begin sending these new points (capturing the person) that were not previously in the reference point cloud.

[0186] When the person opening the door is fully visible and new points (of the captured person) in the point cloud are sent to the client, then in some embodiments the person's movements can be approximated as coarse-level rigid body transformations, and during each frame of the capture, these rigid body transformations can be further estimated from a coarse level to a fine level (as, for example, described above) so that the deformation of the body appearance from captured frame to frame is continuously approximated.

[0187] Fig. 9 is a flow chart illustrating an example method 1000 according to some embodiments. In some embodiments, the method is performed by a server. At step 1002, the server sends a first point cloud to a client, wherein the first point cloud corresponds to a reference point cloud. At step 1004, the server receives a second point cloud. At step 1006, the server hierarchically determines changes in the second point cloud from the reference point cloud. To hierarchically determine the changes, at step 1108, the server identifies a first region in the second point cloud that has changed from the reference point cloud. To hierarchically determine the changes, at step 1110, the server further prioritizes the first regions. To hierarchically determine the changes, at step 1012, for a first region having a highest priority, the server determines whether there is a first rigid 3D transformation that approximates the first change from the reference point cloud. At step 1014, in response to determining that the first rigid 3D transformation exists, the server determines a first rigid 3D transformation that approximates the first change from the reference point cloud. Finally, in step 1016 , in response to determining that the first rigid 3D transformation does not exist, the server further determines a first point to be used to modify the reference point cloud, wherein the first point represents the first change.

[0188] Fig. 10A and 10B is a corresponding flow chart illustrating another example method 1100 according to some embodiments. Note that Fig. 10B yes Fig. 10AIn some embodiments, the method is performed by a server. In step 1102, the server transmits an initial point cloud to a client. In step 1104, the server hierarchically determines changes in a current point cloud from the initial point cloud. To hierarchically determine the changes, in step 1106, the server identifies a primary change region in the current point cloud. To hierarchically determine the changes, in step 1108, the server further uses a first primary change region to determine whether there is a first rigid 3D transformation that approximates a first change from the initial point cloud. In step 1110, in response to determining that the first rigid 3D transformation exists, the server determines the first rigid 3D transformation that approximates the first change from the initial point cloud. In step 1112, in response to determining that the first rigid 3D transformation does not exist, the server further determines a first point to be used to modify the initial point cloud, wherein the first point represents the first change. In step 1114, in order to hierarchically determine the changes, the server also identifies one or more finer-grained residual change regions within the primary change region. To determine the change hierarchically, at step 1116, the server further uses the one or more finer-grained residual change regions and the remaining main change regions to determine whether there is a second rigid 3D transformation that approximates the second change from the initial point cloud. At step 1118, in response to determining that the second rigid 3D transformation exists, the server determines the second rigid 3D transformation that approximates the second change from the initial point cloud. Finally, at step 1120, in response to determining that the second rigid 3D transformation does not exist, the server further determines a second point to be used to further modify the initial point cloud, wherein the second point represents the second change.

[0189] Furthermore, various (eg, related) embodiments have been described above.

[0190] According to some embodiments, a method of representing a time series of point cloud frames as a set of hierarchical 3D transformations applied to a reference point cloud is disclosed. In some embodiments, a series of rigid transformations can be used to change the reference point cloud. In some embodiments, additional signaling can add or remove points from the reference point cloud.

[0191] According to some embodiments, a method for encoder-side selection of a transform for representing content is disclosed.

[0192] According to some embodiments, optimizations for pre-captured content distributed to clients rather than live encoding are disclosed.

[0193] According to some embodiments, signaling of additional contextual analysis data, such as, for example, the location of the identified elements in addition to the point cloud and transformation is disclosed.

[0194] According to some embodiments, a method for providing 3D data to a client may include: receiving 3D data as a first point cloud; in response to receiving a request from the client, sending the first point cloud to the client and storing the first point cloud as a reference point cloud, and then iteratively receiving 3D data as a current point cloud; receiving a current observation viewpoint from the client; and in response to receiving the current observation viewpoint, determining the change of the viewpoint of the current point cloud relative to the viewpoint of the reference point cloud; performing a hierarchical check on the current point cloud relative to the reference point cloud to identify deviated areas; clustering the deviated areas; calculating the scores of the clustered deviated areas; determining the shape correspondence of the clustered deviated areas; and in response to determining the similar shapes of the clustered deviated areas, applying a transformation to the reference point cloud; and sending the transformation and an indication of the clustered areas to the client; in response to not determining the similar shapes of the clustered deviated areas, adding or removing points from the reference point cloud; and sending the added or removed points to the client.

[0195] According to some embodiments, another method for providing 3D data to a client may include: receiving 3D data as a first point cloud; in response to receiving a request from the client, sending the first point cloud to the client and storing the first point cloud as a reference point cloud; starting a process for streaming point cloud data to the client, iteratively executing the process for new point clouds until a termination request is received, the process including: receiving 3D data as a current point cloud; performing a hierarchical check on the current point cloud relative to the reference point cloud to identify deviation areas; clustering the deviation areas; calculating scores of the clustered deviation areas; determining a transformation that approximates the shape of the clustered deviation areas; and in response to determining the transformation, applying the transformation to the reference point cloud, and sending the transformation and an indication of the clustered areas to the client; in response to undetermined transformation, adding or removing points from the reference point cloud; and sending the added or removed points to the client.

[0196] In some embodiments, receiving 3D data may include receiving data from a sensor or receiving a pre-captured dynamic point cloud sequence from a storage medium. In some embodiments, the sensor data may be RGB-D captured data, LIDAR captured data, or some other type. In some embodiments, sending the first point cloud to the client may include sending radiation image data to the client. In some embodiments, the score may be calculated based on the weighted sum of the size of the area, the deviation amount, and the distance of the deviation area from the current viewpoint. In addition, in some embodiments, each method may further include determining whether there is time remaining in the processing budget for processing additional clusters, and if there is time remaining, processing additional clusters. In some embodiments, the method may further include negotiating a processing budget between the server and the client for setting time and bandwidth limits for point cloud updates.

[0197] A system may perform the method using a processor and a non-transitory computer readable medium storing instructions, the instructions being operable when executed by the processor to perform any of the disclosed methods. In some embodiments, the system may also include a 3D sensor.

[0198] Note that the various hardware elements of one or more described embodiments are referred to as "modules," which implement (i.e., execute, run, etc.) the various functions described herein in conjunction with the corresponding modules. As used herein, a module includes hardware that a person skilled in the relevant art deems suitable for a given implementation (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application specific integrated circuits (ASICs), one or more field programmable gate arrays (FPGAs), one or more memory devices). Each described module may also include executable instructions for performing one or more functions described as being performed by the corresponding module, and note that these instructions may take the form of or include hardware (i.e., hardwired) instructions, firmware instructions, and / or software instructions, etc., and may be stored in any suitable non-transitory computer-readable medium or media, such as media commonly referred to as RAM, ROM, etc.

[0199] Although the features and elements are described above in specific combinations, it will be understood by those skilled in the art that each feature or element may be used alone or in any combination with other features and elements. In addition, the methods described herein may be implemented in a computer program, software, or firmware embedded in a computer-readable medium and executed by a computer or processor. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, buffer memory, semiconductor storage devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks and digital versatile disks (DVDs). A processor associated with the software may be used to implement a radio frequency transceiver used in a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

1. A method performed by a server, comprising: Get the first point cloud and the second point cloud in the point cloud time series; determining a deviation area between the first point cloud and the second point cloud; selecting a first deviating region from among the deviating regions based on the corresponding importance scores; generating a representation of a change between the first point cloud and the second point cloud in the first deviation region; updating the first point cloud using the representation; repeating the determining, selecting, generating, and updating steps to obtain a series of representations of respective changes between the first point cloud and the second point cloud, wherein each of the representations specifies a region of the first point cloud and a change to be applied to the specified region, each such change comprising at least one of: a 3D transformation, a set of points to be added, and a set of points to be removed; and The series of representations is transmitted to the client.

2. A server device, comprising: at least one processor; as well as a memory storing instructions that, when executed by the at least one processor, cause the apparatus to: Get the first point cloud and the second point cloud in the point cloud time series, determining a deviation area between the first point cloud and the second point cloud, selecting a first deviating region from among the deviating regions based on the corresponding importance scores, generating a representation of changes between the first point cloud and the second point cloud in the first deviating region, updating the first point cloud using the representation, repeating the determining, selecting, generating, and updating steps to obtain a series of representations of respective changes between the first point cloud and the second point cloud, wherein each of the representations specifies a region of the first point cloud and a change to be applied to the specified region, each such change comprising at least one of: a 3D transformation, a set of points to be added, and a set of points to be removed, and The series of representations is transmitted to the client.

3. The method according to claim 1 or the device according to claim 2, wherein: The generated representation includes a first representation that captures coarse variations and a second representation that captures finer residual variations.

4. The method according to claim 1 or the device according to claim 2, wherein: The first point cloud and the second point cloud are obtained from one or more 3D sensors.

5. The method according to claim 1 or the device according to claim 2, wherein: The point cloud time series is accessed from a storage medium.

6. The method according to claim 1 or the device according to claim 2, wherein: The 3D transformation of the presentation of the series of representations includes rotation and translation.

7. The method according to claim 1 or the device according to claim 2, wherein: The divergent regions between the first point cloud and the second point cloud are determined by using a clustering technique.

8. The method according to claim 1 or the device according to claim 2, wherein: The importance score of the deviating region reflects the visual importance of the change between the first point cloud and the second point cloud in the deviating region.

9. The method according to claim 1 or the device according to claim 2, wherein: The instructions also cause the device to execute: determining a budget for representing the change between the first point cloud and the second point cloud; and The determining, selecting, generating and updating steps are repeated until the budget is exhausted.

10. The method according to claim 9 or the device according to claim 9, wherein: The budget includes at least one of an amount of time, an amount of processing, or an amount of bandwidth available to update the first point cloud based on the changes.

11. A method performed by a client, comprising: Get the first point cloud in the point cloud time series; receiving from a server a series of representations of respective changes to the first point cloud, wherein each representation specifies a region of the first point cloud and a change to be applied to the specified region, each such change comprising at least one of: a 3D transformation, a set of points to be added, and a set of points to be removed; as well as The representation is applied successively to the first point cloud to generate a point cloud representing a second point cloud in the time series of point clouds.

12. A client device, comprising: at least one processor; as well as a memory storing instructions that, when executed by the at least one processor, cause the apparatus to: Get the first point cloud in the point cloud time series, receiving from a server a series of representations of respective changes to the first point cloud, wherein each representation specifies a region of the first point cloud and a change to be applied to the specified region, each such change comprising at least one of: a 3D transformation, a set of points to be added, and a set of points to be removed, and The representation is applied successively to the first point cloud to generate a point cloud representing a second point cloud in the time series of point clouds.

13. The method according to claim 11 or the device according to claim 12, wherein: The region specified by the presentation of the series of representations comprises coordinates of a bounding volume encapsulating the region.

14. The method according to claim 11 or the device according to claim 12, wherein: A first representation and a second representation in the series of representations successively apply changes to respective first and second designated areas that overlap.

15. A method according to claim 11 or an apparatus according to claim 12, wherein the first representation and the second representation of the series of representations capture first and second changes in the area of ​​the first point cloud, respectively, wherein the first change includes a coarse deviation and the second change includes a finer residual deviation.

16. The method according to claim 11 or the device according to claim 12, wherein: The 3D transformation of the presentation of the series of representations includes rotation and translation.

17. The method according to claim 11 or the apparatus according to claim 12, wherein: The series of representations includes corresponding layered 3D transforms.

18. The method according to claim 11 or the apparatus according to claim 12, wherein: The series of representations approximates changes over time between the first point cloud and the second point cloud.

19. The method according to claim 11 or the apparatus according to claim 12, wherein: The instructions also cause the device to execute: Information regarding a processing budget available for generating the point cloud representing the second point cloud is sent to the server.

20. The method according to claim 11 or the apparatus according to claim 12, wherein: The instructions also cause the device to execute: Information about the viewpoint position relative to the second point cloud is sent to the server to facilitate prioritization of transmission of the series of representations to the client.

Images