System and method for optimizing dynamic point clouds based on prioritized transformations

Abstract

To provide a system and a method for optimizing streaming of a dynamic point clouds sequence.SOLUTION: An example disclosed method includes (i) transmitting a first point cloud to a client, wherein the first point cloud corresponds to a reference point cloud, (ii) receiving a second point cloud, and (iii) hierarchically determining changes in the second point cloud from the reference point cloud, wherein hierarchically determining the changes includes (a) identifying first areas in the second point cloud that have changed from the reference point cloud, and (b) for a first area having a highest priority, determining a first rigid 3D transformation that approximates a first change from the reference point cloud, and if the first rigid 3D transformation cannot be determined, further determining first points to be used to modify the reference point cloud, wherein the first points are representative of the first change.SELECTED DRAWING: Figure 5

Description

Background Art

[0001] Cross-reference This application is a non-provisional application of U.S. Patent Provisional Application No. 62 / 645,603, filed on March 20, 2018, entitled "System and Method for Optimizing Dynamic Point Clouds Based on Prioritized Transformations", which is incorporated herein by reference in its entirety, and claims the benefit under 35 U.S.C. § 119(e) thereof.

[0002] As new systems and sensors become available for capturing high-resolution three-dimensional (3D) data, the problem of transmitting dynamic point cloud sequences of large data sizes has newly emerged. For example, tests of the data throughput required to deliver a complete 3D appearance of a single person have revealed several network bottlenecks that limit the practicality of high-resolution data capture and distribution.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Non-Patent Document 2

Non-Patent Document 3

Non-Patent Document 4

Non-Patent Document 5

Non-Patent Document 6

Non-Patent Document 7

Non-Patent Document 8

Non-Patent Document 9

Summary of the Invention

Means for Solving the Problems

[0004] According to some embodiments, a method executed on a server includes transmitting a first point cloud to a client, where the first point cloud corresponds to a reference point cloud; receiving a second point cloud; and hierarchically determining changes in the second point cloud from the reference point cloud. The step of hierarchically determining changes includes identifying a first area in the second point cloud that has changed from the reference point cloud; prioritizing the first area; determining whether there is a first rigid 3D transformation that approximates a first change from the reference point cloud for the first area having the highest priority; determining a first rigid 3D transformation that approximates the first change from the reference point cloud in response to a determination that the first rigid 3D transformation exists; and determining a first point that will be used to modify the reference point cloud in response to a determination that the first rigid 3D transformation does not exist, where the first point represents the first change.

[0005] According to some embodiments, a method executed on a server includes transmitting a first point cloud to a client, where the first point cloud corresponds to a reference point cloud; receiving a second point cloud; and hierarchically determining changes within the second point cloud from the reference point cloud. The step of hierarchically determining changes includes identifying a first area within the second point cloud that has changed from the reference point cloud; prioritizing the first area; determining a first rigid 3D transformation that approximates a first change from the reference point cloud for the first area having the highest priority; and, if the first rigid 3D transformation cannot be determined, further determining a first point that would be used to modify the reference point cloud, where the first point represents the first change.

[0006] In some embodiments, the first point that would be used to modify the reference point cloud includes at least one of (i) a point that would be removed from the reference point cloud or (ii) a point that would be added to the reference point cloud. In some embodiments, the step of identifying the first area includes comparing the second point cloud to the reference point cloud to identify an area within the second point cloud that deviates from the reference point cloud. In some embodiments, the step of prioritizing the first area includes assigning a respective first priority to each of the first areas based at least in part on the current viewpoint of the client. In some embodiments, the step of assigning a respective first priority to each of the first areas includes determining the respective first priorities using the size of the main area, the amount of deviation of the first area from the reference point cloud, and the distance of the first area from the current viewpoint of the client. Further, in some embodiments, the step of determining whether a first rigid 3D transformation exists includes determining whether a shape correspondence between the reference point cloud and the second point cloud has been found.

[0007] In some embodiments, the method further includes receiving a current viewpoint from a client. In some embodiments, the method further includes storing a reference point cloud and, if a first 3D rigid transformation is determined, updating the reference point cloud stored at the server by applying the first rigid 3D transformation to the reference point cloud. In some embodiments, the method further includes sending a display and 3D transformation of a first area having the highest priority to the client to update the reference point cloud at the client. The display of the first main area having the highest priority includes, in some embodiments, bounding volume coordinates.

[0008] In some embodiments, the method further includes storing a reference point cloud and, if a first point is determined, updating the reference point cloud stored at the server by modifying the reference point cloud with the first point. In some embodiments, the method further includes sending a display of the first point to the client to update the reference point cloud at the client.

[0009] In some embodiments, the step of hierarchically determining changes includes identifying one or more first sub-areas within a first area having the highest priority, prioritizing the first sub-areas, and adding the prioritized first sub-areas to the remaining prioritized first areas. In some embodiments, the method includes, for the area having the next highest priority among the remaining prioritized first areas and the prioritized first sub-areas, (i) determining a second rigid 3D transformation that approximates a second change from the updated reference point cloud, or (ii) determining a second point that is to be used to modify the updated reference point cloud, the second point representing the second change. In this regard, in some embodiments, the method includes negotiating with the client about a processing budget, where the processing budget provides at least the amount of time and bandwidth available for point cloud update, and where the step of determining the second rigid 3D transformation or the second point for the area having the next highest priority is only executed when the processing budget is available.

[0010] In some embodiments, the step of identifying one or more first sub-areas includes identifying areas of finer granularity that deviate from the updated reference point cloud by comparing a second point cloud with the updated reference point cloud. In some embodiments, the step of prioritizing the first sub-areas includes assigning a respective second priority to each of the first sub-areas. In some embodiments, the method includes negotiating with the client about a processing budget, the processing budget providing at least the amount of time and bandwidth available for point cloud update.

[0011] In some embodiments, at least one of the reference point cloud or the second point cloud includes sensor data. In some embodiments, the reference point cloud and the second point cloud are received from a storage medium as a pre-captured dynamic sequence of point cloud data. Further, in some embodiments, the method further includes communicating with a plurality of clients and executing the method for each of the plurality of clients.

[0012] According to some embodiments, a method executed at a server includes transmitting an initial point cloud to a client and hierarchically determining changes within a current point cloud from the initial point cloud. The step of hierarchically determining changes includes identifying a main area of change within the current point cloud, determining whether there is a first rigid 3D transformation that approximates a first change from the initial point cloud using the first main area of change, determining a first rigid 3D transformation that approximates the first change from the initial point cloud in response to a determination that the first rigid 3D transformation exists, determining a first point that will be used to modify the initial point cloud in response to a determination that the first rigid 3D transformation does not exist, the first point representing the first change, identifying one or more finer-grained areas of residual change within the main area of change, determining whether there is a second rigid 3D transformation that approximates a second change from the initial point cloud using the one or more finer-grained areas of residual change and the remaining main area of change, determining a second rigid 3D transformation that approximates the second change from the initial point cloud in response to a determination that the second rigid 3D transformation exists, and determining a second point that will be used to further modify the initial point cloud in response to a determination that the second rigid 3D transformation does not exist, the second point representing the second change.

[0013] According to some embodiments, a method, executed at a server, for transmitting time-varying point cloud data includes transmitting an initial point cloud to a client and hierarchically determining changes within a current point cloud from the initial point cloud. Hierarchically determining changes includes identifying a main area of change within the current point cloud, determining a first rigid 3D transformation that approximates a first change from the initial point cloud using the first main area of change, and, if the first rigid 3D transformation cannot be determined, further determining a first point that would be used to modify the initial point cloud, the first point representing the first change. Hierarchically determining changes further includes identifying one or more finer-grained areas of residual change within the main area of change, determining a second rigid 3D transformation that approximates a second change from the initial point cloud using the one or more finer-grained areas of residual change and the remaining main area of change, and, if the second rigid 3D transformation cannot be determined, further determining a second point that would be used to further modify the initial point cloud, the second point representing the second change.

[0014] In some embodiments, the second rigid 3D transformation is determined for one of the one or more finer-grained areas of residual change. In some embodiments, a second rigid 3D transformation is determined for one of the remaining main areas of change. In some embodiments, a second rigid 3D transformation is determined for one of the remaining main areas of change. Further, in some embodiments, the method further includes transmitting to the client at least one of (i) the first rigid 3D transformation and first bounding volume coordinates indicative of the first changed area, (ii) the second rigid 3D transformation and second bounding volume coordinates indicative of the second changed area, (iii) the first point, and (iv) the second point.

[0015] According to some embodiments, a method executed on a server includes receiving first 3D data as an initial point cloud, sending the initial point cloud to a client and storing the initial point cloud as a reference point cloud in response to a request from the client for point cloud data streaming, and repeatedly executing a process for each current point cloud. The process includes receiving second 3D data as the current point cloud, receiving the second 3D data as the current point cloud, and performing a hierarchical inspection of the current point cloud against the reference point cloud. Performing includes identifying an area deviating from the reference point cloud, separating the deviating area into a first cluster, prioritizing the first cluster by calculating a respective first score indicating the importance for each of the first clusters, where each first score is calculated based at least in part on the current view point of the client, determining whether there is a transformation approximating a first deviation from the reference point cloud for the first cluster having the highest priority, if there is a transformation, applying the transformation to the stored reference point cloud to update the reference point cloud and sending the display and transformation of the area within the first cluster to the client, and if there is no transformation, updating the reference point cloud by performing at least one of adding or removing points from the stored reference point cloud and sending the points to the client.

[0016] In some embodiments, the step of performing a hierarchical inspection of the current point cloud with respect to the reference point cloud includes, after processing the first cluster having the highest priority, identifying a sub - area within the first cluster that still deviates from the updated reference point cloud; separating the deviating sub - area into a second cluster; prioritizing the second cluster by calculating respective second scores indicative of the importance for each of the second clusters, wherein each second score is calculated based at least on the current perspective of the client; adding the prioritized second cluster to the remainder of the prioritized first cluster; determining whether there is another transformation approximating a second deviation from the updated reference point cloud for the next cluster having the next highest priority; if there is another transformation, applying the another transformation to the updated stored reference point cloud to further update the reference point cloud; transmitting to the client another display of the sub - area within the processed first cluster and the another transformation; if there is no another transformation, performing at least one of adding or removing additional points from the updated reference point cloud to update the reference point cloud and transmitting the additional points to the client.

[0017] In some embodiments, this process ends upon request from the client or when the processing budget negotiated between the server and the client is no longer available for further processing.

[0018] Other embodiments include a system and a server configured to perform the methods described herein (e.g., having a processor and a non - transitory computer - readable medium storing a plurality of instructions for execution by the processor). In some embodiments, the system also includes at least one 3D sensor. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] A more detailed understanding will be obtained from the following description, which is presented as an example in connection with the accompanying drawings. Further, like reference numerals in the drawings indicate like elements.

[0020]

Figure 1A

Figure 1B

Figure 1C

Figure 1D

Figure 2

Figure 3A

Figure 3B

Figure 4

Figure 5

Figure 6

Figure 7A

Figure 7B

Figure 8

Figure 9

Figure 10A

Figure 10B

[0021] Entities, connections, configurations, etc. shown in and related to the various drawings are presented by way of example and not as limitations. Accordingly, any and all descriptions and other presentations regarding what a particular drawing shows, what a particular element or entity in a particular drawing is or has, and any and all similar descriptions that are to be read as absolute and thus limiting, and that may be read independently or taken out of context, are to be read appropriately only as preceded structurally by clauses such as "In at least one embodiment,...". For the sake of brevity and clarity of presentation, this implicit main clause is not repeated unnecessarily in the detailed description of the drawings.

[0022] Next, a detailed description of exemplary embodiments will be described with reference to the various drawings. It should be noted that this description provides detailed examples of possible implementations, but the details are intended to be exemplary and are not intended to limit the scope of this application.

[0023] An exemplary network for implementing an exemplary embodiment will next be described.

[0024] FIG. 1A is a diagram illustrating an exemplary communication system 100 in which one or more disclosed embodiments may be implemented. The communication system 100 may be a multi-connection system that provides content such as voice, data, video, messaging, broadcast, etc. to a plurality of wireless users. The communication system 100 may enable a plurality of wireless users to access such content by sharing system resources, including wireless bandwidth. For example, the communication system 100 may employ one or more channel access methods such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single carrier FDMA (SC-FDMA), zero-tail unique-word DFT-spread OFDM (ZT UW DTS-s OFDM), unique-word OFDM (UW-OFDM), resource block-filtered OFDM, filter bank multi-carrier (FBMC), etc.

[0025] As shown in Figure 1A, the communication system 100 may include wireless transmit / receive units (WTRUs) 102a, 102b, 102c, 102d, RAN 104 / 113, CN 106 / 115, public switched telephone network (PSTN) 108, Internet 110, and other networks 112, but it should be understood that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and / or network elements. Each of the WTRUs 102a, 102b, 102c, 102d may be any type of device configured to operate and / or communicate in a wireless environment. By way of example, the WTRUs 102a, 102b, 102c, 102d, any of which may sometimes be referred to as a "station" and / or "STA", may be configured to transmit and / or receive wireless signals and may be a user equipment (UE), mobile station, fixed or mobile subscriber station, subscription-based unit, pager, cellular phone, personal digital assistant (PDA), smartphone, laptop, netbook, personal computer, wireless sensor, hotspot or Mi-Fi device, Internet of Things (IoT) device, watch or other wearable, head-mounted display (HMD), vehicle, drone, medical device and application (e.g., remote surgery), industrial device and application (e.g., robot, and / or other wireless devices operating in the context of an industrial and / or automated processing chain), home electronic device, device operating on a commercial and / or industrial wireless network, etc. Any of the WTRUs 102a, 102b, 102c, and 102d may also be interchangeably referred to as a UE.

[0026] The communication system 100 may also include base station 114a and / or base station 114b. Each of base stations 114a, 114b may be any type of device configured to wirelessly interface with at least one of WTRUs 102a, 102b, 102c, 102d to facilitate access to one or more communication networks, such as CN106 / 115, Internet 110, and / or other network 112. By way of example, base stations 114a, 114b may be a transceiver base station (BTS), Node B, eNodeB, home Node B, home eNodeB, gNB, NR NodeB, site controller, access point (AP), wireless router, etc. Although base stations 114a, 114b are each shown as a single element, it will be appreciated that base stations 114a, 114b may include any number of interconnected base stations and / or network elements.

[0027] Base station 114a may be part of RAN104 / 113, which may also include other base stations and / or network elements (not shown), such as a base station controller (BSC), a radio network controller (RNC), a relay node, etc. Base station 114a and / or base station 114b may be configured to transmit and / or receive wireless signals on one or more carrier frequencies, sometimes referred to as a cell (not shown). These frequencies may be an authorized spectrum, an unlicensed spectrum, or a combination of an authorized spectrum and an unlicensed spectrum. A cell may provide wireless service coverage for a particular geographic area, which may be relatively fixed or may change over time. A cell may be further divided into cell sectors. For example, a cell associated with base station 114a may be divided into three sectors. Thus, in one embodiment, base station 114a may include three transceivers, i.e., one for each sector of the cell. In certain embodiments, base station 114a may employ multiple-input multiple-output (MIMO) technology and may utilize multiple transceivers for each sector of the cell. For example, beamforming may be used to transmit and / or receive signals in a desired spatial direction.

[0028] Base stations 114a, 114b may communicate with one or more of WTRUs 102a, 102b, 102c, 102d over an air interface 116, which may be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, millimeter wave, infrared (IR), ultraviolet (UV), visible light, etc.). The air interface 116 may be established using any suitable radio access technology (RAT).

[0029] More specifically, as described above, communication system 100 may be a plurality of access systems and may employ one or more channel access methods, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, etc. For example, base stations 114a and WTRUs 102a, 102b, 102c within RAN 104 / 113 may implement wireless technologies such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), where the air interfaces 115 / 116 / 117 may be established using Wideband CDMA (WCDMA). WCDMA may include communication protocols such as High Speed Packet Access (HSPA) and / or evolved HSPA (HSPA+). HSPA may include High Speed Downlink (DL) Packet Access (HSDPA) and / or High Speed UL Packet Access (HSUPA).

[0030] In certain embodiments, base stations 114a and WTRUs 102a, 102b, 102c may implement wireless technologies such as Evolved UMTS Terrestrial Radio Access (E-UTRA), where the air interface 116 may be established using Long Term Evolution (LTE) and / or LTE-Advanced (LTE-A) and / or LTE-A Pro.

[0031] In certain embodiments, base stations 114a and WTRUs 102a, 102b, 102c may implement wireless technologies such as NR radio access, where the air interface 116 may be established using New Radio (NR).

[0032] In one embodiment, base station 114a and WTRUs 102a, 102b, 102c may implement multiple radio access technologies. For example, base station 114a and WTRUs 102a, 102b, 102c may implement LTE radio access and NR radio access together, for example, using the dual connectivity (DC) principle. Accordingly, the air interfaces utilized by WTRUs 102a, 102b, 102c may be characterized by transmissions sent to / from multiple types of radio access technologies and / or multiple types of base stations (e.g., eNBs and gNBs).

[0033] In other embodiments, base station 114a and WTRUs 102a, 102b, 102c may implement wireless technologies such as IEEE802.11 (i.e., Wireless Fidelity (WiFi)), IEEE802.16 (i.e., Worldwide Interoperability for Microwave Access (WiMAX)), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Interim Standard 2000 (IS-2000), Interim Standard 95 (IS-95), Interim Standard 856 (IS-856), Global System for Mobile Communications (GSM), GMS Evolution Data Rates (EDGE), GSM EDGE (GERAN), etc.

[0034] The base station 114b in Fig. 1A may be, for example, a wireless router, a home Node B, a home eNodeB, or an access point, and may utilize any suitable RAT to smooth the wireless connectivity within a localized area such as an office, a home, a vehicle, a campus, industrial facilities, an aerial corridor (for example, for use by a drone), a road, etc. In certain embodiments, the base station 114b and the WTRUs 102c, 102d may implement a wireless technology such as IEEE 802.11 to establish a wireless local area network (WLAN). In certain embodiments, the base station 114b and the WTRUs 102c, 102d may implement a wireless technology such as IEEE 802.15 to establish a wireless personal area network (WPAN). In yet another embodiment, the base station 114b and the WTRUs 102c, 102d may utilize a cellular-based RAT (such as WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR, etc.) to establish a pico cell or a femto cell. As shown in Fig. 1A, the base station 114b may have a direct connection to the Internet 110. Thus, the base station 114b may not need to access the Internet 110 via the CN 106 / 115.

[0035] RAN 104 / 113 may be any type of network configured to provide voice, data, application, and / or Voice over Internet Protocol (VoIP) services to one or more of WTRUs 102a, 102b, 102c, 102d and may communicate with CN 106 / 115. The data may have different quality of service (QoS) requirements such as different throughput requirements, latency requirements, error tolerance requirements, reliability requirements, data throughput requirements, mobility requirements, etc. CN 106 / 115 may provide call control, billing services, mobile location-based services, prepaid calling, Internet connectivity, video distribution, etc. and / or may perform high-level security functions such as user authentication. Although not shown in Figure 1A, it will be appreciated that RAN 104 / 113 and / or CN 106 / 115 may communicate directly or indirectly with other RANs that employ the same radio access technology (RAT) or a different RAT than RAN 104 / 113. For example, in addition to being connected to RAN 104 / 113 which may utilize New Radio (NR) radio technology, CN 106 / 115 may also communicate with another RAN (not shown) that employs GSM, UMTS, CDMA2000, WiMAX, E-UTRA, or WiFi radio technology.

[0036] CN106 / 115 may also serve as a gateway for WTRU102a, 102b, 102c, 102d to access the PSTN108, the Internet 110, and / or other network 112. The PSTN108 may include a circuit-switched telephone network that provides plain old telephone service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices that use common communication protocols such as TCP, User Datagram Protocol (UDP), and / or IP in the Transmission Control Protocol (TCP) / Internet Protocol (IP) Internet protocol suite. The network 112 may include wired and / or wireless communication networks that are owned and / or operated by other service providers. For example, the network 112 may include another CN connected to one or more RANs that may employ the same RAT or a different RAT as the RAN104 / 113.

[0037] Some or all of the WTRU102a, 102b, 102c, 102d within the communication system 100 may include multimode capabilities (e.g., the WTRU102a, 102b, 102c, 102d may include multiple transceivers for communicating with different wireless networks over different wireless links). For example, the WTRU102c shown in Figure 1A may be configured to communicate with a base station 114a that may employ cellular-based wireless technology and a base station 114b that may employ IEEE802 wireless technology.

[0038] Figure 1B is a system diagram illustrating an exemplary WTRU 102. As shown in Figure 1B, the WTRU 102 may include, among other things, a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a keypad 126, a display / touchpad 128, a non-removable memory 130, a removable memory 132, a power supply 134, a global positioning system (GPS) chipset 136, and / or other peripheral devices 138. It should be understood that the WTRU 102 may include any sub-combination of the foregoing elements while maintaining a state consistent with an embodiment.

[0039] The processor 118 may be a general-purpose processor, a dedicated processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc. The processor 118 may perform signal coding, data processing, power control, input / output processing, and / or any other functionality that enables the WTRU 102 to operate in a wireless environment. The processor 118 may be coupled to the transceiver 120, which may be coupled to the transmit / receive element 122. Although Figure 1B shows the processor 118 and the transceiver 120 as separate components, it should be understood that the processor 118 and the transceiver 120 may be incorporated together in an electronic package or chip.

[0040] The transmitting / receiving element 122 can be configured to transmit signals to or receive signals from a base station (e.g., base station 114a) on the air interface 116. For example, in one embodiment, the transmitting / receiving element 122 can be an antenna configured to transmit and / or receive RF signals. In certain embodiments, the transmitting / receiving element 122 can be an emitter / detector configured to transmit and / or receive, for example, IR, UV, or visible light signals. In yet another embodiment, the transmitting / receiving element 122 can be configured to transmit and / or receive both RF signals and optical signals. It will be appreciated that the transmitting / receiving element 122 can be configured to transmit and / or receive any combination of wireless signals.

[0041] Although the transmitting / receiving element 122 is shown as a single element in FIG. 1B, the WTRU 102 can include any number of transmitting / receiving elements 122. More specifically, the WTRU 102 can employ MIMO technology. Thus, in one embodiment, the WTRU 102 can include two or more transmitting / receiving elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals on the air interface 116.

[0042] The transceiver 120 can be configured to modulate signals for transmission by the transmitting / receiving element 122 and to demodulate signals received by the transmitting / receiving element 122. As described above, the WTRU 102 can have multimode capabilities. Thus, the transceiver 120 can include multiple transceivers to enable the WTRU 102 to communicate via multiple RATs, such as, for example, NR and IEEE 802.11.

[0043] The processor 118 of the WTRU 102 may be coupled to the speaker / microphone 124, keypad 126, and / or display / touchpad 128 (e.g., a liquid crystal display (LCD) display unit or an organic light emitting diode (OLED) display unit) and may receive user input data therefrom. The processor 118 may also output user data to the speaker / microphone 124, keypad 126, and / or display / touchpad 128. Additionally, the processor 118 may access information from and store data in any suitable type of memory, such as the non-removable memory 130 and / or removable memory 132. The non-removable memory 130 may include random access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, etc. In other embodiments, the processor 118 may access information from and store data in a memory that is not physically located on the WTRU 102, such as a server or a home computer (not shown).

[0044] The processor 118 may receive power from the power source 134 and may be configured to distribute and / or control power to other components within the WTRU 102. The power source 134 may be any suitable device for powering the WTRU 102. For example, the power source 134 may include one or more dry cell batteries (e.g., nickel cadmium (NiCd), nickel zinc (NiZn), nickel metal hydride (NiMH), lithium ion (Li-ion), etc.), a solar cell, a fuel cell, etc.

[0045] Processor 118 may be coupled to a GPS chipset 136 configured to provide location information (e.g., longitude and latitude) regarding the current location of WTRU 102. In addition to, or instead of, information from GPS chipset 136, WTRU 102 may receive location information from a base station (e.g., base stations 114a, 114b) on air interface 116 and / or may determine its location based on the timing at which signals are received from two or more nearby base stations. It will be appreciated that WTRU 102 may collect location information by any suitable location determination method while maintaining a state consistent with an embodiment.

[0046] Processor 118 may further be coupled to other peripheral devices 138 that may include one or more software and / or hardware modules that provide additional features, functionality and / or wired or wireless connectivity. For example, peripheral devices 138 may include an accelerometer, an electronic compass, a satellite transceiver, a digital camera (for photos and / or video), a Universal Serial Bus (USB) port, a vibration device, a telephone transceiver, a hands-free headset, a Bluetooth® module, a Frequency Modulation (FM) radio unit, a digital music player, a media player, a video game player module, an Internet browser, a Virtual Reality and / or Augmented Reality (VR / AR) device, an activity tracker, etc. Peripheral devices 138 may include one or more sensors, where the sensors may be one or more of a gyroscope, an accelerometer, a Hall effect sensor, a magnetometer, an orientation sensor, a proximity sensor, a temperature sensor, a time sensor, a geolocation sensor, an altimeter, a light sensor, a touch sensor, a magnetometer, a barometer, a gesture sensor, a biometric sensor, and / or a humidity sensor.

[0047] The WTRU 102 may include full-duplex radio in which some or all of the transmission and reception of signals (e.g., related to specific subframes for both UL (e.g., for transmission) and downlink (e.g., for reception)) may be parallel and / or simultaneous. The full-duplex radio may include an interference management unit for reducing and / or substantially removing self-interference either via hardware (e.g., choke) or via signal processing via a processor (e.g., a separate processor (not shown) or via processor 118). In some embodiments, the WTRU 102 may include half-duplex radio for some or all of the transmission and reception of signals (e.g., related to specific subframes for either UL (e.g., for transmission) or downlink (e.g., for reception)).

[0048] Figure 1C is a system diagram illustrating a RAN 104 and a CN 106, according to an embodiment. As described above, the RAN 104 may employ E-UTRA radio technology to communicate with WTRUs 102a, 102b, 102c over an air interface 116. The RAN 104 may also communicate with the CN 106.

[0049] The RAN 104 may include eNode-Bs 160a, 160b, 160c, but it should be understood that the RAN 104 may include any number of eNode-Bs while maintaining a state consistent with some embodiments. Each of the eNode-Bs 160a, 160b, 160c may include one or more transceivers for communicating with WTRUs 102a, 102b, 102c over the air interface 116. In one embodiment, the eNode-Bs 160a, 160b, 160c may implement MIMO technology. Thus, the eNode-B 106a may use multiple antennas, for example, to transmit a wireless signal to and / or receive a wireless signal from the WTRU 102a.

[0050] Each of eNode-Bs 160a, 160b, and 160c may be associated with a particular cell (not shown) and may be configured to handle radio resource management decisions, handover decisions, scheduling of users in the UL and / or DL, etc. As shown in Figure 1C, eNode-Bs 160a, 160b, and 160c may communicate with each other over the X2 interface.

[0051] CN 106 shown in Figure 1C may include a Mobility Management Entity (MME) 162, a Serving Gateway (SGW) 164, and a Packet Data Network (PDN) Gateway (or PGW) 166. Although each of the foregoing elements is shown as part of CN 106, it will be appreciated that any of these elements may be owned and / or operated by an entity other than the CN operator.

[0052] MME 162 may be connected to each of eNode-Bs 160a, 160b, and 160c within RAN 104 via the S1 interface and may serve as a control node. For example, MME 162 may be responsible for authentication of users of WTRUs 102a, 102b, 102c, activation / deactivation of bearers, selection of a particular serving gateway during the initial attach of WTRUs 102a, 102b, 102c, etc. MME 162 may provide control plane functions for switching between RAN 104 and other RANs (not shown) that employ other radio technologies such as GSM and / or WCDMA.

[0053] SGW 164 may be connected to each of eNodeBs 160a, 160b, and 160c within RAN 104 via the S1 interface. SGW 164 may generally route and transfer user data packets to / from WTRUs 102a, 102b, 102c. SGW 164 may perform other functions such as anchoring of the user plane during eNodeB handover, triggering of paging when DL data is available to WTRUs 102a, 102b, 102c, and management and storage of the context of WTRUs 102a, 102b, 102c.

[0054] SGW 164 may be connected to a PGW 166 that can provide access to a packet switched network such as the Internet 110 to the WTRUs 102a, 102b, 102c, and facilitate communication between the WTRUs 102a, 102b, 102c and IP-enabled devices.

[0055] CN 106 can facilitate communication with other networks. For example, CN 106 can provide access to a circuit switched network such as the PSTN 108 to the WTRUs 102a, 102b, 102c, and facilitate communication between the WTRUs 102a, 102b, 102c and typical landline communication devices. For example, CN 106 can include, or communicate with, an IP gateway (e.g., an IP Multimedia Subsystem (IMS) server) that serves as an interface between CN 106 and the PSTN 108. Additionally, CN 106 can provide access to other networks 112, which can include other wired and / or wireless networks owned and / or operated by other service providers, to the WTRUs 102a, 102b, 102c.

[0056] Although the WTRU is shown as a wireless terminal in FIGS. 1A - 1D, in some representative embodiments, it is contemplated that such a terminal can use a wired communication interface with the communication network (e.g., temporarily or permanently).

[0057] In a representative embodiment, the other network 112 may be a WLAN.

[0058] A WLAN in infrastructure basic service set (BSS) mode may have an access point (AP) for the BSS and one or more stations (STAs) associated with the AP. The AP may have access or an interface to a distribution system (DS), or another type of wired / wireless network that carries traffic within and / or external to the BSS. Traffic for an STA originating from outside the BSS may arrive through the AP and may be delivered to the STA. Traffic originating from an STA to a destination outside the BSS may be sent to the AP to be delivered to their respective destinations. Traffic between STAs within the BSS may be sent, for example, through the AP, where the source STA may send the traffic to the AP and the AP may deliver the traffic to the destination STA. Traffic between STAs within the BSS may be considered and / or referred to as peer-to-peer traffic. Peer-to-peer traffic may be sent between the source STA and the destination STA (e.g., directly between them) using direct link setup (DLS). In some representative embodiments, the DLS may use 802.11e DLS or 802.11z tunneled DLS (TDLS). A WLAN using independent BSS (IBSS) mode may not have an AP, and STAs within and / or using the IBSS (e.g., all of the STAs) may communicate directly with each other. The IBSS communication mode may sometimes be referred to herein as the "ad hoc" communication mode.

[0059] When using the 802.11ac infrastructure operation mode or a similar operation mode, the AP may transmit beacons on a fixed channel, such as a primary channel. The primary channel may be of a fixed width (e.g., 20 MHz high bandwidth) or a width dynamically set via signaling. The primary channel may be the operating channel of the BSS and may be used by the STA to establish a connection with the AP. In some representative embodiments, Carrier Sense Multiple Access / Collision Avoidance (CSMA / CA) may be implemented, for example, within an 802.11 system. In the case of CSMA / CA, the STA (e.g., all STAs), including the AP, may sense the primary channel. If the primary channel is detected / determined to be busy by a particular STA, that particular STA may back off. Only one STA (e.g., only one station) may transmit at a given time within a given BSS.

[0060] A High Throughput (HT) STA may use a 40 MHz wide channel for communication by combining a primary 20 MHz channel with an adjacent or non - adjacent 20 MHz channel to form, for example, a 40 MHz wide channel.

[0061] Very High Throughput (VHT) STAs may support 20 MHz, 40 MHz, 80 MHz, and / or 160 MHz wide channels. 40 MHz and / or 80 MHz channels may be formed by combining contiguous 20 MHz channels. A 160 MHz channel may be formed by combining eight contiguous 20 MHz channels, or by combining two non - contiguous 80 MHz channels, sometimes called an 80 + 80 configuration. In the case of the 80 + 80 configuration, after channel encoding, the data may pass through a segment parser that may split the data into two streams. The Inverse Fast Fourier Transform (IFFT) process, and the time - domain processing may be performed separately on each stream. The streams may be mapped onto two 80 MHz channels, and the data may be transmitted by the transmitting STA. At the receiver of the receiving STA, the above operations for the 80 + 80 configuration may be reversed, and the combined data may be sent to the Media Access Control (MAC).

[0062] Sub - 1 GHz operation modes are supported by 802.11af and 802.11ah. The channel operating bandwidth, and carriers are reduced in 802.11af and 802.11ah compared to those used in 802.11n, and 802.11ac. 802.11af supports 5 MHz, 10 MHz and 20 MHz bandwidths in the TV White Space (TVWS) spectrum, and 802.11ah supports 1 MHz, 2 MHz, 4 MHz, 8 MHz, and 16 MHz bandwidths using non - TVWS spectrum. According to an exemplary embodiment, 802.11ah may support meter - type control / machine - type communication, such as MTC within a macro - coverage area. The MTC device may have limited capabilities, including some capabilities, such as support for some and / or limited bandwidths (e.g., support for only that). The MTC device may include a battery with a battery life exceeding a threshold (e.g., to maintain a very long battery life).

[0063] A WLAN system that can support multiple channels and channel bands such as 802.11n, 802.11ac, 802.11af, and 802.11ah includes channels that can be designated as primary channels. The primary channel may have a bandwidth equal to the maximum common operating bandwidth supported by all STAs within a BSS. The bandwidth of the primary channel can be set and / or limited by an STA from among all STAs operating within a BSS that support a minimum bandwidth operation mode. In the example of 802.11ah, even if an AP and other STAs within a BSS support 2MHz, 4MHz, 8MHz, 16MHz, and / or other channel bandwidth operation modes, the primary channel may be 1MHz wide for an STA (e.g., an MTC type device) that supports (e.g., only supports) the 1MHz mode. Carrier sensing and / or network allocation vector (NAV) setting may depend on the status of the primary channel. For example, if the primary channel is busy by an STA (that only supports the 1MHz operation mode), even if most of the frequency band remains idle and available, transmitting the entire available frequency bandwidth to an AP may be considered busy.

[0064] In the United States, the available frequency bandwidth that can be used by 802.11ah is from 902MHz to 928MHz. In South Korea, the available frequency band is from 917.5MHz to 923.5MHz. In Japan, the available frequency band is from 916.5MHz to 927.5MHz. The total available bandwidth for 802.11ah is from 6MHz to 26MHz depending on the country code.

[0065] FIG. 1D is a system diagram illustrating RAN 113 and CN 115 according to an embodiment. As described above, RAN 113 can communicate with WTRUs 102a, 102b, 102c on air interface 116 by adopting NR radio technology. RAN 113 may communicate with CN 115.

[0066] RAN 113 may include gNBs 180a, 180b, and 180c, but it should be understood that RAN 113 may include any number of gNBs while maintaining a state consistent with a certain embodiment. Each of gNBs 180a, 180b, and 180c may include one or more transceivers for communicating with WTRUs 102a, 102b, and 102c on air interface 116. In one embodiment, gNBs 180a, 180b, and 180c may implement MIMO technology. For example, gNBs 180a and 108b may use beamforming to transmit signals to and / or receive signals from gNBs 180a, 180b, and 180c. Thus, gNB 180a may use multiple antennas to transmit and / or receive wireless signals from, for example, WTRU 102a. In certain embodiments, gNBs 180a, 180b, and 180c may implement carrier aggregation technology. For example, gNB 180a may transmit multiple component carriers to WTRU 102a (not shown). A subset of these component carriers may be on unlicensed spectrum, while the remaining component carriers may be on licensed spectrum. In certain embodiments, gNBs 180a, 180b, and 180c may implement coordinated multipoint (CoMP) technology. For example, WTRU 102a may receive coordinated transmissions from gNB 180a and gNB 180b (and / or gNB 180c).

[0067] WTRUs 102a, 102b, and 102c may communicate with gNBs 180a, 180b, and 180c using transmissions related to scalable numerology. For example, the OFDM symbol interval and / or the OFDM subcarrier interval may be different for different transmissions, different cells, and / or different portions of the wireless transmission spectrum. WTRUs 102a, 102b, and 102c may communicate with gNBs 180a, 180b, and 180c using subframes or transmission time intervals (TTIs) of various or scalable lengths (e.g., including different numbers of OFDM symbols and / or lasting for different lengths of absolute time).

[0068] gNBs 180a, 180b, and 180c can be configured to communicate with WTRUs 102a, 102b, and 102c in a stand-alone configuration and / or a non-stand-alone configuration. In a stand-alone configuration, WTRUs 102a, 102b, and 102c can communicate with gNBs 180a, 180b, and 180c without accessing other RANs (e.g., eNode-Bs 160a, 160b, and 160c). In a stand-alone configuration, WTRUs 102a, 102b, and 102c can utilize one or more of gNBs 180a, 180b, and 180c as a mobility anchor point. In a stand-alone configuration, WTRUs 102a, 102b, and 102c can communicate with gNBs 180a, 180b, and 180c using signals in an unlicensed band. In a non-stand-alone configuration, WTRUs 102a, 102b, and 102c can communicate with / connect to gNBs 180a, 180b, and 180c while also communicating with / connecting to other RANs such as eNode-Bs 160a, 160b, and 160c. For example, WTRUs 102a, 102b, and 102c can implement the DC principle to communicate with one or more gNBs 180a, 180b, and 180c and one or more eNode-Bs 160a, 160b, and 160c substantially simultaneously. In a non-stand-alone configuration, eNode-Bs 160a, 160b, and 160c can serve as a mobility anchor for WTRUs 102a, 102b, and 102c, and gNBs 180a, 180b, and 180c can provide additional coverage and / or throughput for serving WTRUs 102a, 102b, and 102c.

[0069] Each of gNBs 180a, 180b, and 180c may be associated with a specific cell (not shown) and may be configured to handle radio resource management decisions, handover decisions, scheduling of users in UL and / or DL, support for network slicing, dual connectivity, interworking between NR and E-UTRA, routing of user plane data to user plane functions (UPFs) 184a, 184b, routing of control plane information to access and mobility management functions (AMFs) 182a, 182b, etc. As shown in FIG. 1D, gNBs 180a, 180b, and 180c may communicate with each other over the Xn interface.

[0070] CN 115 shown in FIG. 1D may include at least one AMF 182a, 182b, at least one UPF 184a, 184b, at least one session management function (SMF) 183a, 183b, and optionally, data networks (DNs) 185a, 185b. Although each of the foregoing elements is shown as part of CN 115, it should be understood that any of these elements may be owned and / or operated by entities other than the CN operator.

[0071] AMF 182a and 182b may be connected to one or more of gNBs 180a, 180b, and 180c within RAN 113 via the N2 interface and may serve as control nodes. For example, AMF 182a and 182b may be responsible for user authentication of WTRUs 102a, 102b, and 102c, support for network slicing (e.g., handling of different PDU sessions with different requirements), selection of specific SMFs 183a and 183b, management of the registration area, termination of NAS signaling, mobility management, etc. Network slicing may be used by AMF 182a and 182b to customize the CN to support WTRUs 102a, 102b, and 102c based on the type of services of the WTRUs 102a, 102b, and 102c being utilized. For example, different network slices may be established for different use cases such as services that rely on ultra-reliable low-latency (URLLC) access, services that rely on enhanced massive mobile broadband (eMBB) access, services for machine type communication (MTC) access, etc. AMF 162 may provide control plane functions for switching between RAN 113 and other RANs (not shown) that employ other radio technologies such as non-3GPP access technologies like LTE, LTE-A, LTE-A Pro, and / or WiFi.

[0072] SMF 183a and 183b may be connected to AMF 182a and 182b within CN 115 via the N11 interface. SMF 183a and 183b may also be connected to UPFs 184a and 184b within CN 115 via the N4 interface. SMF 183a and 183b may select and control UPFs 184a and 184b and configure traffic routing through UPFs 184a and 184b. SMF 183a and 183b may perform other functions such as management and allocation of UE IP addresses, management of PDU sessions, policy enforcement and QoS control, provision of downlink data notifications, etc. The PDU session type may be IP-based, non-IP-based, Ethernet-based, etc.

[0073] UPF184a and 184b may be connected to one or more of gNB180a, 180b, 180c within RAN113 via the N3 interface, which can provide access to a packet switched network, such as the Internet 110, to WTRU102a, 102b, 102c and facilitate communication between WTRU102a, 102b, 102c and IP-enabled devices. UPF184, 184b may perform other functions such as packet routing and forwarding, user plane policy enforcement, support for multi-home PDU sessions, handling of user plane QoS, buffering of downlink packets, and providing mobility anchoring.

[0074] CN115 can facilitate communication with other networks. For example, CN115 may include, or communicate with, an IP gateway (e.g., an IP Multimedia Subsystem (IMS) server) serving as an interface between CN115 and the PSTN108. Additionally, CN115 may provide access to other networks 112, which may include other wired and / or wireless networks owned and / or operated by other service providers, to WTRU102a, 102b, 102c. In one embodiment, WTRU102a, 102b, 102c may be connected to local data networks (DN) 185a, 185b through UPF184a, 184b via the N3 interface to UPF184a, 184b and the N6 interface between UPF184a, 184b and DN185a, 185b.

[0075] In view of FIGS. 1A-1D and the corresponding descriptions of FIGS. 1A-1D, one or more of the functions described herein with respect to one or more of WTRUs 102a-d, base stations 114a-b, eNodeBs 160a-c, MME 162, SGW 164, PGW 166, gNBs 180a-c, AMFs 182a-b, UPFs 184a-b, SMFs 183a-b, DNs 185a-b, and / or any other device described herein may be performed by one or more emulation devices (not shown). The emulation device(s) may be one or more devices configured to emulate one or more or all of the functions described herein. For example, the emulation device(s) may be used to test other devices and / or to simulate network and / or WTRU functionality.

[0076] The emulation device(s) may be designed to implement one or more tests of other device(s) in a laboratory environment and / or in an operator network environment. For example, one or more emulation device(s) may perform one or more or all of the functions while being fully or partially implemented and / or deployed as part of a wired and / or wireless communication network to test other devices in the communication network. One or more emulation device(s) may perform one or more or all of the functions while being temporarily implemented / deployed as part of a wired and / or wireless communication network. The emulation device(s) may be directly coupled to another device for testing purposes and / or may perform the tests using over-the-air wireless communication.

[0077] One or more emulation devices are not implemented / deployed as part of a wired and / or wireless communication network, but can perform one or more functions, including all. For example, an emulation device can be utilized in a test scenario in a test laboratory and / or in a non-deployed (e.g., test) wired and / or wireless communication network to implement tests for one or more components. One or more emulation devices can be test equipment. Direct RF coupling and / or wireless communication via an RF circuit (which may include one or more antennas) can be used by the emulation device to transmit and / or receive data.

[0078] As described above, systems and methods for optimizing the streaming of a dynamic point cloud sequence according to some embodiments are disclosed.

[0079] More specifically, systems and methods for optimizing the streaming of a dynamic point cloud sequence according to some embodiments are disclosed. In some embodiments, for example, rather than delivering all captured point cloud data in order as a complete snapshot, the point cloud can be dynamically handled as the system constantly changes with hierarchical rigid body transformations. In some embodiments, these transformations are continuously prioritized according to the importance of the visual effects they each cause to the point cloud appearance. In some embodiments, the transformations are then communicated from a server that is processing the captured sensor data for the client as a continuous stream of transformations (e.g., area and rigid body transformation pairs). This eliminates the need to continuously send updated raw point cloud data. Such an exemplary method can optimize data distribution for multiple scenarios.

[0080] Exemplary point cloud collection and viewing according to some embodiments are then described.

[0081] FIG. 2 illustrates an exemplary system arrangement 200 in which an exemplary embodiment of the present disclosure may be employed. As shown, the system arrangement 200 includes, in some embodiments, a server (e.g., a virtual reality (VR) server) 202 that can capture and process point clouds. The VR server 202 may include a processor 204 and a non-transitory computer-readable memory 206 that includes a plurality of instructions executable by the processor 204 to execute various method embodiments disclosed herein. Although not explicitly shown, the processor 204 and the memory 206 may be interconnected via a bus or similar mechanism. Coupled to the VR server 204 are a number of sensors 208, examples of which may include cameras (e.g., camera arrays) such as 3D stereo cameras, RGB-D cameras, and light detection and ranging (LIDAR) data collectors. The VR server 202 is also coupled to a database 210 that can hold, for example, point cloud data (e.g., previously stored dynamic point cloud data, newly captured point cloud data, etc.). Although the database 210 is shown as being located external to the VR server 202, in some embodiments, the database 210 and the VR server 202 may be co-located (e.g., the server 202 may include the database 210).

[0082] Furthermore, server 202 may be coupled to the Internet 220 and other networks 222 to communicate with client 212. Client 212 may include a processor 214 and a non-transitory computer-readable memory 216 that may contain a plurality of instructions executable by processor 204 to implement various method embodiments disclosed herein. Additionally, in some embodiments, memory 216 may be configured to store point cloud data. However, a separate storage medium may be provided instead for that purpose. Further, as shown, client 212 includes a VR viewing device 218. For example, either or both of server 202 and client 212 (e.g., VR viewing device 218 itself) may be a WTRU as described above with respect to FIGS. 1A-1D. The various entities shown in FIG. 2 may be coupled to each other via any suitable wired and / or wireless links and / or other intermediate elements shown hereinafter.

[0083] Exemplary point clouds according to some embodiments are described next.

[0084] FIG. 3A illustrates an exemplary scene 300 that may be received and processed according to some embodiments. The scene includes a plurality of buildings and some closer objects at a distance, imaged from an observer's perspective at a certain apparent height. Further, the captured scene may have a "fixed" background and dynamic elements moving therein. However, if the viewer's position is dynamically changed, the "fixed" background may no longer be fixed. Also, a "static" scene changes, for example, when a video camera moves. In some cases, as the viewer's perspective changes by moving lower or closer to a building or other scene element, the relative angles to the points in the point cloud may change. The point cloud may be detected in a real-world scene, generated with virtual objects, or any combination of these techniques, or other techniques where applicable. However, changing the point cloud data may require a fairly wide bandwidth for transmission.

[0085] Figure 3B illustrates an exemplary point cloud (image) 400 that can be processed, according to some embodiments. Although not explicitly shown in Figure 3B, the point cloud image 400 will typically include many points within the point cloud. The points represent three-dimensional (3D) coordinates that have been detected as being where a portion of the object 402 exists. Such detection can occur using 3D sensors such as light detection and ranging (LIDAR), stereo video, and RGB-D cameras. Additionally, the point cloud data can include 3D location, and radiographic image data or volume pixels (voxels).

[0086] As VR and augmented reality (AR) platforms are developed for mass adoption by more consumers, the demand for high-resolution 3D content will increase. The traditional de facto standard for such full 3D content is polygonal 3D graphics that are manually created by modeling and rendered using tools and techniques used to produce real-time 3D games. However, newly emerging mixed reality (MR) display and content capture technologies such as VR head-mounted displays (HMDs), AR HMDs, RGB-D sensors, and bright-field cameras are setting new requirements for fully immersive 3D content production and distribution.

[0087] Ahead of MR driving the development of reality capture technologies, it can be expected that in the near future, human-computer interaction systems will become more spatially aware. Environmental sensing technologies that enable an understanding of the structure of the environment are being increasingly embedded in smart home environments, automotive systems, and mobile devices. These spatial awareness systems collect point cloud data from the environment and thus, for certain applications, may require efficient tools for communicating environmental data in scenarios where such spatial data is transmitted across multiple systems and services.

[0088] Traditional manual 3D modeling of 3D elements is often insufficient to capture the nuances of the environment and the human appearance in the real world, leading to the development of newer techniques for capturing fully 3D content. Such recent techniques include, for example, the use of multi-view camera stages (see, e.g., Non-Patent Document 1), the use of multiple RGB-D sensors (see, e.g., Non-Patent Document 2), and the use of bright-field cameras. For many of these methods, the capture system creates a sequence of dynamic point clouds with a temporally varying structure. As these newly emerging methods that enable the dynamic capture of real-world scenes such as point clouds become widely used, techniques for efficiently distributing a sequence of dynamic point cloud data become increasingly important (see, e.g., Non-Patent Document 3).

[0089] The raw capture of real-world scenes in a way that preserves both image quality and depth / structure information strongly demands capture and distribution systems due to the generally large size of the captured data. Exemplary methods according to some embodiments for distributing a sequence of dynamic point cloud data created using dynamic sensors can be adapted to the different throughputs of distribution channels and render performance in a way that prioritizes content updates based on relative visual importance. The visual importance may vary for each client, and clarifying this during data distribution can improve the overall quality of the experience for the user.

[0090] In some embodiments, a possible optimization is to not treat a sequence of point cloud data as simply a snapshot of the complete environmental geometry for some time, but instead treat it as a series of changes within the environmental geometry that can be streamed to the client, and may include prioritizing changes that have a visual change that is more significant than a more subtle change. Instead of streaming the "full frame" of the environmental geometry, the transformation of a lower region of the geometry can instead be streamed. In some embodiments, the prioritization of the changes to be streamed may depend on a factor of the size of the area of the geometry affected by the transformation and the amount of the resulting change (e.g., the amount of the transformation), and the amount of the visual impact that the transformation has on a particular client. Further, since the visual impact may depend on the observation of the client of the geometry transformation (e.g., how far from the client the transformation occurs), the visual impact depends on the individual client.

[0091] Considering a scene captured by video with a fixed background, and a dynamic element moving within that captured scene, when the observer's viewpoint is changed, the position of the fixed background is no longer fixed with respect to the observer.

[0092] In some embodiments, a technique for optimizing the amount of data that may be required by a point cloud sequence is based on converting the point cloud to a voxel grid, for example, using extensive offline processing of pre-captured data (see, e.g., Non-Patent Document 1) to extract a full point cloud that is then reduced to a set of full key frames with delta frames in between, such as an octree spatial partitioning (see, e.g., Non-Patent Document 4).

[0093] However, none of these methods are understood to take into account that within data captured from a normal everyday environment, most objects are either static or in a motion state that can be modeled as a rigid body transformation. Further, the point cloud data can be considered hierarchically.

[0094] According to some embodiments, the spatially hierarchical nature of a point cloud (e.g., an environment) includes a plurality of sub-objects that can be examined at different levels of detail. Similarly, in some embodiments, motion can be hierarchically decomposed from the motion of an entire object to the motion of the parts that make up that object and then to the motion of individual points that may actually be caused by sensor noise (e.g., rather than actual motion). In some embodiments, the hierarchical consideration of the point cloud relates to how visible to a viewer changes with a large area and / or a large amount of movement are generally compared to more subtle and smaller changes (e.g., changes related to sensor noise). Thus, according to some embodiments, point cloud updates can be considered hierarchical in their priority: for example, it is more important to visualize larger visual changes than smaller, more subtle changes.

[0095] A method and system for integrating 3D point cloud data are disclosed herein by representing 3D point cloud data as a system that always changes (e.g., temporally) with a hierarchically rigid transformation prioritized based on the visual impact on the overall appearance of the point cloud according to some embodiments.

[0096] As described above, systems and methods for optimizing the streaming of dynamic point cloud sequences according to some embodiments are disclosed. In some embodiments, for example, rather than delivering all captured point cloud data in order as a complete snapshot, the point cloud may be treated dynamically as a system that is constantly changing with hierarchical rigid body transformations. In some embodiments, these transformations are continuously prioritized according to the importance of the visual effects they each cause to the point cloud appearance. In some embodiments, the transformations may then be communicated from the server processing the captured sensor data to the client as a continuous stream of transformations (e.g., area and rigid body transformation pairs). These exemplary techniques according to some embodiments may eliminate the step of continuously sending updated raw point cloud data. Such methods may optimize data distribution for multiple scenarios.

[0097] Figure 4 illustrates an exemplary message sequence diagram 500 according to some embodiments. In one exemplary method, illustrated in Figure 4 according to some embodiments, the server captures point cloud data from the sensor and instead of sending the raw point cloud data, a stream of transformations at different levels of the point cloud sub-regions that approximate the dynamic changes within the captured point cloud is sent to the client in an (e.g., optimized) form, and the data is streamed to the client. Figure 4 illustrates an exemplary sequence of operations performed by the client and the server, as well as an overview of the messaging that occurs between the server and the client in an exemplary general usage session.

[0098] That is, referring to FIG. 4, at 508, the server 504 can initialize one or more sensors 506 and send any (e.g., suitable) message to start capturing point cloud data. At 510, the sensor 506 can communicate a first (e.g., initial) point cloud (data) to the server 504. The point cloud data includes three-dimensional (3D) data. At 512, the server 504 can wait for a request from the client 502 for point cloud data streaming. Although not explicitly shown in FIG. 4, while waiting for the client request, the server 504 can continue to receive new point cloud data captured by the sensor 506. At 514, the server 504 can receive a client request for point cloud data streaming. In response to the client request, at 516, the server 504 can store the current (e.g., final) point cloud received from the sensor 506 as a reference point cloud. Further, at 518, the server 504 can communicate that reference point cloud to the client 502.

[0099] At 520, the client 502 can provide the server 504 with the current client location with respect to the reference point cloud (e.g., as part of a continuation of a point cloud data streaming request). As will be described in more detail, the current client location will be used, in some embodiments, to determine distances, for example, when prioritizing areas within a given point cloud with respect to the client viewpoint. Note that in some embodiments, the current client location can be provided as part of or in parallel with the client request for point cloud data streaming executed at 514. Further, in some embodiments, the client 502 can signal its viewpoint along with the point cloud streaming request such that the server 504 can, in response, potentially adjust the resolution of the reference point cloud (e.g., when the client 502 already knows the initial viewpoint that will be used without first receiving the full reference point cloud).

[0100] At 522, the server 504 receives a second (e.g., current, e.g., newly captured) point cloud (data) from the sensor 506. At 524, the server 504 accordingly processes the reference point cloud and the second point cloud to (i) if they exist, perform a 3D transformation (e.g., a rigid change approximating the transformation within the second point cloud from the reference point cloud), and / or (ii) determine points that will be removed and / or added to modify the reference point cloud, for example, to substantially reproduce (e.g., replicate) the deviation between the reference point cloud and the second point cloud. When the transformation and / or points are determined, the server 504 may update the reference point cloud stored in the server 504.

[0101] Further, at 526, the server 504 may send to the client 502 a display of an area within the reference point cloud that will be transformed, for example, using the determined transformation (if they can be determined) or points that will be added to and / or removed from the reference point cloud, so that the client 502 can update a local copy of the reference point cloud on the client side. In this way, instead of continuously receiving a new full set of raw updated point cloud data from the server, the client 502 can update to reflect the changes that occurred within the newly captured point cloud.

[0102] Next, an exemplary overall process illustrated in FIG. 4 will be described in more detail according to some embodiments.

[0103] FIG. 5 is a flowchart 600 illustrating an exemplary method of processing point cloud data according to some embodiments. In some embodiments, the exemplary method of FIG. 5 may be performed by a server such as, for example, the server 504 of FIG. 4 or the server 202 of FIG. 2.

[0104] As shown in FIG. 5, in step 602, the server may receive (e.g., continuously) the point cloud data captured by the sensor. In 604, the server waits for the browsing client to request the point cloud data stream. In step 606, the server determines whether a client request has been received. If a client request has been received, in step 610, the server stores the current point cloud as the reference point cloud, and in step 612, the server sends the current point cloud to the client. In step 614, the server starts (or, e.g., continues) a process for streaming the point cloud data to the client. In some embodiments, the process may be executed for each new point cloud instance captured from the sensor and received at the server.

[0105] The process starts (or, e.g., continues) in step 616, where the server receives from the browsing client the current viewpoint that will be used to render a given point cloud with respect to the reference point cloud. In some embodiments, the client viewpoint is used to determine what changes within a given point cloud are important to the client, and one measure of importance is determined, e.g., by the amount of visual impact a given change causes from the client viewpoint perspective.

[0106] In step 618, the server receives the newly captured point cloud. Then, in step 620, the server hierarchically examines the newly captured point cloud in comparison with the reference point cloud to isolate the changed area. As shown in FIG. 5, in some embodiments, step 620 may include a sub-process 622 having several further sub-steps.

[0107] More specifically, in sub-step 624, the server identifies the changed area. In some embodiments, the goal is to identify which areas of the captured point cloud have changed since the most recent update in order to approximate a transformation that best estimates the changed point locations within the cluster corresponding to the changed area. Thus, sub-step 626 involves clustering the areas that deviate between the reference point cloud and the captured point cloud. In some embodiments, the step of clustering includes isolating areas of the most recently captured point cloud that are different from the reference point cloud.

[0108] In sub-step 628, the server can determine (e.g., calculate as in FIG. 5) a respective score for each cluster, where in some embodiments, each score indicates the importance of the visual impact of the deviation. In some embodiments, such importance can be calculated as, for example, a weighted sum of the size of the deviating area, the amount of the deviation, and the distance of the deviating area from the current (virtual) viewpoint that the client is using to inspect the client-side point cloud. Of course, this is just an example, and other formulations may be used.

[0109] In sub-step 630, for the first cluster having the highest score (thus, an example of the highest priority), the server determines the shape correspondence between two point clouds (i.e., the captured point cloud and the reference point cloud) near the area where the deviation is clustered. As a general problem, according to some embodiments, when the area with deviation is first detected, for example, it can generally be assumed that the deviation is caused by a moving segment. In some embodiments, in order to find where the segment has moved, it may be necessary to examine a larger area from the point cloud representing the second time step. If the deviation is caused by a rigid transformation of an element, that element may be relatively close to its original location. Thus, in some embodiments, shape correspondence (e.g., matching) is performed on an area that may be slightly larger than the area of the deviation. In some embodiments, the size of the larger area can be controlled by examining the length of the time between frames and the expected movement speed of the element. Then, in sub-step 632, for example, if a similar shape is found, in sub-step 634, the server can determine (e.g., solve for) a transformation that approximates the deviation between the reference point cloud and the captured point cloud. In sub-step 636, the server then adds (e.g., applies) that transformation to the reference point cloud and, in sub-step 638, sends the display and transformation of the area with deviation to the client.

[0110] In some embodiments, the area of deviation may be sent as the bounding volume coordinates encapsulating that area. In some embodiments, the transformation may be sent as a transformation matrix, for example, a 3×4 transformation matrix. As a general problem, the transformation matrix is a standard way to describe transformations in linear algebra and computer graphics where a single matrix defines a 3D rigid body transformation including rotation and translation. Then, the new point location can be obtained by multiplying the point coordinates (to be transformed) by the transformation matrix. See, for example, Non-Patent Document 5.

[0111] If no similar shape is found in sub-step 632, sub-process 622 moves to sub-step 640, where the server determines the points that will be removed from and / or added to the reference point cloud according to the captured point cloud, and then, in sub-step 642, streams the added and / or removed points to the client.

[0112] In sub-step 644, in some embodiments, the server may isolate additional finer-grained areas (also referred to herein as "sub-areas" for example) from the just-processed clusters that have not yet been addressed. These unaddressed areas may still have, for example, a significant amount of deviation. In some embodiments, the server may cluster those sub-areas into new deviation clusters and calculate an importance score (or other criteria for example) for each of those new clusters as described above.

[0113] In sub-step 646, in response to the determination of the availability of time, processing, and communication bandwidth resources on the server and / or client side, the server can continue to process and partition the remaining clusters. For example, the server can return to sub-step 628 and continue to process the next unprocessed cluster with the next highest importance score and thus (for example) the next highest priority (from among the combinations of the previously formed clusters and the new clusters corresponding to the just-processed sub-areas). According to this example, however, if the server determines that it has no processing budget remaining, sub-process 622 ends and the processing can return, for example, to step 614.

[0114] On the other hand, for example, if it is determined in step 648 that the server or client has signaled a request to end the streaming, the process can end. Otherwise, the process can return to step 604.

[0115] Implemented in some embodiments, generally, the exemplary methods and systems proposed herein, as illustrated in the examples of FIGS. 4 and 5, may enable efficient streaming of a sequence of dynamic point cloud data. In some embodiments, the amount of data that may need to be transmitted can be optimized by considering, for example, changes within the point cloud data as a continuous hierarchical rigid transformation prioritized according to, for example, the visual impact on a client viewing the data. In some embodiments, this can ensure that available bandwidth and processing power are consumed for content updates that may have the greatest impact on, for example, the quality of the user's experience.

[0116] In contrast to some other techniques for point cloud optimization, the exemplary methods and systems disclosed herein according to some embodiments generally work well in common uncontrolled capture environments, such as ad-hoc capture of motion for indoor or outdoor environments where, for example, most of the geometry is static and the transformations can generally approximate rigid body animations, such as individual moving objects.

[0117] As spatial sensing becomes increasingly embedded in smart environments, automotive systems, and mobile devices, the exemplary techniques proposed herein according to some embodiments may provide an efficient way to stream data between systems and services that share environmental data. In most exemplary cases, environmental data can be expected to include both static environmental geometry and rigid objects moving within that environment. In such cases, in some embodiments, the exemplary compression techniques described herein according to some embodiments may be suitable.

[0118] According to some embodiments, the changes are prioritized according to the user's current perspective with respect to the point cloud data and may be distributed hierarchically such that, for example, each area with a change may have further sub-areas that describe the changes in the data at different levels. In the case of point cloud data, due to sensor noise, each frame may have certain changes. Advantageously, using the techniques proposed herein according to some embodiments, only the most important changes are communicated within the available bandwidth range.

[0119] FIG. 6 illustrates a flowchart of another exemplary method 700 for processing point cloud data according to some embodiments. In some embodiments, the exemplary method of FIG. 6 may be performed by a server, such as, for example, server 504 of FIG. 4 or server 202 of FIG. 2. In some embodiments, the exemplary method of FIG. 6 may be applicable when, for example, instead of inspecting the full point cloud for transient changes or between time steps, the server inspects the area of the point cloud visible to the client (e.g., a snapshot point cloud as in step 720) at a given time instance.

[0120] As shown in FIG. 6, at step 702, the server may receive (e.g., continuously) 3D data representing the user environment. The 3D data may be collected by sensors, an example of which includes RGB-D, LIDAR, and / or combinations of these and / or other sensors.

[0121] At 704, the server waits for a client joint request for point cloud data and radiography data (e.g., here), and at step 706, the server determines whether a client request has been received. If a client request has been received, at step 708, the server stores the current point cloud (3D data captured by the sensor) and radiography data as a reference point cloud, and at step 710, the server also sends the current point cloud and radiography data to the client.

[0122] In step 712, the server starts (e.g., continues) a process for streaming point cloud data to the client. In some embodiments, this process can be performed for each new point cloud instance captured from the sensor and received at the server.

[0123] This process starts (or, e.g., continues) in step 714, where the server receives and stores updates to the captured (e.g., sensor-collected) point cloud data (e.g., from the captured image of the changing real-world scene) and, if any, updates to, e.g., radiographic data. Generally, in some embodiments, the point cloud data may be in the form of raw geometry, for example, and the color for that geometry may be derived from a radiographic image, for example, distributed separately. However, in some embodiments, the geometry and associated color may instead be included within the point cloud data.

[0124] In step 716, the server receives the current viewpoint of the client (e.g., within an immersive scene). In response to receiving the current viewpoint of the client, in steps 718 - 720, the server determines the change within that viewpoint relative to the viewpoint associated with the reference point cloud and may receive a snapshot point cloud of the (captured scene) from this new perspective / viewpoint. The example of FIG. 6 can achieve the optimization of the process by limiting, for example, the inspection of the temporal variation of the point cloud to only the area currently being viewed by the client.

[0125] In step 722, the server hierarchically examines the snapshot point cloud or the currently newly captured point cloud against the reference point cloud to isolate the changed area. As shown in FIG. 6, in some embodiments, step 722 may include a sub-process 724 having several further sub-steps.

[0126] More specifically, in sub-step 726, the server identifies the changed area. For example, similar to some aspects of the sub-process illustrated in FIG. 5, the server may cluster the areas that deviate between the reference point cloud and the snapshot point cloud in sub-step 726. Also, the server can determine (e.g., calculate) the respective scores of each cluster, where in some embodiments, each score indicates the importance of the visual impact of the deviation. In some embodiments, such importance can be calculated as, for example, a weighted sum of the size of the deviated area, the amount of deviation, and the distance of the deviated area from the current viewpoint of the client.

[0127] In sub-step 728, for the first cluster having the highest score, and thus, for example, an example of the highest priority, the server determines the shape correspondence between the two point clouds (i.e., the snapshot point cloud and the reference point cloud) in which the deviation is clustered. Then, if a similar shape is found (e.g., detected), in sub-step 730, the server can determine (e.g., solve) a transformation that approximates the deviation between the reference point cloud and the snapshot point cloud. In sub-step 734, the server then adds (e.g., applies) the transformation to the reference point cloud and, in sub-step 736, sends the display of the area with the deviation and the transformation to the client that is viewing it. These sub-steps are described in more detail in the description of FIG. 5.

[0128] If no similar shape is found in sub-step 730, the sub-process moves to sub-step 738, where the server determines the points that will be removed from and / or added to the reference point cloud according to the snapshot point cloud, and in sub-step 740, streams the added and / or removed points to the client, rather than, for example, the entire set of point cloud data.

[0129] In sub-step 742, in some embodiments, the server may isolate additional finer-grained areas (also referred to herein as, for example, "sub-areas") that have not yet been addressed from the clusters that have just been processed. In some embodiments, the server may cluster those sub-areas into new deviation clusters and calculate an importance score (e.g., or other criteria) for each of those new clusters, as described with respect to FIG. 5.

[0130] In sub-step 744, if the processing budget (e.g., the processing time reserved to process one instance of the captured point cloud, the bandwidth available to send the processing capacity to be converted or estimated on the client and / or server side) is still made available, the server can continue with the processing and partitioning of the remaining clusters. For example, the server may, for example, return to sub-step 728 and continue processing the next unprocessed cluster with the next highest importance score, and thus, for example, the next highest priority, (e.g., from a combination of previously formed clusters and new clusters corresponding to the sub-areas that have just been processed). However, if the server determines that it has no processing budget remaining, sub-process 724 may end and the process may return to step 712.

[0131] On the other hand, in step 746, for example, if it is determined that the server or client is signaling a request to end the streaming, the process may end. Otherwise, the process may return to step 704.

[0132] Figures 7A and 7B are flowcharts of another exemplary method of processing point cloud data, according to some embodiments. FIG. 7B is a continuation of the method steps shown in FIG. 7A. Together, FIGS. 7A and 7B illustrate an exemplary process that may be performed by a server (for example) that may be responsible for capturing a point cloud obtained from the collection of sensor data from sensors available for environmental sensing. The sensor may be any of, for example, an RGB-D sensor, an array of cameras, a stereo camera that creates depth information, a LIDAR, etc. At the start of the process, at step 802, the server initializes the sensor to be in a state of creating raw sensor data, and at step 804, waits for the client to request (for example, an optimized) point cloud stream.

[0133] In some embodiments, when the client requests to receive (for example, an optimized) dynamic point cloud stream from the server, at step 806, first, the server and the client may negotiate an appropriate processing budget. In some embodiments, the processing budget defines how much time and bandwidth can be used to stream data from the server to the client for each single dynamic point cloud update step. In some embodiments, the bandwidth of the network connection between the client and the server may be tested to negotiate the budget that will be used. In this regard, in some embodiments, the client may define (for example, the required) update frame rate that defines, for example, how much time the server has for processing each update step. Also, in some embodiments, the performance of the client may be tested to define approximately how many transformations the client can handle between each update step. In some embodiments, the client may also estimate how many point cloud points the client can process and render during each display update cycle.

[0134] When a budget for processing is defined, at step 808, the server may capture the latest point cloud data using raw sensor data and, at step 810, store that latest point cloud data as a reference point cloud. At step 812, the server may also send the reference point cloud to the client. In some embodiments, when storing the reference point cloud, the server may reduce the reference point cloud complexity (e.g., using a suitable compression technique), for example, if the data size is too large for the server or client to handle in real time. Note that the first reference frame may not depend on the client's perspective.

[0135] In addition to sending a full reference point cloud at the start of a session, in some embodiments, it is possible to send a new reference point cloud periodically, for example, to avoid data degradation caused by, for example, cumulatively approximating changes within the point cloud as a hierarchical transformation. The new reference point cloud may be sent, for example, using a fixed interval, such that a new full reference point cloud is sent every Nth update step. In some embodiments, other methods may be used to determine to ensure that the magnitude of the cumulative change warrants sending a new reference point cloud. Such a method may, for example, check the overall change between the currently captured scene and the previously sent scene as the reference point cloud and, when the change exceeds a threshold, be able to send a new reference point cloud.

[0136] When the reference point cloud is being sent to the client, this process proceeds to a continuous sub-process that is executed for each time step of the point cloud streaming. In the continuous sub-process, at step 814, the server first receives or predicts the latest client location with respect to the reference point cloud. Next, at step 816, the server captures the raw sensor data into the latest point cloud, and that latest point cloud is then processed such that the changes that have occurred since the last capture are transformed as transformations prioritized for the client.

[0137] Since point cloud data is a format for describing virtual content in a full 3D format, different viewpoints may be selected to inspect the scene data. Thus, in some embodiments, when optimizing the content to more precisely conform to the quality requirements for a particular client, the viewpoints used by the client are also taken into account. In various embodiments herein, as the magnitude of the visual impact of certain changes is prioritized such that changes within the dynamic point cloud are recognized by the viewer, the viewpoint with respect to the content used by the viewer affects the weighting of the visual impact of that particular change. In some embodiments, changes closer to the viewer are generally more visible than similar changes to objects further away from the user (or viewer), and thus may be prioritized accordingly. Accordingly, in the various embodiments described herein, it is valuable for the client to continuously report back to the server the viewpoints used to inspect the content.

[0138] Streaming of changes in the captured point cloud that have occurred since the previously processed time step begins in step 818 by first isolating areas of the point cloud where there are deviations from the previously captured point cloud. The newly captured point cloud (and thus the current point cloud to be processed) is compared to a reference point cloud. In some embodiments, for example, all areas of captured points that characterize an overall geometry that is significantly deviated are isolated into separate clusters.

[0139] In some embodiments, a cluster is, for example, an isolated (e.g.,) contiguous area of a point cloud that is different from a reference point cloud. In some embodiments, a contiguous area may refer to an area consisting of a collection (e.g., a group) of points that are relatively close to each other (e.g., located at positions less than a given threshold distance from each other). As will be described in more detail, for example, when an original cluster area is processed, the cluster can be inspected again (e.g., in subsequent processing) to identify any finer-grained areas or sub-areas that are still deviated. In some embodiments, the identification of the cluster then enables the identification of which areas of the point cloud have changed since the last update step, since it is possible to approximate a transformation that best estimates the changed point locations within that cluster. In some embodiments, clustering can be achieved by first subtracting a newly captured point cloud from a reference point cloud so that corresponding points within the newly captured point cloud can be marked as not deviated if there are points within an allowable distance, and thus comparing the point locations between the reference point cloud and the captured point cloud with some tolerance of Euclidean distance. According to these examples, purging the points in the captured point cloud that are not deviated results in the deviated areas. These areas can then be clustered into isolated deviated areas.

[0140] Point cloud registration and difference comparison can be achieved by an iterative closest point method, or other similar methods. See, for example, the papers in Non-Patent Document 6; Non-Patent Document 7; and Non-Patent Document 8.

[0141] Next, referring to FIG. 7B, when the deviated areas are separated into clusters (step 818), at step 820, the clusters are prioritized according to, for example, the visual impact of the changes included within a given cluster on the client's field of view. As discussed above, in some embodiments, the client perspective is important because it is used to determine what changes in the point cloud data are important to that client. In some embodiments, the visual impact may be determined such that changes that are relatively large, close to the browsing user, close to the current focus in the browsing direction, and further away from the browsing user and in a direction further towards the rear of the browsing user are more important than smaller changes. In some embodiments, the magnitude of the change may depend on both the area of the point cloud that is changing and the distance of the area from the viewer. Depending on the situation, a change that is far from the viewpoint may have no significant visual impact on the viewer.

[0142] In some embodiments, the distance used when calculating the priority for each cluster (also shown as "importance" in FIG. 7B) can be the distance of that point from a virtual viewpoint (e.g., used by the client to inspect the point cloud on the client side). This distance may be calculated from the current client location provided in step 814. In some embodiments, for a cluster, the distance may be the average distance of each point within the cluster from the virtual viewpoint used by the client. In some embodiments, all points within the point cloud may be defined using a global point cloud coordinate system, and the client viewpoint may use this same coordinate system. For example, if the current viewpoint is located at coordinates (x, y, z) within the point cloud, the viewpoint may be given as a combination of coordinates (x2, y2, z2) and a vector (x3, y3, z3). Depending on the situation, the viewpoint vector may be a unit vector. Further, in some embodiments, additional information such as the field of view cone used by the client may be signaled to the server. For example, referring again to the exemplary process of FIG. 6, only a portion of the point cloud visible to the client may be processed. In some embodiments, in order to enable the server to determine the visible area, the server may need to know, for example, the field of view (e.g., the field of view cone) in addition to the viewpoint and direction.

[0143] In step 822, the cluster with the highest priority (e.g., the calculated highest importance criterion (e.g., score) as in FIG. 7B) is processed. As described above with respect to FIGS. 5 and 6 and described in more detail with respect to FIG. 8, the processing of a given cluster (e.g., the cluster with the highest priority) involves, in some embodiments, determining the 3D transformation to be applied to the reference point cloud or, if such a transformation cannot be determined, determining the points to be added and / or removed from the reference point cloud according to the changes.

[0144] In step 824, the cluster areas within the reference point cloud are updated by the server according to the processing results. Further, as shown in FIG. 7B, in step 824, information regarding the processing results (e.g., information related to conversion or point removal and / or addition) may also be sent by the server to the client for updating the reference point cloud on the client side. In step 826, the just-processed clusters are re-inspected by the server to identify further areas within the processed clusters that are still deviated (also referred to herein as "sub-areas" or "areas of finer granularity of residual change"). Once identified, new clusters are formed and, as previously explained, the importance of each new cluster is calculated.

[0145] In step 828, the server determines whether there is still sufficient processing budget remaining to continue the processing. If the determination result is affirmative, the process returns to step 822, re-determines the next highest-priority cluster, and the newly formed clusters and their corresponding importance values are added to the previously formed clusters in order to process the processing blocks as described.

[0146] On the other hand, if the determination result is negative, the process moves to step 830, where, for example, the client determines whether the end of the current session is requested. If the determination result is negative, the process returns to step 814. If the determination result is affirmative, the process moves to step 832, where it is determined whether the end of the process is requested. If so, the process ends at 834. If not, the process returns to step 804, where, as shown in FIG. 7A, the server waits for another request from the client for point cloud streaming.

[0147] Exemplary cluster processing according to some embodiments is described in more detail next.

[0148] In some embodiments, in processing each cluster, the goal can be to find a rigid transformation that accurately describes the deviation between the reference point cloud and the captured point cloud. If no transformation is found that describes the deviation, then in some embodiments, points can be added and / or removed from the reference points to make the reference point cloud, for example, similar to the captured point cloud.

[0149] FIG. 8 is a flowchart of an exemplary method of processing a cluster according to some embodiments. In some embodiments, the processing can be performed by a server.

[0150] Processing of the clusters can involve assigning each cluster a respective priority such that the cluster with the highest priority is selected for initial processing.

[0151] Accordingly, at step 902, the server can select the cluster with the highest priority for processing. As described above, in some embodiments, the determination of the priority of a given cluster can be based on a determination (e.g., calculation) of the importance of the cluster (e.g., significance, or a score indicating, for example, priority), where the importance indicates the visual impact of the changes included within the cluster with respect to the current viewpoint of the viewer. After the cluster to be processed is selected, at step 904, the server compares the reference point cloud and the newly captured point cloud to find a shape correspondence of the cluster area between the two point clouds. In some embodiments, the goal can be to check whether the same overall geometric shape, which can mean that the cluster area has been transformed between two time points, exists within both point clouds. Depending on whether a shape correspondence is found (at step 906), in some embodiments, the process branches either to (at step 908) solve the transformation or to (e.g., directly) move to the process of point addition and / or removal (at step 912).

[0152] It will be appreciated that there are a number of exemplary techniques for how shape correspondence between two point clouds can be performed. This process can involve, for example, the process of registering two point cloud datasets. The point cloud registration process generally involves estimating a correspondence based on similarities between features and positions from which a transformation can be estimated (such as in the form of a rigid transformation matrix that can align it to another dataset when applied to one of the datasets). See, for example, the paper in Non-Patent Document 9.

[0153] As one exemplary example, in some embodiments, the server may be configured to perform a fairly rough shape estimation between deviation clusters that are relatively close to each other within a reference point cloud and a newly captured point cloud. As a result, if a match appears to exist, the server may then run an Iterative Closest Point (ICP) algorithm to find a transformation (such as an optimal one). In some embodiments, if necessary, the server may perform additional verification if the error between the registered clusters is less than a given threshold that ensures, for example, the existence of a correspondence.

[0154] By using the advantages of the examples described herein, as described herein according to some embodiments, hierarchical point cloud data processing or cluster inspection corrects residual errors in the data, so it should be noted that the accuracy of shape correspondence may not need to be relatively high. Thus, in some embodiments, even clusters that do not match well enough may be sufficient to run ICP on the clusters because subsequent sub-area cluster processing (described above and described in more detail below) can correct the residual error.

[0155] Furthermore, in some embodiments, finding a shape correspondence (e.g., a match) may involve creating a bounding volume for the cluster area and matching it to a bounding volume of approximately the same size / shape. In this case, if higher accuracy of the match is required, the bounding volume is generated using more complex (e.g., more complex than a mere box) geometry.

[0156] In step 906, if a similar geometric shape detected as an area deviated from one point cloud is detected from the second group of points, the server may determine that the deviation is caused by the transformation of the deviated area. In this case, in some embodiments, in step 908, a transformation that can best describe the transformation is solved. Generally, according to some embodiments, a linear system B = R * A + t is formed to consider and solve for a rigid transformation (e.g., translation and rotation). From the linear system, R and t are solved by minimizing the least squares error calculated from the deviation of points between two corresponding point cloud areas. Multiple different analytical or iterative linear solvers may be used to solve for R and t that give the least squares error.

[0157] The transformation may include known methods (e.g., transformation matrices) used in linear algebra and computer graphics. The transformation matrix can combine rotation and translation into a single matrix. By multiplying the point coordinates by the transformation matrix, points can be transformed. The 3D transformation matrix generally has a shape of 3×4. In the case of a rigid transformation, the first 3×3 part can be interpreted as describing rotation, and the latter 3×1 part can describe translation. See, for example, the paper in Non-Patent Document 5.

[0158] In step 908, a transformation (e.g., a rigid transformation) approximating the deviation between two point clouds is solved. In some embodiments, the solved transformation may be, for example, optimally, a reference point cloud cluster for the captured point cluster. In step 910, the server may modify the reference point cloud by applying the transformation to the area in the reference point cloud that is deviated (hereinafter sometimes referred to as the "changed area" herein). In step 910, correspondingly, the display of the transformation and the deviated area may also be sent by the server to the client so that the client can update a local copy of the reference point cloud on the client side. In some embodiments, the display may be in the form of bounding volume coordinates indicating the changed area. In this way, the client can locally track the server's updates to the reference point cloud.

[0159] (In step 906), if no shape correspondence is found, in step 912, for example, to define point addition (e.g., points that will be added) to a reference point cloud that replicates the appearance of the deviated area within the captured point cloud and / or point removal (e.g., points that will be removed) from the reference point cloud, the captured point cloud areas of the reference point cloud and the clusters can be inspected by the server. For illustration, if there are holes in the geometry within the reference point cloud where the data regarding it is visible at that time, a portion of new data that fills the holes within the captured point cloud is added to the reference point cloud by adding points. Alternatively, if points that appear in the reference point cloud no longer exist within the captured point cloud, those points may be removed from the reference point cloud. Such point removal may be caused, for example, by the object moving outside the area where it was captured. In step 912, as points are added and / or removed from the reference point cloud by the server, in some embodiments, as shown in FIG. 8, point information is also sent (e.g., streamed) to the client so that the client can perform the same point addition / removal operations on its local copy of the reference point cloud of the client. In some embodiments, the added points can be downsampled from the full-resolution data that exists within the captured point cloud. This can occur, for example, when the resolution of the captured point cloud is considerably higher than the resolution used within the reference point cloud.

[0160] When the transformation is being performed on the reference point cloud (step 910), or when points are being added to and / or removed from the reference point cloud (step 912), in step 914, the server may re-examine the cluster area (sometimes herein also referred to as the "main area of change" or "main cluster area") to cluster possible sub-areas within the just-processed cluster that may still deviate or indicate a residual change (sometimes herein also referred to as the "area of finer granularity of residual change" or "area of finer granularity"). In some embodiments, to identify one or more sub-areas within the processed cluster having the highest priority, the server compares the captured point cloud with the updated reference point cloud to identify any areas of finer granularity that may still deviate from the updated reference point cloud.

[0161] As previously explained, the updated reference point cloud may be the original reference point cloud, where either a transformation has been applied or it has been modified by point addition and / or removal. In some embodiments, new clusters may be defined for the identified sub-areas, and in step 916, the new clusters may be prioritized (e.g., each priority may be assigned) by determining (e.g., calculating) the respective importance for each new cluster. Such calculations may be performed in a similar manner as described above for the clusters corresponding to the main cluster area. Then, in step 918, if the server determines that processing budget still remains, the server may add these new clusters to the list of clusters to be processed. Thus, the list of clusters will then include the previously prioritized main cluster areas (based on their importance) and the new clusters with their respective importance values or priorities. In some embodiments, the processing budget includes the time available for the server to calculate the transformations and / or point additions / removals required between each point cloud update cycle. If the processing budget is no longer available, then in step 920, the server may return to the main processing as shown in FIGS. 7A and 7B (e.g., see FIG. 7B and its corresponding description (e.g., steps 830 and 832)).

[0162] If there is still processing budget remaining, the cluster with the highest priority is processed. When a new cluster is isolated for a sub - area within a cluster that still has deviations, the process returns to step 902, where the server selects the next cluster with the next highest priority that is to be processed from the list of prioritized clusters. In this regard, in some embodiments, the next cluster with the next highest priority can be one of the remaining clusters previously identified for the main area of change, or a cluster corresponding to one of the sub - areas (e.g., a sub - area within the cluster that was just processed with the previous highest priority). In some embodiments, the process shown in FIG. 8 prioritizes the clusters with the highest visual impact continuously from among the identified clusters to ensure that as many visible changes as possible are reproduced between point - cloud update cycles within the reference point - cloud and sent to the client.

[0163] In some embodiments, the update process can be continuously executed on the client - side or server - side until the user requests that the process be terminated (e.g., as shown in the exemplary processes described in FIGS. 5 - 8). Then, for example, in response to the request, the server - side process can either return to waiting for client - connection requests or completely terminate the process.

[0164] Exemplary variations according to some embodiments are then described.

[0165] In some embodiments, the server can handle connections with multiple clients simultaneously. In this case, the server can, for example, continuously listen for client connection requests, and whenever a client signals an intention to connect to the server, for example, the server can send the current reference point cloud to the clients that are connected. Thus, according to some embodiments, the various methods described herein can be executed for each of the clients actively connected to the server. In some embodiments, the server can, for example, receive a given location from each of the connected clients in order to consider the location when determining (e.g., calculating) respective importance values (e.g., scores) for the clusters to be processed for a particular client. As described above, in some embodiments, the client location can be used, for example, to determine the distance of an area within a given point cloud with respect to the client's perspective. In this regard, in some embodiments, as previously explained, the importance calculation (e.g., regarding cluster priorities) is at least partially based on the distance of the area of change from the current perspective of the client.

[0166] In some embodiments, the various embodiments of the disclosed methods can be executed by the server based on a pre-captured dynamic point cloud sequence. That is, according to some embodiments, in this particular variation, the point cloud data may be stored on or retrieved from a storage medium rather than a sensor. Further, in some embodiments, the data can be converted into a (e.g., optimized) stream of prioritized transformations (e.g., using the various techniques described herein according to some embodiments) as a preprocessing step before any client requests the data.

[0167] In some embodiments, the server may additionally inspect the point cloud data to estimate an area correspondence having a predefined object model. The server may then show the viewer some natural marker that the client can use for context adjustment of the viewing application. In this regard, in some embodiments, as the point cloud (data) is processed, for example, a separate shape classification may be performed on the point cloud to detect whether a known object type can be identified from the point cloud.

[0168] In some embodiments, the server may analyze the point cloud and provide the client with context information along with the reference point cloud. For example, the server may analyze the point cloud to estimate the floor plane within the point cloud and provide this information to the client. The client may then use the floor plane information to align the point cloud display such that the floor plane characterized within the point cloud data coincides with the floor plane of the viewer's physical environment. In some embodiments, this is an example of how object recognition can be used. More specifically, a separate object recognition may be performed by the server on the point cloud, and if something “contextually” important, for example, in this case, the floor plane, is detected, the server may send that information to the client.

[0169] In some embodiments, the contextual understanding of point cloud data can be used by the server to filter out portions of the content. In this regard, in some embodiments, instead of simply signaling to the client the contextually meaningful information (e.g., as described above), the server can use that information to process the point cloud data on the server side. In some embodiments, the filtering can be triggered by a client request or some (e.g., server-side) predefined setting. For example, the client can request the server to stream only updates for dynamic elements within the scene or to filter out the floor and ceiling of the captured content. Thus, in some embodiments, the server can stream only updates for elements that are (e.g., at least partially) dynamic (and thus likely to change over time) rather than elements that are static.

[0170] Exemplary operations according to some embodiments are described next. In one possible exemplary operation, the camera array can target walls and doors. From the camera array data, a 3D point cloud can then be created that can be distributed to the client. On the client side, the 3D point cloud can be viewed using MR glasses in a way that enables motion parallax. Using the disclosed methods and systems according to some embodiments, the server can create a point cloud from the camera array data. When the client connects to the server, the server can start streaming to the client a sequence of dynamic point clouds, starting from a first point cloud that can be set as a reference point cloud. Local device tracking can be executed on the client device, and when the client receives the reference point cloud and selects an initial viewpoint, the client can start sending viewpoint locations relative to the reference point cloud. In some embodiments, the client can select an initial viewpoint for the point cloud data based on local user preferences.

[0171] When the client first connects to the server and requests a (e.g., optimized) dynamic point cloud stream, the server sends the full point cloud to be used as the reference point cloud. After the reference point cloud is sent to the client, in some embodiments, only updates to this reference point cloud are sent (e.g., over a certain time period).

[0172] If the scene does not change at all over a period of time, in some embodiments, the main data exchange involves the client sending its location with respect to the point cloud to the server. During this time, the server continuously estimates the changes occurring in the dynamic point cloud data using the updated client viewpoint location when prioritizing the changes that occur in order to send the changes with the greatest visual impact to the client. In some embodiments, when capturing a static scene where elements are not moving or changing, the sensors used to capture the point cloud may have measurement noise or inconsistent measurement densities, which may cause the captured point cloud sequence to exhibit dynamic behavior even if it is recorded from a static scene. In some embodiments, the server continuously detects changes in the point cloud data, prioritizes the changes using the latest client location, and streams the changes to the client as transforms based on the priorities associated with those changes.

[0173] Furthermore, in this exemplary operation (capturing walls and doors), the door may start to open. As the door starts to open, the server may capture the point cloud data from the time steps. This process may isolate the door as a cluster of deviation areas and find a transform that approximates the movement of the area of the point cloud that includes the door for each time step. When the sensor captures an image of a person opening the door, in some embodiments, this process may also start sending these new points that did not previously exist in the reference point cloud (capturing that person).

[0174] When the person opening the door is not fully visible and new points within the point cloud (of the captured person) are sent to the client, then, in some embodiments, the movement of that person can be approximated as a rigid body transformation at a coarse level, and between each frame of the capture, these rigid body transformations can be further estimated from a coarse level to a more detailed level such that the deformation of the appearance of the body for each captured frame is continuously approximated (e.g., as described above).

[0175] Figure 9 is a flowchart illustrating an exemplary method 1000 according to some embodiments. In some embodiments, this method is executed by a server. At step 1002, the server transmits a first point cloud to the client, where the first point cloud corresponds to a reference point cloud. At step 1004, the server receives a second point cloud. At step 1006, the server hierarchically determines the changes within the second point cloud from the reference point cloud. To hierarchically determine the changes, at step 1008, the server identifies a first area within the second point cloud that has changed from the reference point cloud. To hierarchically determine the changes, at step 1010, the server further prioritizes the first area. To hierarchically determine the changes, at step 1012, for the first area having the highest priority, the server determines whether there is a first rigid 3D transformation that approximates the first change from the reference point cloud. At step 1014, in response to the determination that there is a first rigid 3D transformation, the server determines a first rigid 3D transformation that approximates the first change from the reference point cloud. Finally, at step 1016, in response to the determination that there is no first rigid 3D transformation, the server further determines a first point that will be used to modify the reference point cloud, where the first point represents the first change.

[0176] Figures 10A and 10B are respective flowcharts illustrating another exemplary method 1100 according to some embodiments. Note that Figure 10B is a continuation of the method steps shown in Figure 10A. In some embodiments, this method is executed by a server. At step 1102, the server transmits an initial point cloud to the client. At step 1104, the server hierarchically determines changes within the current point cloud from the initial point cloud. To hierarchically determine the changes, at step 1106, the server identifies a main area of change within the current point cloud. To hierarchically determine the changes, at step 1108, the server further uses the first main area of change to determine whether there is a first rigid 3D transformation that approximates the first change from the initial point cloud. At step 1110, in response to the determination that there is a first rigid 3D transformation, the server determines a first rigid 3D transformation that approximates the first transformation from the initial point cloud. At step 1112, in response to the determination that there is no first rigid 3D transformation, the server determines a first point that will be used to modify the initial point cloud, and the first point represents the first change. At step 1114, to hierarchically determine the changes, the server further identifies one or more finer-grained areas of residual change within the main area of change. To hierarchically determine the changes, at step 1116, the server further uses one or more finer-grained areas of residual change and the remaining main area of change to determine whether there is a second rigid 3D transformation that approximates the second change from the initial point cloud. At step 1118, in response to the determination that there is a second rigid 3D transformation, the server determines a second rigid 3D transformation that approximates the second change from the initial point cloud. Finally, at step 1120, in response to the determination that there is no second rigid 3D transformation, the server further determines a second point that will be used to further modify the initial point cloud, and the second point represents the second change.

[0177] Furthermore, various (e.g., related) embodiments have been described above.

[0178] According to some embodiments, a method for representing a time series of point cloud frames as a set of hierarchical 3D transformations applied to a reference point cloud is disclosed. In some embodiments, a series of rigid transformations may be used to modify the reference point cloud. In some embodiments, additional signaling may add or remove points from the reference point cloud.

[0179] According to some embodiments, a method for encoder-side selection of transformations for content representation is disclosed.

[0180] According to some embodiments, optimization of pre-captured content for distribution to clients rather than live encoding is disclosed.

[0181] According to some embodiments, additional context analysis data, such as, for example, the location of recognized elements, may be signaled in addition to point clouds and transformations.

[0182] According to some embodiments, a method of providing 3D data to a client may include receiving 3D data as a first point cloud, sending the first point cloud to the client and storing the first point cloud as a reference point cloud in response to receiving a request from the client, hierarchically receiving 3D data as a current point cloud, receiving a current viewing perspective from the client, determining a change within the perspective of the current point cloud relative to the perspective of the reference point cloud in response to receiving the current viewing perspective, performing a hierarchical inspection of the current point cloud relative to the reference point cloud to identify areas of deviation, clustering the areas of deviation, calculating a score for the clustered areas of deviation, determining a shape correspondence for the clustered areas of deviation, applying a transformation to the reference point cloud in response to determining a similar shape for the clustered areas of deviation, sending the transformed and clustered areas for display to the client, adding points to or removing points from the reference point cloud in response to not determining a similar shape for the clustered areas of deviation, and sending the added or removed points to the client.

[0183] According to some embodiments, another way to provide 3D data to a client is to receive the 3D data as a first point cloud, and in response to receiving a request from the client, send the first point cloud to the client and store the first point cloud as a reference point cloud, and start a process for streaming point cloud data to the client, the process being repeatedly executed for new point clouds until a request to end is received, the process including receiving the 3D data as a current point cloud, performing a hierarchical inspection of the current point cloud against the reference point cloud to identify areas of deviation, clustering the areas of deviation, calculating a score for the clustered areas of deviation, determining a transformation that approximates the shape of the clustered areas of deviation, applying the transformation to the reference point cloud in response to the determination of the transformation, sending a display of the transformed and clustered areas to the client, and in response to not determining a transformation, adding points to or removing points from the reference point cloud, and may include starting the process, and sending the added or removed points to the client.

[0184] In some embodiments, the step of receiving 3D data may include receiving the data from a sensor or receiving a pre-captured dynamic point cloud sequence from a storage medium. In some embodiments, the sensor data may be RGB-D captured data, LIDAR captured data, or some other type. The step of sending the first point cloud to the client may, in some embodiments, include sending radiographic image data to the client. In some embodiments, the score may be calculated based on a weighted sum of the size of the area, the amount of deviation, and the distance of the deviation area from the current viewpoint. Additionally, in some embodiments, each of these methods may further include determining whether time remains within the processing budget for processing additional clusters, and if time remains, processing the additional clusters. In some embodiments, the method may further include negotiating a processing budget between the server and the client for setting time and bandwidth limits for point cloud updates.

[0185] The system can execute a method using a processor and a non-transitory computer-readable medium storing instructions that, when executed by the processor, operate to execute any of the disclosed methods. In some embodiments, the system may also include a 3D sensor.

[0186] One or more of the various hardware elements of the described embodiments may be referred to as "modules" that carry out (i.e., perform, execute, etc.) the various functions described herein with respect to each module. As used herein, a module may include hardware that would be considered suitable by one of ordinary skill in the relevant art for a given implementation (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application specific integrated circuits (ASICs), one or more field programmable gate arrays (FPGAs), one or more memory devices). Each described module may also include executable instructions to carry out one or more functions described as being performed by that module, and those instructions may be in the form of, or may include, hardware (i.e., hardwired) instructions, firmware instructions, software instructions, etc., and may generally be stored in one or more any suitable non-transitory computer-readable media such as those generally referred to as RAM, ROM, etc.

[0187] Although features and elements are described above in certain combinations, one of ordinary skill in the art will appreciate that each feature or element may be used alone or in any combination with other features and elements. Additionally, the methods described herein may be implemented in a computer program, software, or firmware incorporated within a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROMs and digital versatile disks (DVDs). Processors associated with software may be used to implement a radio frequency transceiver for use within a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

1. A method performed by a client, comprising: obtaining a reference point cloud representing a first point cloud in a time sequence of point clouds; receiving, from a server, a series of representations of changes to the reference point cloud, each series identifying an area of ​​the reference point cloud and a change to be applied to the identified area, each such change including at least one of a 3D transformation, a set of points to be added to the reference point cloud, and a set of points to be removed from the reference point cloud; applying the representations in sequence to the reference point cloud to generate a point cloud representing a second point cloud in the time sequence of the point clouds; A method comprising:

2. The method of claim 1 , wherein an area identified by a representation in the sequence of representations includes a bounding volume that encapsulates the area.

3. The method of claim 1 , wherein a first representation and a second representation of the sequence of representations apply changes in sequence to overlapping respective first identified areas and second identified areas.

4. 2. The method of claim 1 , wherein a first representation and a second representation of the sequence of representations capture first and second changes, respectively, to an area of ​​the control point cloud, the first changes comprising a coarse deviation and the second changes comprising a fine residual change.

5. The method of claim 1 , wherein the 3D transformation of a representation in the sequence of representations comprises a rotation and a translation of points within the area identified by the representation.

6. The method of claim 1 , wherein the sequence of representations comprises respective hierarchical 3D transformations.

7. The method of claim 1 , wherein the sequence of representations approximates changes from the first cloud of points to the second cloud of points over time.

8. The method of claim 1 , further comprising sending to the server information about an available processing budget for generating the point cloud representing the second point cloud.

9. 10. The method of claim 1, further comprising sending information about a view location relative to the second point cloud to the server to facilitate prioritization of transmission of the sequence of representations to the client.

10. An apparatus employed by a client, comprising: At least one processor; a memory storing instructions that, when executed by the at least one processor, cause the device to: obtaining a reference point cloud representing a first point cloud in a time sequence of point clouds; receiving, from a server, a series of representations of changes to the reference point cloud, each series identifying an area of ​​the reference point cloud and a change to be applied to the identified area, each such change including at least one of a 3D transformation, a set of points to be added to the reference point cloud, and a set of points to be removed from the reference point cloud; applying the representations in sequence to the reference point cloud to generate a point cloud representing a second point cloud in the time sequence of the point clouds; memory and An apparatus comprising:

11. The apparatus of claim 10 , wherein an area identified by a representation in the sequence of representations includes a bounding volume that encapsulates the area.

12. 11. The apparatus of claim 10, wherein a first representation and a second representation of the sequence of representations apply changes in sequence to overlapping respective first identified areas and second identified areas.

13. 11. The apparatus of claim 10, wherein a first representation and a second representation of the sequence of representations capture first and second changes, respectively, to an area of ​​the control point cloud, the first changes comprising a coarse deviation and the second changes comprising a fine residual change.

14. The apparatus of claim 10 , wherein the 3D transformation of a representation in the sequence of representations comprises a rotation and a translation of points within the area identified by the representation.

15. The apparatus of claim 10 , wherein the sequence of representations comprises respective hierarchical 3D transformations.

16. The apparatus of claim 10 , wherein the sequence of representations approximates changes from the first cloud of points to the second cloud of points over time.

17. The instructions cause the device to: sending to the server information about an available processing budget for generating the point cloud representative of the second point cloud; The apparatus of claim 10 , further comprising:

18. The apparatus of claim 17 , wherein the information about the processing budget includes at least one of an amount of time and an amount of bandwidth.

19. The instructions cause the device to: sending information about a view location relative to the second point cloud to the server to facilitate prioritizing transmission of the sequence of representations to the client; The apparatus of claim 10 , further comprising:

20. A non-transitory computer-readable medium comprising instructions executable by at least one processor for execution by a client of a method, the method comprising: obtaining a reference point cloud representing a first point cloud in a time sequence of point clouds; receiving, from a server, a series of representations of changes to the reference point cloud, each series identifying an area of ​​the reference point cloud and a change to be applied to the identified area, each such change including at least one of a 3D transformation, a set of points to be added to the reference point cloud, and a set of points to be removed from the reference point cloud; applying the representations in sequence to the reference point cloud to generate a point cloud representing a second point cloud in the time sequence of the point clouds; 2. A non-transitory computer readable medium comprising:

Images