System and method for generating a three-dimensional map of an indoor space

Abstract

Three-dimensional (3D) maps may be generated for different regions based on scanning the regions using sensors of the mobile computing device. During each scan, ultra-wideband communications (UWB) may be used to measure the location of the mobile computing device relative to a fixed-position smart device. The 3D map of the region may be registered to a fixed location (i.e., anchor location) of the smart device based on location measurements obtained during the scan, so that the 3D maps can be combined into a combined 3D map. The combined (i.e., combined) 3D map may then be used to facilitate location-specific operation of the mobile computing device or other smart devices.

Description

Technical Field

[0001] Cross - Reference to Related Applications This application is a continuation of U.S. Patent Application No. 17 / 445,751, filed on August 24, 2021, claims priority thereto, and is hereby incorporated by reference in its entirety.

[0002] Disclosure Field The present disclosure relates to an augmented reality system, and more specifically, to a system and method for combining (i.e., fusing) 3 - dimensional (3D) maps generated by multiple scans and using the combined 3D map for a mobile application.

Background Art

[0003] Background With depth perception on a mobile device, a user can scan an indoor space (e.g., a room) to create a 3D model (i.e., a 3D map). Depth perception can utilize multiple sensors including an inertial measurement unit (IMU) for determining direction and a dedicated depth sensor or camera for determining the depth (i.e., range) between the mobile computing device and objects in the room as the mobile computing device is physically moved (i.e., scanned) to sense various areas. A 3D map can be combined with an image to generate a realistic virtual room, and the user can interact with the virtual room in an immersive and realistic experience (e.g., a virtual tour). A 3D model can also be used to generate more realistic virtual objects in an augmented reality (AR) environment. For example, the 3D model can help an AR software place virtual objects such as buildings or houses behind real - world objects in an indoor space. There are multiple requirements for properly scanning an indoor space, and these requirements may be beyond the user's control. If not all requirements are met, the scan may be insufficient, and the corresponding 3D map may be incomplete and / or distorted. This is especially true when scanning a large indoor space with multiple areas.

Summary of the Invention

[0004] Summary In at least one aspect, the present disclosure generally describes a method for generating a 3D map. The method includes performing a first scan of a first area using a depth sensor of a mobile computing device. The method further includes determining the position of the mobile computing device relative to an anchor position during the first scan using an ultra-wideband (UWB) position sensor of the mobile computing device. The method further includes generating a first 3D map of the first area, the first 3D map being referenced to the anchor position.

[0005] In a possible implementation, the method further includes performing a second scan of a second area using a depth sensor of the mobile computing device. The mobile computing device is positioned relative to the anchor position during the second scan using the UWB position sensor of the mobile computing device such that the mobile computing device can generate a second 3D map of the second area. Similar to the first 3D map, the second 3D map is referenced to the anchor position. The method further includes combining the first 3D map and the second 3D map based on the (common) anchor position.

[0006] The first area can be a first room of an indoor space such as a building or a house, and the second area can be a second room of the indoor space. The anchor position can be a position within the indoor space where a smart device (e.g., a smart home device) is fixedly located.

[0007] The smart device can be configured for UWB communication with a UWB position sensor of a mobile computing device. Through UWB communication, the range and / or direction of the mobile computing device relative to the smart device is determined and can be used to identify (i.e., track) the position of the mobile computing device during scanning. A 3D map can be spatially arranged to form a third 3D map that covers both areas. This spatial arrangement is possible because both 3D maps are generated (i.e., reconstructed) based on the anchor positions of the smart device within the indoor space.

[0008] In another aspect, the present disclosure generally describes a method for controlling the operation of a (mobile) smart device based on its position. The method includes generating a combined 3D map. The 3D map includes a first area and a second area scanned by a mobile computing device. The first area and the second area are arranged within the combined 3D map according to their relative positions with respect to the anchor positions determined by UWB communication. The method further includes tagging a plurality of positions within the combined 3D map. The method further includes tracking the smart device based on UWB communication and the combined 3D map, determining that the smart device is at a tagged position within the combined 3D map, and adjusting the operation of the smart device based on the tagged position.

[0009] In a possible implementation, since the boundary of a building or a house can be identified based on the combined 3D map, positions outside the boundary of the building or the house can be tagged. When the smart device is at a position outside the boundary of the building or the house, its access (e.g., network) can be restricted.

[0010] In another possible implementation, since an object can be identified based on an image associated with a combined 3D map, an area around the object can be tagged. When a smart device enters the area around the object, information can be displayed on the smart device. For example, the area around an AeroBike (registered trademark) can trigger the smart device to display information related to training on the AeroBike.

[0011] In another possible implementation, a room can be identified based on a combined 3D map. Further, the room can be tagged according to a room type determined based on an object identified in an image associated with the combined 3D map. When a smart device enters the room, the playback of media can be controlled (based on the type of the room). For example, a bed identified in the image can be used to tag the room as a bedroom, and the playback on a smart device (e.g., a smart speaker) can be made suitable for the bedroom.

[0012] In another aspect, the present disclosure outlines a system for generating a 3D map. The system includes smart devices (e.g., smart home devices) fixedly arranged at anchor positions and configured for UWB communication. The system further includes a mobile computing device (e.g., a mobile phone, a tablet, AR glasses). The mobile computing device includes a depth sensor configured to collect a first set of depth data corresponding to a first region when the depth sensor is scanned over the first region during a first scan, and a second set of depth data corresponding to a second region when the depth sensor is scanned across the second region during a second scan. The mobile computing device further includes a UWB tag configured to determine the range between the mobile computing device and the anchor position during the first scan and the second scan based on UWB communication between the smart device and the UWB tag. The mobile computing device further includes a processor that can be configured by software instructions to execute a 3D mapping method. The 3D mapping method includes registering a first set of depth data at the anchor position based on the range between the mobile computing device and the anchor position during the first scan, and generating a first 3D map for the anchor position based on the registered first set of depth data. The 3D mapping method further includes registering a second set of depth data at the anchor position based on the range between the mobile computing device and the anchor position during the second scan, and generating a second 3D map for the anchor position based on the registered second set of depth data. The 3D mapping method further includes combining the first 3D map and the second 3D map.

[0013] The foregoing exemplary overview, as well as other exemplary objects and / or advantages of the present disclosure, and the manner in which they are achieved, are further described in the following detailed description and its accompanying drawings.

Brief Description of the Drawings

[0014]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

DETAILED DESCRIPTION OF THE INVENTION

[0015] Components in the drawings are not necessarily to scale with each other. Throughout several of the figures, like reference numerals indicate corresponding parts.

[0016] DETAILED DESCRIPTION A mobile computing device (e.g., a mobile phone, a tablet, augmented reality (AR) glasses) can be configured to include a depth sensor (i.e., a depth camera) capable of scanning an area (e.g., a room, a hall, etc.) of an indoor space (e.g., a house, a building, an office, etc.) to create a 3D map of the indoor space (i.e., a 3D model, a 3D scan). The 3D map is a rendering of the indoor space that includes the relative dimensions of objects (e.g., walls, furniture, items) arranged in a spatial relationship that accurately depicts the indoor space. The 3D map can be used directly to obtain an interactive view of the space (e.g., a virtual tour). Alternatively, information can be derived from the 3D map to enhance or enable some application (e.g., computer-aided design, augmented reality, etc.).

[0017] As shown in FIG. 1, acquisition of depth data for a 3D map can be achieved by physically scanning a depth sensor included in mobile computing device 110 along path 120 during a 3D scan of indoor area 100 to correspond to the limited field of view (FOV) of the depth sensor. Mobile computing device 110 can monitor its position / orientation during the 3D scan (e.g., using an inertial measurement unit (IMU) and / or a camera), thereby enabling registration of depth data to the relative position / orientation of the mobile computing device during the scan. Since the depth data is relative to the position / orientation of the mobile computing device during the 3D scan and not relative to a physical coordinate system, it can be difficult to determine the relative position / orientation of sets of depth data from different 3D scans. Thus, it can be difficult to create 3D maps of multiple different areas of an indoor space, and currently, most solutions have technical problems.

[0018] To create 3D maps of multiple different regions by continuously scanning them in a single pass, more processing power and / or memory may be required than is practical, and if the scan is interrupted, the user may have to restart the time-consuming scanning process. To create a combined 3D map of multiple regions by scanning them individually (i.e., in multiple scan sessions) and then combining the 3D maps, overlapping regions in the 3D maps may be required to determine relative positions / directions. Obtaining and registering these overlapping regions can be difficult for several reasons. First, it can be difficult to obtain usable overlapping regions if there are no recognizable landmarks in the overlapping regions and they are not always available. Second, if the user scans the overlapping regions inappropriately, it can be difficult to obtain usable overlapping regions and additional requirements may be imposed on the user. Even if usable overlapping regions are obtained, it may be difficult to automatically register them because processing is required, and it can be time-consuming to manually combine the 3D maps based on the overlapping regions.

[0019] The disclosed systems and methods provide a technical solution for creating a 3D map of multiple regions of an indoor space by obtaining location data (i.e., position data) during each 3D scan, such that each 3D scan is registered (or registrable) to a physical location (i.e., an anchor position) in the indoor space, and then the 3D maps are combined based on a common anchor position (i.e., an anchor point). The disclosed anchor positions can be the positions of smart devices such as smart home devices (e.g., smart speakers, hub devices, smart thermostats, etc.), which are typically not moved or are permanently installed (i.e., fixedly positioned). The location data can be obtained by location tracking based on ultra-wideband (UWB) communication between a mobile computing device that performs the scan and a smart device. The present disclosure further describes possible technical opportunities for collecting information from the combined 3D map and using the combined 3D map in mobile applications including augmented reality.

[0020] Figure 2 is a plan view of an indoor space including a mobile computing device and a smart device configured as an indoor positioning system (IPS). It should be noted that only the mobile computing device and the smart device are shown, but the IPS can be implemented using various configurations of various possible devices for various possible applications. Here, one possible implementation of the positioning / tracking of the mobile computing device 210 during a 3D scan by the smart device 220 fixedly positioned at the anchor position will be described.

[0021] As shown in FIG. 2, the mobile computing device 210 can move within the area (i.e., the bedroom) of the indoor space 200, while the smart device 220 remains in a fixed position. As shown in the figure, the smart device is at the anchor position. The anchor position may be within the indoor space 200 or outside the indoor space 200. For example, a smart device in an adjacent indoor space (not shown) can be used as an anchor point.

[0022] The mobile computing device 210 and the smart device 220 are configured to communicate via the UWB communication link 230. The UWB communication between these devices can be used to identify and track the position (i.e., location) of the mobile computing device 210 relative to the smart device 220. For example, the mobile computing device 210 can identify and track its position relative to the smart device during a 3D scan of the area.

[0023] The mobile computing device 210 and the smart device 220 can each include at least one UWB tag (i.e., a UWB module). The UWB tag can be configured to transmit and receive information via UWB communication. This information can be used to determine the relative position of the mobile computing device 210 and / or the smart device 220, and when the UWB tag is configured to output the relative position, it may be called a UWB position sensor. The relative position can be determined by the UWB position sensor using various approaches.

[0024] A UWB position sensor (i.e., a UWB sensor) can determine the range (r) between a mobile computing device 210 and a smart device 220 based on UWB communication. For example, the range (r) may be based on the round-trip time (RTT) for information (e.g., a handshake) exchanged between the mobile computing device 210 and the smart device 220. In some implementations, the UWB position sensor can determine not only the range (r) but also the angle (θ). This angle can be relative to a fixed coordinate system 225 (x, y, z) located at the anchor position (shown in FIG. 2) or a known position offset from the anchor position. The determination of the angle (θ) can be based on the time difference of arrival (TOA). The UWB position sensor can be composed of a plurality of antennas arranged in an array such that each of the plurality of antennas receives the UWB signal at a different time. By calculating the time difference of arrival, the incident angle of the UWB signal corresponding to the angle (θ) can be determined. These measurements may further include steps to remove multipath signals and can also be extended to other dimensions (i.e., planes). For example, UWB communication can help determine the relative altitude between two devices (not shown).

[0025] In some implementations, the IPS includes a plurality of smart devices (i.e., a plurality of anchor positions). Accordingly, the UWB communication can include smart device identification information (i.e., device ID) so that the mobile computing device 210 can determine the appropriate relative position with respect to each smart device. Using a plurality of smart devices may enable triangulation based on the ranges between the mobile computing device and each smart device. Additionally, using a plurality of smart devices may enable detection of the movement of the smart devices. For example, each smart device can monitor the positions of other smart devices, and the movement of a smart device can trigger the smart device to update / change its position and / or update / change its device ID.

[0026] Figure 3 is a block diagram of a UWB tag according to a possible implementation of the present disclosure. The illustrated UWB tag can be integrated as part of a mobile computing device and / or smart device (i.e., non-integrated implementation) or as a stand-alone device coupled to a mobile computing device and / or smart device (i.e., single implementation). In a single implementation, the UWB tag 300 can include a processor 310. The processor 310 can be configured to execute operations (e.g., ranging, positioning) according to software instructions. The software instructions (i.e., software, code, etc.) are stored in a memory 320 (e.g., non-transitory computer-readable memory) and can be read therefrom. The processor 310 is communicatively coupled to the memory 320 and can be configured to obtain software for execution and read and write information resulting from the execution of the software. For example, data regarding round-trip time, range, angle, position, etc. can be stored in (and obtained from) the memory 320. When the UWB tag 300 is integrated as part of a mobile computing device or smart device (i.e., non-integrated implementation), the processor 310 can be implemented as the central processing unit (CPU) of the mobile computing device or smart device. For example, components configured to provide UWB tag functionality in AR glasses can utilize the central processing unit of the AR glasses.

[0027] The UWB tag 300 can further include a clock 315 implemented in hardware (e.g., logic circuitry) or software (e.g., cycle counter). The clock 315 can control the timing of digital processing and can function as a timestamp useful for calculating the timing of events (e.g., duration, interval). The events can correspond to communications of the UWB tag 300 (e.g., round-trip time) or other events related to the handshake protocol of UWB communication.

[0028] The UWB tag 300 can further include a digital signal processor (DSP330) configured to assist or replace a processor for specific functions. For example, the DSP can be configured to perform aspects related to communication between UWB tags (e.g., packet formation, signal identification, etc.). The processor 310 and / or the DSP 330 can configure a UWB transmitter / receiver (i.e., UWB transceiver 340) to communicate signals on the UWB communication link 350 via the UWB antenna 345. The signals can correspond to a UWB protocol that can include a handshake operation (i.e., handshake protocol). The UWB communication link 350 can function as a communication channel of a UWB network 355 that includes a plurality of UWB tags. In some implementations, processing can be shared by a plurality of UWB tags. In these implementations, the UWB communication link 350 can serve to relay information being processed between the UWB tags.

[0029] The UWB tag 300 can further include a data transceiver 360 (e.g., a Bluetooth (registered trademark) transceiver, a WiFi transceiver, a 5G transceiver, etc.), and the data transceiver can be configured by the processor 310 and / or the DSP 330 to communicate signals on a data communication link 370 via a data antenna 365. The data communication link 370 can function as a communication channel for a data network other than the UWB network. For example, the data communication link 370 can be a Bluetooth communication link configured such that one or more UWB tags within the UWB network 355 can communicate with a mobile computing device via Bluetooth communication. In other words, in addition to one or more of the UWB tags being part of (i.e., communicating with) the UWB network 355, they can also be part of (i.e., communicate with) a data network 375 (e.g., a WiFi network, a CDMA network, a Bluetooth network). This additional data communication link 370 can be considered as a port for another device (e.g., an AR device, a VR device, a mobile phone, a tablet, etc.) to communicate with the UWB tag 300. This port may be useful in implementations where another device is configured to perform part of the processing required for positioning, or in implementations where another device is configured to receive the results of positioning (e.g., in AR applications, VR applications, etc.).

[0030] The UWB tag 300 may further include an inertial measurement unit (IMU 390). The IMU 390 can include one or more accelerometers and magnetometers configured to measure the movement and orientation of the UWB tag 300. In the case of a non-integrated implementation, the IMU 390 can be the IMU of a mobile computing device or a smart device. For example, components that provide UWB tag functionality to AR glasses can utilize the IMU of the AR glasses.

[0031] The UWB tag 300 can further include a power source such as a battery 380 (e.g., a rechargeable battery) for energizing and functioning the components. In the case of non-integrated implementation, the battery may be a battery for a mobile computing device or a smart device. For example, the components providing the UWB tag function in AR glasses can be powered from the battery of the AR glasses instead of a dedicated battery for the UWB tag.

[0032] Returning to FIG. 2, the UWB position data can be stored together with 3D scan data (e.g., depth data) collected by the mobile computing device 210 and used to reconstruct a 3D map registered in the fixed coordinate system 225. The mobile computing device is used to perform multiple scans at various times and can use various smart devices as anchor points, so there can be various ways to identify, store, and combine data to generate 3D maps of multiple regions (e.g., rooms) within the indoor space 200.

[0033] Figure 4 shows a possible 3D scanning scenario for an indoor space. The scanning scenarios shown are described in detail for purposes of explaining aspects of the present disclosure, but it should be understood that obvious variations to this scanning scenario and other possible scanning scenarios are within the scope of the disclosed technology. As shown in FIG. 4, the indoor space 400 includes a first smart device 410 (e.g., a smart home hub) disposed at a first anchor position and a second smart device 420 (e.g., a smart home thermostat) disposed at a second anchor position. The mobile computing device 430 is configured to perform a 3D scan of an area (e.g., a room) within the indoor space 400. The 3D scan may be part of a mapping process in which the user participates, or may be part of another process (e.g., an AR game) in which the user need not participate. The mobile computing device 430 is further configured to determine its position relative to the first smart device 410 and the second smart device 420. Accordingly, the 3D scan can generate a 3D map registered at the first anchor position (i.e., the first smart device 410) and / or the second anchor position (i.e., the second smart device 420).

[0034] 3D maps registered at the same anchor position can be combined to form a combined 3D map that includes more information. For example, the combined 3D map may include (i) a first room of the indoor space scanned during a first 3D scan, and (ii) a second room of the indoor space scanned during a second 3D scan. Combining the registered 3D maps can also generate partial 3D maps at different times and then combine them over time to form a more complete 3D map. For example, the combined 3D map may include (i) a first partial 3D scan of a first room scanned during a first scan, and (ii) a second partial 3D scan of the first room scanned during a second scan. In other words, the combined 3D map can include more areas and / or details than each of the component maps.

[0035] As shown in FIG. 4, the mobile computing device 430 performs a first 3D scan 431 of the first room (i.e., the bedroom 401) of the indoor space 400. During the first 3D scan 431, the mobile computing device 430 collects relative position data based on UWB communication with the first smart device 410 as described above. This position data can be identified as a relative position with respect to the first anchor position by an identifier (i.e., ID) associated with the first smart device 410.

[0036] During the first 3D scan, the mobile computing device 430 can also collect relative position data based on UWB communication with the second smart device 420 as described above. This position data can be identified as a relative position with respect to the second anchor position by an ID associated with the second smart device 420.

[0037] At another time, the mobile computing device 430 performs a second 3D scan 432 of the second room (i.e., the office 402) of the indoor space 400. During the second 3D scan 432, the mobile computing device 430 collects relative position data based on UWB communication with the first smart device 410 as described above. This position data can be identified as a relative position with respect to the first anchor position by an ID associated with the first smart device 410.

[0038] As a result of each 3D scan, 3D scan data can be obtained that includes some combination of IMU data, camera data, and / or depth sensor data. The 3D scan data can be associated with UWB position data (i.e., position data, UWB data) that can include distances and / or directions relative to anchor positions. The position data can be identified by its ID (e.g., device ID) and can indicate an anchor position. Several possible ways to reconstruct and combine the scan / position data to generate a combined 3D map from the exemplary scenario shown in FIG. 4 are shown in FIGS. 5-7 and are described next.

[0039] FIG. 5 is a flowchart showing a first possible method for generating a combined 3D map from the 3D scan scenario shown in FIG. 4. As described above, a first data set 501 is obtained that includes the 3D scan data (i.e., 3D-DATA) of bedroom 401 and the position data (i.e., POS-DATA(ID=1)) for the first smart device 410 as a result of the first 3D scan. The first 3D scan also results in a second data set 502 that includes the 3D scan data (i.e., 3D-DATA) of bedroom 401 and the position data (i.e., POS-DATA(ID=2)) for the second smart device 420. In this example, the 3D scan data of the first data set 501 and the second data set 502 are the same (i.e., the result of the same scan), but in practice they can be different. For example, two 3D scans of bedroom 401 can be obtained, where the first 3D scan of the bedroom is obtained based on the first anchor position and the second 3D scan of the bedroom can be obtained based on the second anchor position.

[0040] As described above, as a result of the 3D scan of Office 402, a third data set 503 including the 3D scan data of Office 402 (i.e., 3D-DATA) and the position data for the first smart device 410 (i.e., POS-DATA(ID=1)) is obtained. The first data set 501, the second data set 502, and the third data set 503 can be stored in the memory 520 for later reconstruction and combination. The memory may be the local memory of the mobile computing device. Alternatively, the memory may be a remote memory (e.g., cloud memory) that the mobile computing device can access via a network. In this implementation, the data from each 3D scan is stored without being reconstructed into a 3D map. In other words, the raw data of each scan is stored without being reconstructed into a 3D map. This approach may have advantages in terms of simplicity because the stored data is in its raw state and can be reconstructed and combined as needed.

[0041] As shown in FIG. 5, this method further includes a reconstruction (i.e., RECON) and combination (i.e., MERGE) process 530 in which the raw data is converted into a combined 3D map. The reconstruction / combination process 530 can include searching for a data set (i.e., the metadata of the data set) that shares a coordinate system (i.e., has position data from a common anchor point). For example, as shown, the reconstruction and combination process 530 operates on the first data set 501 and the third data set 503 because each position data has the same identifier (ID=1). The output of the reconstruction and combination process 530 is a combined 3D map 510 that spatially arranges and orients the bedroom 401 and the office 402 in FIG. 4. The combined 3D map 510 may be in a 3D format (e.g., obj). The combined 3D map may or may not include metadata that identifies the anchor position used for the combination.

[0042] FIG. 6 is a flowchart showing a second possible method for generating a combined 3D map from the 3D scanning scenario shown in FIG. 4. The first data set 501, the second data set 502, and the third data set 503 are each applied to a 3D reconstruction algorithm 610. By reconstructing (i.e., RECON) the first data set 501, a first registered 3D map 601 of the bedroom 401 having data registered to the first anchor point is obtained. Thus, the first registered 3D map 601 has an associated ID (i.e., ID = 1) indicating in which coordinate system (i.e., anchor position) (i.e., the first smart device 410) the 3D map is registered. By reconstructing the second data set 502, a second registered 3D map 602 of the bedroom 401 having data registered to the second anchor point is obtained. Thus, the second registered 3D map 602 has an associated ID (i.e., ID = 2) indicating in which coordinate system (i.e., anchor position) (i.e., the second smart device 420) the 3D map is registered. By reconstructing the third data set 502, a third registered 3D map 603 of the office 402 having data registered to the first anchor point is obtained. Thus, the third registered 3D map 603 has an associated ID (i.e., ID = 1) indicating in which coordinate system (i.e., anchor position) (i.e., the first smart device 410) the 3D map is registered.

[0043] The first registered 3D map 601, the second registered 3D map 602, and the third registered 3D map 603 can be stored in the memory 620 for later use and / or combination. The memory may be the local memory of the mobile computing device. Alternatively, the memory may be a remote memory (e.g., cloud memory) that the mobile computing device can access via a network. In this implementation, available 3D maps are stored and identified for later combination. This approach can have advantages in terms of usability and versatility since the stored and registered 3D maps can be used as individual maps and combined as needed. The reconstruction and storage of each registered 3D map can be performed at different times, so the registered maps of the area can accumulate in the storage device over time.

[0044] This method further includes a combining (i.e., MERGE) process 630 that combines the registered 3D maps to generate a combined D map 640. The combining process 630 can include checking the identifiers of the registered maps, combining the registered 3D maps that have the same identifier, and not combining the registered 3D maps that do not have the same identifier. As shown in FIG. 6, the first registered 3D map 601 and the third registered 3D map 603 are both combined 630 because they both have the same identifier (i.e., ID = 1).

[0045] The data of each registered 3D map can include 3D points and / or a 3D mesh (e.g., wireframe) created from the 3D points. The 3D points can be mapped to a coordinate system defined by the anchor positions. In some implementations, the registered 3D map includes an image of the area. For example, an image of the area can be rendered on the 3D surface representing the area.

[0046] Figure 7 is a flowchart showing a third possible method for generating a combined 3D map from the 3D scanning scenario shown in Figure 4. The first registered 3D map 601, the second registered 3D map 602, and the third registered 3D map 603 stored in the memory 620 can each include an image of the respective area. Thus, this method can include an identification process 710 for analyzing the 3D map image for additional information about the area. This additional information can include characteristics of the area (e.g., size, shape, color, composition, etc.) and / or objects within the area (e.g., electrical appliances, furniture, equipment, etc.). In the identification process, this information can be recognized using an image recognition algorithm. This method can further include associating the recognized information with tags that describe the area. For example, by image analysis of the first registered 3D map 601, the bed in the bedroom 401 can be identified. The image analysis can be performed automatically, thereby advantageously eliminating the user's responsibility of accurately tagging the area during 3D mapping. Thus, the registered map of the first registered 3D map 601 can be tagged as a bedroom. After tagging, the first registered / tagged 3D map 701 is tagged as a bedroom (i.e., TAG = BEDROOM), the second registered / tagged 3D map 702 is tagged as a bedroom (i.e., TAG = BEDROOM), and the third registered / tagged 3D map 703 is tagged as an office (i.e., TAG = OFFICE). The registered and tagged 3D maps can be stored in a memory 730 that is local or remote (e.g., cloud) to the mobile computing device. This method further includes a combining (i.e., MERGE) process 740 for combining the registered / tagged 3D maps to generate a combined 3D map 750. As shown in Figure 6, since the first registered / tagged 3D map 701 and the third registered / tagged 3D map 703 both have the same identifier (i.e., ID = 1), they are combined 740. The result of the combination is a combined 3D map 750 that includes the tagged areas.In a combination, the registered 3D maps may need to have the same identifier, but parts of the registered 3D maps within the combined 3D may not be tagged.

[0047] The combined 3D map can be used in various applications. The combined 3D map can be used without significantly changing the visualization of the indoor space. For example, the combined 3D map can be presented as part of a 3D tour of an indoor space or to help generate realistic virtual objects in AR applications such as AR cloud anchors, allowing multiple users to add persistent virtual objects to the AR scene that can be observed / interacted with from different (or the same) observation points at different times (or simultaneously). The combined 3D map can also be used as a basis for measurement and / or design. For example, the combined 3D map can be used for on-site surveying and creating computer-aided design (CAD) models. The combined 3D map can also be useful for providing additional functionality in applications that utilize indoor positioning, especially when the combined 3D map includes tagged areas / positions.

[0048] The indoor positioning system (IPS) described above can also be used for purposes other than 3D mapping. These purposes can be further enhanced by associating the collected positioning data with the combined 3D map. For example, the operation of a smart device can be adjusted based on its tracked position within the combined 3D map of the area.

[0049] FIG. 8 shows possible uses of the combined 3D map from the 3D scanning scenario shown in FIG. 4. Although the uses shown to illustrate aspects of the present disclosure are described in detail, it should be understood that obvious variations to this use and other possible uses are within the scope of the disclosed technology.

[0050] As shown in FIG. 8, a first smart device (e.g., smart home hub 801) at a first anchor position tracks the positions of multiple smart devices (or receives positions from multiple smart devices). The multiple smart devices shown include a first smart speaker 810, a second smart speaker 820, and AR glasses 830 (worn by the user). Each smart device can either track itself or be tracked to obtain its relative position with respect to the smart home hub 801. For example, indoor positioning using UWB can determine that the first smart speaker 810 is at a first position (x1, y1, z1), the second smart speaker 820 is at a second position (x2, y2, z2), and the AR glasses 830 (worn by the user) are at a third position (x3, y3, z3). Each determined position is relative to a coordinate system 840 based on the first anchor position where the smart home hub 801 is located.

[0051] The determined positions can be compared with the combined 3D map 750. Additionally, the determined positions may be compared with additional information determined by tagging. For example, the application can conclude that the first smart speaker 810 is in the bedroom and the second smart speaker 820 is in the office based on a comparison of the first and second positions with respect to tagged areas (i.e., bedroom, office) within the combined 3D map 750. The combined 3D map 750 can also help define the boundaries of the indoor space. For example, since the third position is within the area defined by the combined 3D map, the application can also conclude that the AR glasses 830 (worn by the user) are within the boundaries of the indoor space.

[0052] An application can use the determined location of smart devices to affect how they operate. For example, since the AR glasses 830 are determined to be within the boundaries of the indoor space, the application can automatically provide functions (i.e., access) to the user, so that when the user says "Computer, play music in the office", the command is executed. In other words, by using the tracking for the combined 3D, security means can be provided to the application. To execute the command, the application can configure the second smart speaker 820 to play music based on its location within the office. The first smart speaker 810 is configured to remain silent based on its location within the bedroom.

[0053] In addition to regions, specific locations within the combined 3D map can be tagged based on the recognized objects. As shown in the figure, the fourth location (x4, y4, z4) can be tagged based on the object recognized at that location (e.g., the exercise machine 850). An application that tracks a smart device with respect to the combined 3D map 750 can configure the smart device to change its function (e.g., start / stop a process) when the smart device approaches an object (e.g., ≤ 1 meter). For example, when the AR glasses 830 (i.e., the user) approach the exercise machine 850, statistics from previous training can be displayed on the AR display of the AR glasses.

[0054] FIG. 9 is a flowchart of a method for generating a combined 3D map according to a possible implementation of the present disclosure. Method 900 includes selecting 910 an area to scan. Selecting can include placing a mobile computing device within an area of an indoor space for scanning. The method further includes performing 920 a 3D scan of the area (e.g., a room) using the mobile computing device. The 3D scan can include moving the mobile computing device while sensors (e.g., depth sensors, cameras, IMUs) capture information about the area and the mobile computing device. The method further includes identifying 930 the position of the mobile computing device relative to the anchor positions during the 3D scan using a UWB position sensor (e.g., of the mobile computing device). Identifying can include calculating the range and / or angle between the mobile computing device and a smart device configured for UWB and positioned at the anchor positions using UWB signals. The method further includes generating 940 a 3D map of the area. The 3D map can include 3D points relative to a coordinate system based on the anchor positions (e.g., the anchor positions are the origin of the coordinate system). This process can be repeated to generate 950 a collection (i.e., a set) of 3D maps. The method further includes combining 970 3D maps having the same anchor positions 960 to form a combined 3D map 980.

[0055] FIG. 10 shows an example of a computer device 1000 and a mobile computer device 1050 that can be used with the techniques described herein (e.g., to implement a mobile computing device, a smart device, an AR device, etc.). The computing device 1000 includes a processor 1002, a memory 1004, a storage device 1006, a high-speed interface 1008 that connects to the memory 1004 and a high-speed expansion port 1010, and a low-speed interface 1012 that connects to a low-speed bus 1014 and the storage device 1006. Each of the components 1002, 1004, 1006, 1008, 1010, and 1012 can be interconnected using various buses and can be mounted on a common motherboard or in other manners as required. The processor 1002 can process instructions for execution within the computing device 1000, including instructions stored in the memory 1004 or the storage device 1006, and can display graphic information for a GUI on an external input / output device such as a display 1016 coupled to the high-speed interface 1008. In other implementations, multiple processors and / or multiple buses can be used, along with multiple memories and memory types, as required. Also, multiple computing devices 1000 can be connected, and each device can provide a portion of the required operations (e.g., as a server bank, a group of blade servers, or a multiprocessor system).

[0056] The memory 1004 stores information within the computing device 1000. In one embodiment, the memory 1004 is one or more volatile memory devices. In another implementation, the memory 1004 is one or more non-volatile memory devices. The memory 1004 can also be another form of computer-readable medium, such as a magnetic disk or an optical disk.

[0057] The memory device 1006 can provide large-capacity storage to the computing device 1000. In one embodiment, the memory device 1006 may be a computer-readable medium or may include a computer-readable medium, such as a floppy (registered trademark) disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices including a storage area network or other devices within other configurations. The computer program product can be specifically incorporated into an information carrier. The computer program product may also include instructions that execute one or more of the methods as described above when executed. The information carrier is a computer or machine-readable medium such as the memory 1004, the memory device 1006, or the memory on the processor 1002.

[0058] The high-speed controller 1008 manages operations that consume a large amount of the bandwidth of the computing device 1000, while the low-speed controller 1012 manages operations that consume little bandwidth. Such a function assignment is just an example. In one implementation, the high-speed controller 1008 is coupled to a high-speed expansion port 1010 that can receive the memory 1004, the display 1016 (e.g., via a graphics processor or accelerator), and various expansion cards (not shown). In this implementation, the low-speed controller 1012 is coupled to the memory device 1006 and the low-speed expansion port 1014. The low-speed expansion port may include various communication ports (e.g., USB, Bluetooth, Ethernet (registered trademark), wireless Ethernet), and can be coupled to one or more input / output devices such as a keyboard, a pointing device, a scanner, etc., or a networking device such as a switch or a router, for example, via a network adapter.

[0059] As shown in the figure, computing device 1000 can be implemented in many different forms. For example, it may be implemented as a standard server 1020, or may be implemented multiple times within a group of such servers. It may also be implemented as part of a rack server system 1024. Additionally, it can be implemented on a personal computer such as a laptop computer 1022. Alternatively, the components of computing device 1000 can be combined with other components within a mobile device (not shown) such as device 1050. Each of such devices can include one or more computing devices 1000, 1050, and the entire system can be composed of multiple computing devices 1000, 1050 that communicate with each other.

[0060] Computing device 1050 includes, among other components, a processor 1052, a memory 1064, input / output devices such as a display 1054, a communication interface 1066, and a transceiver 1068. A storage device such as a microdrive or other device may also be provided for device 1050 to provide additional storage. Each of the components 1050, 1052, 1064, 1054, 1066, and 1068 are interconnected using various buses, and some of the components may be mounted on a common motherboard or in other suitable ways.

[0061] Processor 1052 can execute instructions within computing device 1050, including instructions stored in memory 1064. The processor may be implemented as a chipset of chips including multiple separate analog and digital processors. The processor can provide coordination of other components of device 1050, such as, for example, a user interface, applications executed by device 1050, and control of wireless communication by device 1050.

[0062] Processor 1052 can communicate with the user via a display interface 1056 coupled to a control interface 1058 and a display 1054. The display 1054 may be, for example, a TFT LCD (Thin Film Transistor Liquid Crystal Display), an LED (Light Emitting Diode) or an OLED (Organic Light Emitting Diode) display, or other suitable display technology. The display interface 1056 can include appropriate circuitry for driving the display 1054 to present graphic and other information to the user. The control interface 1058 can receive commands from the user and convert them for transmission to the processor 1052. Further, an external interface 1062 can be provided to communicate with the processor 1052 to enable short-range communication between the device 1050 and other devices. The external interface 1062 can provide, for example, wired communication in some implementations, or wireless communication in other implementations, or multiple interfaces can be used.

[0063] Memory 1064 stores information within computing device 1050. Memory 1064 can be implemented as one or more of a computer-readable medium, a volatile memory device, or a non-volatile memory device. Extended memory 1074 is also provided and can be connected to device 1050 via expansion interface 1072, which can include, for example, a SIMM (Single In-line Memory Module) card interface. Such extended memory 1074 can provide additional storage space for device 1050 or can also store applications or other information for device 1050. Specifically, extended memory 1074 can include instructions for executing or supplementing the processes described above and can also include secure information. Thus, for example, extended memory 1074 may be provided as a security module for device 1050 and may be programmed with instructions that permit secure use of device 1050. Further, secure applications can be provided via the SIMM card along with additional information such as placing identification information in a non-hackable manner on the SIMM card.

[0064] The memory may include, for example, flash memory and / or NVRAM memory as described below. In one implementation, the computer program product is specifically incorporated into an information carrier. The computer program product includes instructions that execute one or more of the methods as described above when executed. The information carrier is a computer or machine-readable medium such as memory 1064, extended memory 1074, or memory on processor 1052, which can be received via, for example, transceiver 1068 or external interface 1062.

[0065] Device 1050 can communicate wirelessly via communication interface 1066, which may include digital signal processing circuitry as needed. Communication interface 1066 can provide communication under various modes or protocols, such as GSM (registered trademark) voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA (registered trademark), CDMA2000, or GPRS. Such communication may occur, for example, through radio frequency transceiver 1068. Additionally, short-range communication may occur, such as using Bluetooth, Wi-Fi, or other similar transceivers (not shown). Further, GPS (Global Positioning System) receiver module 1070 can provide additional navigation and location-related wireless data to device 1050, and this data can be used as appropriate by applications running on device 1050.

[0066] Device 1050 can also communicate using audio codec 1060, which can receive information spoken by the user and convert it into usable digital information. Audio codec 1060 can similarly generate audible sound for the user, for example, via a speaker within the handset of device 1050. Such sound may include sound from voice calls, recorded sound (e.g., voice messages, music files, etc.), or sound generated by applications operating on device 1050.

[0067] As shown in the figure, computing device 1050 can be implemented in many different forms. For example, it may be implemented as a mobile phone 1080. It may also be implemented as part of a smartphone 1082, a personal digital assistant, or other similar mobile devices.

[0068] The various implementations of the systems and techniques described herein can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations with one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor, which can be special purpose or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a memory system, at least one input device, and at least one output device.

[0069] These computer programs (also called programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented in high-level procedural and / or object-oriented programming languages, and / or in assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., magnetic disks, optical disks, memory, programmable logic devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0070] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device (LED (light-emitting diode), or OLED (organic LED), or LCD (liquid crystal display) monitor / screen) for displaying information to the user, a keyboard, and a pointing device (e.g., mouse or trackball) for the user to input to the computer. Other types of devices can also be used to provide interaction with the user. For example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback). Input from the user can be received in any form, such as acoustic, voice, or tactile input.

[0071] The systems and techniques described herein can be implemented in a computing system that includes back-end components (such as data servers), or middleware components (such as application servers), or front-end components (e.g., a client computer having a graphical user interface or a web browser through which a user can interact with the implementation of the systems and techniques described herein), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), and the Internet.

[0072] A computing system can include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship between clients and servers is created by computer programs that are executed on respective computers and have a client-server relationship with each other.

[0073] In some implementations, the computing device shown in the figures can include sensors that interface with an AR headset / HMD device 1090 to generate an augmented environment for viewing content inserted into the physical space. For example, one or more sensors included in the computing device 1050 or other computing devices shown in the figures can provide input to the AR headset 1090 or, generally, to the AR space. Sensors can include, but are not limited to, touchscreens, accelerometers, gyroscopes, pressure sensors, biometric sensors, temperature sensors, humidity sensors, ambient light sensors, etc. The computing device 1050 can use the sensors to determine the absolute position and / or detected rotation of the computing device in the AR space, which can be used as input to the AR space. For example, the computing device 1050 may be incorporated into the AR space as a virtual object such as a controller, a laser pointer, a keyboard, a weapon, etc. Depending on how the user positions the computing device / virtual object when it is incorporated into the AR space, the user can position the computing device so as to view the virtual object in a particular way within the AR space. For example, if the virtual object represents a laser pointer, the user can operate the computing device as if it were an actual laser pointer. The user can use the device in the same way as when using a laser pointer, such as moving the computing device left and right, up and down, or in a circular motion. In some implementations, the user can aim at a target position using a virtual laser pointer.

[0074] In some implementations, one or more input devices included in or connected to computing device 1050 can be used as input to the AR space. Input devices can include, but are not limited to, touchscreens, keyboards, one or more buttons, trackpads, touchpads, pointing devices, mice, trackballs, joysticks, cameras, microphones, headsets or buds with input capabilities, game controllers, or other connectable input devices. When a computing device is incorporated into the AR space, a user who interacts with the input devices included in computing device 1050 can cause specific actions to occur within the AR space.

[0075] In some implementations, the touchscreen of computing device 1050 can be rendered as a touchpad in the AR space. A user can interact with the touchscreen of computing device 1050. The interaction can be rendered, for example, in an AR headset 1090 as movement on a rendered touchpad within the AR space. The rendered movement can be used to control virtual objects within the AR space.

[0076] In some implementations, one or more output devices included in computing device 1050 can provide output and / or feedback to a user of AR headset 1090 within the AR space. The output and feedback can be visual, tactile, or auditory. Output and / or feedback can include, but are not limited to, vibration, turning one or more lights or strobes on / off, or blinking and / or flashing, sounding an alarm, playing a chime, playing a song, and playing an audio file. Output devices can include, but are not limited to, vibration motors, vibration coils, piezoelectric devices, electrostatic devices, light-emitting diodes (LEDs), strobes, speakers, etc.

[0077] In some implementations, computing device 1050 can appear as another object in a computer-generated 3D environment. User interaction with computing device 1050 (e.g., rotating, shaking, touching the touch screen, swiping a finger on the touch screen) can be interpreted as interaction with an object in the AR space. In an example of a laser pointer in the AR space, computing device 1050 appears as a virtual laser pointer in a computer-generated 3D environment. When the user operates computing device 1050, the user in the AR space sees the movement of the laser pointer. The user receives feedback from the interaction with computing device 1050 in the AR environment on computing device 1050 or on AR headset 1090. The user's interaction with the computing device can be converted into interaction with a user interface generated in the AR environment of the controllable device.

[0078] In some implementations, computing device 1050 can include a touch screen. For example, the user can operate the touch screen to interact with the user interface of the controllable device. For example, the touch screen can include user interface elements such as sliders that can control the properties of the controllable device.

[0079] Computing device 1000 is intended to represent various forms of digital computers and devices, including but not limited to laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Computing device 1050 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are intended to be merely examples and are not intended to limit the implementation of the invention described and / or claimed in this document.

[0080] Numerous embodiments have been described. Nevertheless, it will be understood that various modifications can be made without departing from the spirit and scope of this specification.

[0081] Furthermore, the logical flows shown in the figures do not require the particular order or sequential order shown to achieve desirable results. Additionally, steps can be provided or removed from the flows described, and other components can be added to or removed from the systems described. Accordingly, other embodiments are also within the scope of the claims.

[0082] In addition to the above description, users may be provided with control so that they can select whether and when they can be enabled to collect user information (e.g., the user's social network, social actions, or activities, occupation, user preferences, or information regarding the user's current location) by the systems, programs, or functions described herein, and also whether the user will be sent content or communications from the server. Further, certain data may be processed in one or more ways before being stored or used so that information that can identify an individual is removed. For example, the user's identity information may be processed so that information that can identify the user's individual cannot be identified, or the user's geographical location may be generalized at the location where the location information is obtained so that the user's specific location cannot be identified (e.g., city, postal code, state level). Thus, the user can control what information is collected about the user, how that information is used, and what information is provided to the user.

[0083] Although specific features of the described implementations are shown as described herein, many modifications, substitutions, changes, and equivalents will occur to those skilled in the art. Accordingly, it should be understood that the appended claims are intended to cover all such modifications and changes that fall within the scope of the implementations. These are presented by way of example only and not as limitations, and it should be understood that various changes are possible in form and detail. Any part of the apparatus and / or method described herein can be combined in any combination, except mutually exclusive combinations. The implementations described herein can include various combinations and / or sub-combinations of the functions, components, and / or features of the different implementations described.

[0084] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art. Methods and materials similar or equivalent to those described herein can be used in the practice or testing of the present disclosure. As used in this specification and the appended claims, the singular forms "a", "an", and "the" include plural referents unless the context clearly dictates otherwise. The term "comprising" as used herein and variations thereof are used synonymously with the term "including" and variations thereof and are open-ended and non-limiting terms. The term "any" or "optionally" as used herein means that the subsequent described feature, event, or circumstance may or may not occur, and the description includes both the case where the feature, event, or circumstance occurs and the case where it does not occur. In this specification, ranges may be expressed as "about" one particular value and / or "about" another particular value. When such a range is expressed, aspects include from one particular value and / or to another particular value. Similarly, when values are expressed as approximations using the antecedent "about", it will be understood that the particular value forms another aspect. It will further be understood that each endpoint of each range is important both in relation to the other endpoint and independently of the other endpoint.

Claims

Claim 1 A method for generating a three-dimensional (3D) map, the method comprising: performing a first scan of a first region using a depth sensor of a mobile computing device; identifying the position of the mobile computing device relative to an anchor position during the first scan using an ultra-wideband position sensor of the mobile computing device; generating a first 3D map of the first region relative to the anchor position. Claim 2 performing a second scan of a second region using the depth sensor of the mobile computing device; identifying the position of the mobile computing device relative to the anchor position during the second scan using the ultra-wideband position sensor of the mobile computing device; generating a second 3D map of the second region relative to the anchor position; further comprising combining the first 3D map and the second 3D map based on the anchor position, the method according to claim 1. Claim 3 The method according to claim 2, wherein the first region is a first room in a building or a house, and the second region is a second room in the building or the house. Claim 4 The method according to claim 3, wherein the anchor position is the position of a smart device fixedly arranged in the building or the house and configured for ultra-wideband communication. Claim 5 The method according to claim 3, wherein the anchor position is the position of a smart device fixedly arranged outside the building or the house and configured for ultra-wideband communication. Claim 6 Combining the first 3D map and the second 3D map based on the anchor position comprises: spatially arranging the first 3D map and the second 3D map to form a third 3D map covering both regions, the method according to any one of claims 2 to 5. Claim 7 The method according to any one of claims 1 to 5, wherein the depth sensor is a lidar sensor, a camera, or an ultrasonic sensor. Claim 8 The method according to any one of claims 1 to 5, wherein the mobile computing device is a mobile phone, a tablet, or augmented reality glasses. Claim 9 The method according to any one of claims 1 to 5, wherein the ultra-wideband position sensor is an ultra-wideband tag.

10. Identifying the position of the mobile computing device relative to a fixed position includes determining a range between the ultra-wideband tag and a smart device configured for ultra-wideband communication with the ultra-wideband tag, the smart device being located at the anchor position, the method according to claim 9.

11. Identifying the position of the mobile computing device relative to a fixed position includes determining a range between the ultra-wideband tag and a plurality of smart devices configured for ultra-wideband communication with the ultra-wideband tag, one of the plurality of smart devices being located at the anchor position, the method according to claim 9.

12. A method of controlling the operation of a smart device based on the position of the smart device, the method including generating a combined 3D map including a first region and a second region scanned by a mobile computing device, the first region and the second region being arranged in the combined 3D map according to their relative positions with respect to an anchor position, the relative positions being determined by ultra-wideband (UWB) communication, the method including tagging a plurality of positions in the combined 3D map, tracking the smart device based on UWB communication and the combined 3D map, determining that the smart device is at a tagged position in the combined 3D map, and adjusting the operation of the smart device based on the tagged position.

13. Tagging the plurality of positions in the combined 3D map includes identifying a boundary of a building or a house based on the combined 3D map, and tagging positions outside the boundary of the building or the house, the method according to claim 12.

14. Adjusting the operation of the smart device based on the tagged position includes restricting access of the smart device to a network when the smart device is at the position outside the boundary of the building or the house, the method according to claim 13.

15. Tagging the plurality of positions in the combined 3D map comprises identifying an object based on an image associated with the combined 3D map, and tagging an area around the object, and is the method according to any one of claims 12 to 14. **Claim 16** Adjusting the operation of the smart device based on the tagged positions comprises presenting information about the object on the smart device when the smart device enters the area around the object, and is the method according to claim 15. **Claim 17** Tagging the plurality of positions in the combined 3D map comprises identifying a room based on the combined 3D map, and tagging the room, and is the method according to any one of claims 12 to 14. **Claim 18** Adjusting the operation of the smart device based on the tagged positions comprises controlling the playback of media on the smart device based on when the smart device enters the room, and is the method according to claim 17. **Claim 19** Tagging the room comprises identifying an object in the room based on an image associated with the combined 3D map, determining a room type based on the object in the room, and tagging the room according to the room type, and is the method according to claim 17. **Claim 20** A system for generating a three-dimensional (3D) map, the system comprising a smart device fixedly arranged at an anchor position and configured for ultra-wideband (UWB) communication, and a mobile computing device, wherein the mobile computing device a depth sensor configured to collect a first set of depth data corresponding to the first region when the depth sensor is scanned over the first region during a first scan, and a second set of depth data corresponding to the second region when the depth sensor is scanned over the second region during a second scan, a UWB tag configured to determine the range between the mobile computing device and the anchor position during the first scan and the second scan based on UWB communication between the smart device and the UWB tag, and by software instructions Register the first set of depth data at the anchor position based on the range between the mobile computing device and the anchor position during the first scan. Generate a first 3D map for the anchor position based on the registered first set of depth data. Register the second set of depth data at the anchor position based on the range between the mobile computing device and the anchor position during the second scan. Generate a second 3D map for the anchor position based on the registered second set of depth data. Combine the first 3D map and the second 3D map. A processor configured as such. A system including the above.

Images