Scanning Framework for Spatial Mapping

Abstract

The disclosed implementations generate a virtual representation of space based on a model. The model is updated using image data according to a difference metric. The difference metric is determined for a portion of space based on the image data and the current state of the model. The virtual representation is provided to a user device.

Description

Technology Field

[0001] Cross-reference regarding related applications

[0002] This application is a continuation of U.S. Application No. 18 / 584,887 filed on February 22, 2024, and claims priority to the entire disclosure of which is incorporated herein by reference. Background Technology

[0003] Augmented Reality (AR) and Virtual Reality (VR) are key technologies that are driving innovation across a wide range of industries and changing the way users interact with and consume digital content.

[0004] An embodiment of the present disclosure relates to creating and maintaining a model that generally represents a real-world space and can be used to provide a virtual representation of that space (e.g., a room, partitioned space, conference room, niche, courtyard, entrance, passageway, etc.) to a device (e.g., a user device). The device may be a computing device (e.g., a mobile device, a laptop computing device, a head-mounted display (HMD) device, or a mixed reality (XR) device such as an augmented reality (AR) and / or virtual reality (VR) device). In some embodiments, these systems receive image data from an imaging device (e.g., a camera) and determine a difference metric for a portion of the space represented by the model. The image data is updated with image data specific to the portion of the space or a portion of the image data. Thus, these systems map the real-world space to a virtual representation of the real-world space (e.g., mapping when triggered by a condition, mapping periodically, mapping continuously), which reduces the discrepancy between the real-world space and the virtual representation, for example, when a user interacts with the virtual representation through the device.

[0005] It is understood that the method according to the present disclosure may include any combination of the embodiments and features described herein. That is, the method according to the present disclosure is not limited to the combination of embodiments and features specifically described herein, but may include any combination of the embodiments and features provided.

[0006] Details regarding one or more embodiments of the present disclosure are set forth in the accompanying drawings and the description below. Other features and advantages of the present disclosure will be apparent from the description and drawings, and from the claims. Brief explanation of the drawing

[0007] The following describes in detail the aspects of the subject matter of the invention together with the attached drawings. FIGS. 1a and 1b illustrate an exemplary environment in which a device is used to create and update a model representing space, depending on the implementation as the described scanning system. FIG. 2 illustrates an exemplary architecture that can be used to implement the embodiments of the present disclosure. FIG. 3 illustrates two exemplary implementations of the modules of the exemplary architecture of FIG. 2. FIG. 4 illustrates an exemplary environment that can be used to carry out implementations of the present disclosure. FIG. 5 illustrates a flowchart of a non-limiting process that can be performed by implementations of the present disclosure. FIG. 6 illustrates an exemplary system comprising a computer or computing device that may be programmed or otherwise configured to implement the system or method of the present disclosure. Specific details for implementing the invention

[0008] Computing systems (e.g., mobile devices, laptop computing devices, XR, augmented and / or virtual reality (VR) systems) enable users to experience content in a space (e.g., a room, partitioned space, conference room, niche, courtyard, entrance, hallway, etc.) using computing devices (e.g., mobile devices, laptop computing devices, HMDs, XR devices, AR devices, or VR devices) configured to present a virtual environment. For example, a computing system may be configured to enable users to experience content in a 3D space using computing devices configured to present a three-dimensional (3D) environment (e.g., a virtual 3D environment). The 3D environment may include 3D assets representing virtual objects and 3D models of spaces or sections of a larger real-world environment generated from scans. For example, users can work within a virtual representation of their physical office from any location using a headset device (e.g., a head-mounted device). Some systems allow users to load pre-scanned assets (e.g., through a user interface) through a rendering process. At least one technical problem with this approach is that the scanned space is static, and therefore the discrepancy between the real and virtual spaces increases over time. For example, when someone moves or adjusts the position of an object within the space after an initial scan, a discrepancy (the position of the moved object) exists between the real-world space and the virtual representation. Furthermore, since generating a virtual representation through a model requires a large number of images, scanning the space using a device takes a significant amount of time and is not user-friendly.

[0009] The implementations described herein provide at least one technical solution to these technical problems. In particular, the implementations of the described system provide a scanning framework for mapping real-world space into virtual space (e.g., mapping when triggered by a condition, periodic mapping, continuous or continuous mapping, continuous or continuous mapping over a period of time) such that there is little discrepancy between real-world space and virtual space. In some implementations, virtual space is represented by a common asset or model (e.g., a Neural Radiation Field (NeRF) model that can be used to generate a rendering of virtual space through point clouds, light fields, etc.). The initial build of the model can be generated rapidly (e.g., within seconds) based on an initial scan using a relatively small number of images (e.g., fewer than 100, fewer than 1,000, fewer than 10,000, etc.) and can provide a rough scene reconstruction of individual spaces. In some implementations, the model is generated and stored locally on a user device (e.g., a headset) and is updated based on a semantic difference calculation method to more accurately reflect the current state of individual real-world space. Generally, semantic difference calculation methods determine differences between groups by comparing information (e.g., spatially associated features or objects). Semantic difference calculation methods define a metric between feature points in space at time point 0 and feature points entering at a different time point T. Specifically, the described scanning system is configured to determine a difference metric between a mapped 3D space and information collected from that space (e.g., information collected via a headset) based on semantic difference calculation methods. In some implementations, a virtual model is updated with image data (or a subset of image data) based on the difference metric.

[0010] In some implementations, the described scanning system does not involve explicit scanning of the space by the user. Instead, scanning may be performed by processing data received from the device's sensors, such as an external camera, while the device is being used or worn, for example, to generate a 3D representation of the space. Therefore, since the responsibility for scanning the space is transferred from the user to the described scanning system, which passively records data while the device is being used, the device user experience does not change from the user's perspective. Additionally, scanning and setup times are reduced, providing the user with a solution that can be utilized even while on the move, based on device usage.

[0011] In some implementations, when a user uses a device (e.g., a headset set to passthrough mode), scenes and objects recorded in the image data are stitched together manually and dynamically to build and update a model. Once a model is created from the space, the scanning system is configured to process the incoming image data to map features of the real-world space to features of the individual models (e.g., matching them) and to determine a difference metric between the features of the real-world space captured in the image data and these features represented in the model using a semantic difference operation method. In some examples, one or more features may be newly added to the real-world space or removed from the real-world space after the latest model update. In such cases, the difference metric captures the addition or removal of features. In some implementations, the model is then updated spatially and / or temporally using the image data (or a subset of the image data) based on the difference metric. In other words, the difference metric is used to determine when to update the model using information captured in the image data. Therefore, the user does not need to identify the space or provide labels for objects within the space when creating and / or updating individual models.

[0012] FIGS. 1A and 1B illustrate an exemplary environment (100) in which a device (102) (e.g., a headset) is used to generate a model (130) representing a space (110) according to an implementation of the described scanning system (FIG. 1A) and to update the model (130) to an updated model (131) (FIG. 1B). As illustrated, the space includes a feature (112).

[0013] FIG. 1a illustrates that a device (102) captures an initial set of image data (120) for a space (110) and a feature (112) within the space (110) at an initial time point (time 0). A model (130) is generated using the initial set of image data (120) according to the implementation of the described scanning system.

[0014] FIG. 1b illustrates that the device (102) captures an additional set of image data (121) for a space (110) and a feature (112) at a later time (time N). The additional set of image data (121) includes information related to a new feature (114) within the space (110).

[0015] As illustrated in FIG. 1b, the process (150) illustrates an exemplary series of steps for how a model (130) can be updated based on an additional set of image data (121). In step (152), the additional set of image data (121) is processed to determine a representation (122) of space (110) at time N (a second scan time by the device (102)). The representation (122) is compared with the current model (130) of space (110) to determine a difference metric for a region or area (124) within space (110) containing a new feature (114). In step (154), based on the difference metric and threshold, the scanning system is configured to update the model (130) using the additional set of image data (121). More specifically, in some implementations, a subset of an additional set of image data (121) containing information related to the new feature (114) or the region surrounding the new feature (114) is identified (as the region or area (124). In step (156), the model (130) is updated to a model (131) that represents the space (110) at time N. Accordingly, the discrepancy between the real-world space (110) and the virtual representation by the current model (131) is reduced. In other words, the model (131) includes a model (130) having information related to the new feature (114) updated using information related to the new feature (114) captured in the representation (122) (i.e., data captured in the image data (121)).

[0016] For simplification, an additional set of image data (121) is illustrated and described as having information related to these new features (114). However, the additional set of image data (121) may also include information related to changes in features (e.g., a new location of one of the features (112)) (e.g., additionally included or only such information). In the described implementation, the model (130) is updated using the additional set of image data (121) according to the process (150) to represent these changes in a similar manner.

[0017] FIG. 2 illustrates an exemplary architecture (200) for the described scanning system. As illustrated, the exemplary architecture (200) includes a device (102), a scanning module (210), a training module (212), a semantic difference calculation module (220), an asset update module (222), a common asset (230), and a rendering module (240). The device (102) is substantially similar to the computing device (610) described below with reference to FIG. 6. Additionally, while the device (102) in the drawings and descriptions included herein is an AR / VR headset-type device, it is considered that implementations of the present disclosure may be realized using any suitable computing device, such as the user computing device (402, 404, 406, 408) described below with reference to FIG. 4. In some implementations, the modules (210, 212, 220, 222, 240) are executed through the electronic processor of the device (102). In some implementations, the modules (210, 212, 220, 222, 240) are provided through a backend system (e.g., a backend system (430) described later with reference to FIG. 4), and the device (102) is configured to communicate with the backend system through a network.

[0018] In some implementations, the scanning model (210) receives an initial set of image data (120). As described above with reference to FIGS. 1a and 1b, the initial set of image data (120) may be collected by the device (102) during an initial scan of a space such as space (110). The device may include a headset or mobile device and may include an imaging sensor / device (e.g., a camera). In some implementations, the device (102) includes one or more video see-through (VST) cameras, and the set of image data (120) is red, green, and blue (RGB) readings provided by the cameras. In some implementations, the device (102) is configured to prompt the user to begin constructing a virtual representation of the space and / or provide the user with commands to perform an initial scan. In some implementations, the initial scan is performed manually when the user uses the device (102).

[0019] In some implementations, the configuration module (212) generates a common asset (230). In some implementations, the configuration module (212) generates the common asset (230) based on image data (120) received from the scanning module (210). The common asset (230) is a representation, such as a light field data structure of real-world space or spaces, generated from an active model, such as a dense NeRF model of real-world space or spaces. In some implementations, the common asset (230) is a model (e.g., a dense NeRF model). The common asset (230) is determined based on image data of real-world space (e.g., an initial set of image data (120) and sets of image data (121) received thereafter). In some implementations, the configuration module (212) generates the common asset (230) based on multiple images from an initial scan. In this example, the common asset (230) may have missing information or a rough scene reconstruction of individual real-world spaces. In manual mode, for example, the device (102) may provide virtual features or items to attract the user's attention and collect additional sets of image data (121) processed by the configuration module (212) to update the common asset (230). For example, the configuration module (212) may determine that a specific area or zone within the space represented by the common asset (230) is incomplete (e.g., lacking sufficient information) based on a determined confidence score, and the device (102) may be configured to provide virtual features in the direction of the area (e.g., through a display) to collect additional images from the area by causing the user to look at the area (thereby causing the imaging device to face the area).In other words, the device (102) may be configured to mask the direction to the user to collect additional information for the virtual representation, instead of displaying, for example, a virtual space with empty areas or missing elements / features. In some implementations, the configuration module (212) is configured to initially fill in the gaps in the model created based on data collected from similar environments. For example, if one side of an object or area within the space is not included in the initial set of image data (120), the configuration module (212) identifies similar elements or features from a common / training image set to fill in the gaps until more information is collected from the device (102) (for example, until collected by the semantic difference calculation module (220).

[0020] Once the common asset (230) is established, in some implementations, a semantic difference operation module (220) processes additional sets of image data (121) collected by the device (102) while the device is in use (e.g., in passthrough mode). In some implementations, the semantic difference operation module (220) determines a difference metric for regions or regions of the common asset (230). The difference metric is a measure indicating the difference between a region in the real world at time N (determined by the set of image data (121) collected at time N) and the current state of the common asset (230) (or the state of the common asset at a specific time). In some examples, the semantic difference operation module (220) processes additional sets of received image data (121) to determine whether the common asset (230) contains a virtual representation of the real world space expressed in the set of image data (121) (e.g., whether a model for that real world space has been created). In some implementations, if no virtual representation is found, the semantic difference operation module (220) provides a set of image data (121) to the scanning module (210) as an initial set of image data (120).

[0021] Referring to FIG. 3, in some implementations, a semantic difference operation module (220) is configured to compare a difference metric for a specific area within space with a threshold. The threshold includes limits or boundaries to be compared with the difference metric to determine whether to update the common asset (230). In some implementations, the threshold is configurable (e.g., by a user) and is set based on the type of model used to build the common asset (230), the granularity of the common asset (230), sensor or device specifications (e.g., device (102) or sensor specifications of device (102)), user preferences, etc. For example, the threshold may represent the amount of change within the model (e.g., as a percentage or absolute value) that triggers the integration of information from the collected image data into the model.

[0022] In an exemplary implementation, the semantic difference calculation module (220) is configured to perform an image-based segmentation technique on a received image (310) (e.g., each received image) provided by the device (102). In an exemplary implementation, the semantic difference calculation module (220) is configured to calculate the nearest neighbor segmentation difference between the segmented images (illustrated at time point 0 and time point N in FIG. 3). In an exemplary implementation, to determine whether the scene depicted in the received image (310) at time point N contains a new object or semantic information (e.g., illustrated as a region or area (312)), the semantic difference calculation module (220) is configured to compare the difference thus determined with a threshold. Based on the threshold (e.g., if the difference metric exceeds the threshold), the semantic difference calculation module (220) is configured to provide a set of image data (121) to the asset update module (222).

[0023] Referring again to FIG. 2, in some implementations, a semantic difference operation module (220) is configured to determine a subset of a set of image data (121) associated with a specific region in space associated with a difference metric (e.g., a set of image data (121) containing information within a set radius of the region) and to provide the corresponding subset of image data to an asset update module (222). In some implementations, the asset update module (222) is configured to update a common asset (230) based on a set of image data (121) (or a subset of a set of image data (121)). For example, when the common asset (230) is a NeRF model, the set of image data (121) (or a subset of a set of image data (121)) is an incremental NeRF input. In some implementations, the asset update module (222) is configured to modify an identified region (based on a difference metric associated with the region) by choosing to interpolate toward a local surgery value of the common asset (230). In some implementations, the semantic difference operation module (220) is configured to update elements of the common asset (230) based on a weighted average reflecting a confidence metric determined according to a difference metric and a threshold when a new object or feature is discovered. For example, the confidence metric may be determined based on a delta (e.g., a measure of difference or change) between the difference metric and the threshold. For example, the confidence metric may be a numerical value (e.g., a percentage or an absolute value) representing a confidence level based on the delta, and the delta is a numerical value (e.g., a percentage or an absolute value) representing the difference between the difference metric and the threshold.

[0024] In the example illustrated in FIG. 3, the output image (320) provided by the asset update module (222) includes differential semantic information portions highlighted as 322. In an exemplary implementation, the identified region (312) is first polled in world space around a geometric point (e.g., center). Local NeRF reconstruction (e.g., NeRF model) is then performed around a point within a radius set using the previous image(s) (320). The resulting image (320) is replaced with an existing common asset (230) within that radius or interpolated within the common asset (230) (e.g., convex interpolation into a 4D light field space expressed as a Х local_nerf_old + (1-a) Х local_nerf_new, where a represents the common asset (230)).

[0025] Referring again to FIG. 2, the rendering module (240) is configured to generate a virtual rendering of space based on common assets (230) and to provide said virtual rendering to the device (102) for viewing. In some implementations, the rendering module (240) is configured to generate a biocular rendering based on common assets (230). In some implementations, the rendering module (240) determines a light field or point cloud for space based on common assets (230), which may be used to determine the virtual rendering. Because the common assets (230) are updated, the virtual rendering provided by the rendering module (240) corresponds more closely to the current state of the individual real-world space (or the state represented at some point T). In some implementations, the rendering module (240) is configured to provide visual effects based on common assets (230), such as shading techniques (e.g., techniques for smoothly fading out objects) or light field flow methods (e.g., natural movement over a time interval). In some implementations, the rendering module (240) is configured to provide multiple renderings of the space. For example, a heat map may be used to identify distinct common configurations of the space. In such implementations, the rendering module (240) is configured to provide the user with an option to select a specific configuration of the space.

[0026] FIG. 4 illustrates an exemplary environment (400) that may be used to implement the embodiments of the present disclosure. The exemplary environment (400) includes computing devices (402, 404, 406, 408), a backend system (430), and a communication network (410). The communication network (410) may include a wireless portion and a wired portion. In some cases, the communication network (410) is implemented using one or more existing networks, e.g., a cellular network, the Internet, a terrestrial mobile radio (LMR) network, a Bluetooth network, a wireless local area network (e.g., Wi-Fi), a wireless accessory personal area network (PAN), a machine-to-machine (M2M) network, and a telephone network. The communication network (410) may also include networks to be developed in the future. In some implementations, the communication network (410) includes the Internet, an intranet, an extranet, or an intranet and / or extranet that communicates with the Internet. In some implementations, the communication network (410) includes a telecommunications network or a data network.

[0027] In some implementations, the communication network (410) connects websites, devices (e.g., computing devices (402, 404, 406, 408)), and backend systems (e.g., backend systems (430)). In some implementations, the communication network (410) may be accessed via a wired or wireless communication link. For example, mobile computing devices (e.g., smartphone devices (402) and tablet devices (406)) may use a cellular network to access the communication network (410).

[0028] In some examples, users (422, 424, 426, 428) interact with the system through a graphical user interface (GUI) (e.g., the user interface (625) described below with reference to FIG. 6) or a client application installed and running on their individual computing devices (402, 404, 406, 408).

[0029] In some examples, computing devices (402, 404, 406, 408) provide viewing data (e.g., a virtual representation of space) on screens that users (422, 424, 426, 428) can interact with. In some examples, computing devices (402, 404, 406, 408) provide a virtual representation of space (e.g., via a headset or earpiece) determined according to an implementation of the described system (e.g., based on a model generated from collected image data). In some implementations, computing devices (402, 404, 406, 408) are substantially similar to the computing device (610) described below with reference to FIG. 6. The computing devices (402, 404, 406, 408) may include any suitable type of computing device, such as (e.g., each) a desktop computer, a laptop computer, a handheld computer, a tablet computer, a personal digital assistant (PDA), an augmented reality (AR) / virtual reality (VR) device, a cellular phone, a network appliance, a camera, a smartphone, an enhanced general packet radio service (EGPRS) mobile phone, a media player, a navigation device, an email device, a game console, or a suitable combination of two or more of these devices or other data processing devices.

[0030] For simplification, four user computing devices (402, 404, 406, 408) are illustrated in FIG. 4. In the illustrated exemplary environment (400), computing device (402) is illustrated as a smartphone, computing device (404) is illustrated as a tablet computing device, computing device (406) is illustrated as a desktop computing device, and computing device (408) is illustrated as an AR / VR device. However, it is considered that implementations of the present disclosure may be realized with any of the suitable computing devices mentioned previously. Additionally, implementations of the present disclosure may use any number of devices.

[0031] In some implementations, the backend system (430) includes at least one server device (432) and optionally at least one data store (434). In some implementations, the server device (432) is substantially similar to the computing device (610) shown below with reference to FIG. 6. In some implementations, the server device (432) is a server-class hardware type device. In some implementations, the backend system (430) includes a computer system using clustered computers and components to function as a single pool of seamless resources when accessed via a communication network (410). For example, these implementations may be used in data center, cloud computing, storage area network (SAN), and network attached storage (NAS) applications. In some implementations, the backend system (430) is deployed using virtual machine(s).

[0032] In some implementations, the data store (434) is a storage for continuously storing and managing a data set (e.g., training data or image data containing common elements that can be used to build a model representing real-world space or to supplement missing common elements of a model). Exemplary data stores that may be used within the described system include data stores such as databases, as well as simpler types of storage such as files, emails, etc. In some implementations, the data store (434) includes a database. In some implementations, a database is a series of bytes or an organized set of data managed by a database management system (DBMS).

[0033] In some implementations, the backend system (430) hosts one or more computer implementation services provided by the described system, which allow users (422, 424, 426, 428) to interact using individual computing devices (402, 404, 406, 408). For example, in some implementations, the backend system (430) is configured to create and update a model of real-world space and provide a virtual representation of real-world space to the user computing devices (402, 404, 406, 408) according to the model.

[0034] FIG. 5 illustrates a flowchart of an exemplary process (500) that can be implemented by an implementation of the present disclosure. The exemplary process (500) can be implemented by systems and components described with reference to FIGS. 2 through 4 and FIG. 6. The exemplary process (500) generally shows in more detail how a model representing real-world space is updated based on image data collected from space.

[0035] For clarity of presentation, the following description generally describes an exemplary process (500) in the context of FIGS. 1a through 4 and FIGS. 6. However, it will be understood that the process (500) may be performed, for example, by any other suitable system, environment, software, and hardware, or a combination of suitable systems, environments, software, and hardware. In some implementations, various operations of the process (500) may be executed in parallel, in combination, in loops, or in any order.

[0036] In step 502, image data of a space represented as a model is received from an imaging device. In some implementations, the space includes a room or compartment of a larger environment.

[0037] The process (500) proceeds from step 502 to step 504, where a difference metric for a portion of space is determined based on the current state of the image data and the model (i.e., based on the image data and the current model). In some implementations, the difference metric for a portion of space is determined according to a semantic difference calculation method. In some implementations, the semantic difference calculation method defines multiple metrics for feature points from a portion of space and compares the feature points in the image data with feature points defined by the model. In some implementations, the semantic difference calculation method compares the feature points in the image data with feature points defined by the model according to the nearest neighbor segmentation difference. For example, the difference metric may be determined by considering the location data (e.g., 2D or 3D coordinates) of the image data and the current model and / or the brightness and / or color values ​​of the feature points (e.g., samples or pixels representing the feature points). Calculating the difference metric may include determining the difference between the location data and / or the brightness and / or color values.

[0038] The process (500) proceeds from step 504 to step 506, where the model is updated with an updated model based on image data and a difference metric. In some implementations, the model is updated with an updated model using a subset of image data based on a difference metric and a threshold. In some implementations, the subset of image data includes a portion of the space and is determined according to the difference metric. In some implementations, the subset of image data is incorporated into the updated model according to weight values ​​that reflect the delta between the difference metric and the threshold. For example, the orientation or position of an object in space may be represented in a virtual model according to the new orientation or position of the object and the previous orientation or position of the object within the collected image data. As a simple example, an object that is represented as rotated 90 degrees relative to another object in space and rotated 120 degrees relative to another object in image data may be represented in an updated model with a value between 90 and 120 degrees relative to the other object, determined by weight values ​​(for example, some information in the image data is given twice the weight of the corresponding information in the previous model, so that the object whose position has changed may be represented as 110 degrees relative to the other object in the updated model, and the position of the object in the model eventually matches the position in the real world (position rotated 120 degrees relative to the other object) according to subsequent image data / updates). In some implementations, a subset of image data is determined based on the position and radius values ​​of a part of space.

[0039] The process (500) proceeds from step 506 to step 508, where a virtual representation of space is generated (e.g., rendered) based on an updated model. In some implementations, the model is generated based on an initial set of image data of space received from an imaging device. In some implementations, the model includes a neural radiation model. In some implementations, the virtual representation of space is provided to a user device for display. In some implementations, incomplete sections of the model are identified. In some implementations, a prompt for collecting additional information is determined based on the incomplete sections of the model and the image data. In some implementations, the prompt is provided to the user device. In some implementations, the incomplete sections of the model are updated using image data collected from a similar environment. In some implementations, the user device includes an AR / VR headset or a mobile device. In some implementations, the user device includes an imaging device. In some implementations, the virtual representation of space is generated based on a point cloud or light field determined according to the updated model. In some implementations, the virtual representation includes a biocular rendering of space. From step 508, the process (500) is terminated or repeated.

[0040] FIG. 6 illustrates an exemplary computing system (600) comprising a computer or computing device (610) that may be programmed or otherwise configured to implement the system or method of the present disclosure. For example, the computing device (610) may be programmed or otherwise configured to implement the process (500). In some cases, the computing device (610) includes an operating system configured to execute executable instructions. The operating system is software that includes programs and data, for example, that manage the hardware of the device and provide services for the execution of applications.

[0041] In the illustrated embodiment, the computer or computing device (610) includes an electronic processor (612) (also referred to herein as ‘processor’ and ‘computer processor’), such as a central processing unit (CPU) or a graphics processing unit (GPU), which is optionally a single-core, multi-core processor, or multiple processors for parallel processing. The illustrated embodiment also includes memory (617) (e.g., random access memory, read-only memory, flash memory), an electronic storage unit (614) (e.g., hard disk or flash), a communication interface module (615) for communicating with one or more other systems (e.g., network adapter or modem), and peripheral devices (616) such as a cache, other memory, data storage, microphone, speaker, etc. In some embodiments, the memory (617), storage unit (614), communication interface module (615), and peripheral devices (616) communicate with the electronic processor (612) via a communication bus (shown in solid line), such as a motherboard. In some embodiments, the bus of the computing device (610) includes multiple buses. The hardware components described above of the computing device (610) may be used, for example, to facilitate the operation of an operating system and one or more applications running through the operating system. For example, a virtual representation of space may be provided through a user interface (625). In some implementations, the computing device (610) includes more or fewer components than illustrated in FIG. 6 and performs functions other than those described herein.

[0042] In some implementations, the memory (617) and storage unit (614) comprise one or more physical devices used to store data or programs temporarily or permanently. In some implementations, the memory (617) is volatile memory and may use power to retain stored information. In some implementations, the storage unit (614) is non-volatile memory and retains stored information when power is not supplied to the computer. In additional implementations, the memory (617) or storage unit (614) is a combination of devices such as those disclosed herein. In some implementations, the memory (617) or storage unit (614) is distributed across multiple machines, such as network-based memory or memory in multiple machines performing the operation of the computing device (610).

[0043] In some cases, the storage unit (614) is a data storage unit or data repository for storing data. In some cases, the storage unit (614) stores files such as drivers, libraries, and storage programs. In some implementations, the storage unit (614) stores data received by the device (e.g., image data and / or common assets or models). In some implementations, the computing device (610) is a network (e.g.,

[0044] It includes one or more additional data storage units located externally, such as being located on a remote server communicating through the network (410) described above with reference to FIG. 4.

[0045] In some embodiments, the platform, system, medium, and method as described herein are implemented by machine or computer executable code stored in an electronic storage location of a computing device (610), such as memory (617) or storage unit (614), e.g., a non-transient computer-readable storage medium. In additional embodiments, the computer-readable storage medium is optionally removable from the computer. Non-limiting examples of computer-readable storage media include compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), flash memory devices, solid-state memory, magnetic disk drives, magnetic tape drives, optical disk drives, cloud computing systems and services, etc. In some cases, the computer-executable code is encoded on the medium permanently, substantially permanently, semi-permanently, or non-transiently.

[0046] In some implementations, the electronic processor (612) is configured to execute code. In some implementations, machine-executable or machine-readable code is provided in the form of software. In some examples, the code is executed by the electronic processor (612) during use. In some cases, the code is retrieved from the storage unit (614) and stored in memory (617) for easy access by the electronic processor (612). In some situations, the storage unit (614) is excluded, and the machine-executable instructions are stored in memory (617).

[0047] In some cases, the electronic processor (612) is a component of a circuit, such as an integrated circuit. One or more other components within the computing device (610) may optionally be included in the circuit. In some cases, the circuit is an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA). In some cases, the operations of the electronic processor (612) may be distributed across multiple machines (individual machines may have one or more processors) that can be coupled directly or via a network.

[0048] In some cases, the computing device (610) is optionally operablely coupled to a communication network, such as the network (410) described above with reference to FIG. 4, via a communication interface module (615) that may include a digital signal processing circuit. In some implementations, the communication interface module (615) provides communication according to various modes or protocols, particularly GSM (global system for mobile) voice calls, Short Message Service (SMS), Enhanced Messaging Service (EMS) or Multimedia Messaging Service (MMS) messaging, Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Broadband Code Division Multiple Access (WCDMA), CDMA2000, or General Packet Radio Service (GPRS). Such communication may occur, for example, through a transceiver. Additionally, short-range communication may occur, for example, using Bluetooth, Wi-Fi, or other such transceivers.

[0049] In some cases, the computing device (610) includes or communicates with one or more output devices (620). In some cases, the output device (620) includes a display for transmitting visual information to a user. In some cases, the output device (620) is a touch-sensing display that combines a display with a touch-sensing element operable to detect touch input and functions as both the output device (620) and the input device (630). In other cases, the output device (620) is a combination of devices such as those disclosed herein. In some cases, the output device (620) displays a user interface (625) generated by the computing device.

[0050] In some cases, the computing device (610) includes or communicates with one or more input devices (630) configured to receive information from a user. In some cases, the input device (630) is a keyboard. In some cases, the input device (630) is a keypad (e.g., a phone-based keypad). In some cases, the input device (630) is a cursor control device including, as a non-limiting example, a mouse, trackball, trackpad, joystick, game controller, or stylus. In some cases, as described above, the input device (630) is a touchscreen or multi-touchscreen. In other cases, the input device (630) is a microphone for capturing voice or other sound input. In other cases, the input device (630) is an imaging device such as a camera. In yet another case, the input device is a combination of devices such as those disclosed herein.

[0051] It should also be noted that multiple hardware and software-based devices, as well as multiple different structural components, may be used to implement the described examples. Additionally, for the purposes of discussion, the implementations may include hardware, software, and electronic components or modules, which may be exemplified and described as if most of the components were implemented only in hardware. In some implementations, the electronic-based aspects of the present disclosure may be implemented as software executable by one or more processors, such as an electronic processor (612) (e.g., stored on a non-transient computer-readable medium). As such, it should be noted that multiple hardware and software-based devices, as well as multiple different structural components, may be used to implement various implementations.

[0052] While specific drawings illustrate hardware and software located within a specific device, it should be understood that these drawings are for illustrative purposes only. In some implementations, the illustrated components may be combined or divided into separate software, firmware, or hardware. For example, instead of being located within and performed by a single electronic processor, logic and processing may be distributed among multiple electronic processors. Regardless of how they are combined or divided, the hardware and software components may be located on the same computing device or distributed between different computing devices connected by one or more networks or other suitable communication links.

[0053] Additionally, various implementations of the systems and technologies described herein may be realized in digital electronic circuits, integrated circuits, specially designed application-specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable or interpretable on a programmable system comprising a storage system, at least one input device, and at least one programmable processor, which may be specialized or general-purpose, coupled to receive data and instructions from at least one output device and to transmit data and instructions thereto.

[0054] Such computer programs (also known as programs, software, software applications, or code) comprise computer-readable instructions or machine instructions for a programmable electronic processor and may be implemented in a high-level procedural or object-oriented programming language or assembly / machine language. As used herein, the terms 'machine-readable medium' and 'computer-readable medium' refer to any computer program product, device, or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions or data to a programmable processor, and include a machine-readable medium that receives machine instructions as machine-readable signals. The term 'machine-readable signal' refers to any signal used to provide machine instructions or data to a programmable processor.

[0055] The functions of computer-readable instructions can be combined or distributed as desired in various environments. In some implementations, a computer program contains a single instruction sequence. In some implementations, a computer program contains multiple instruction sequences. In some implementations, a computer program is provided from a single location. In other implementations, a computer program is provided from multiple locations. In various implementations, a computer program contains one or more software modules. In various implementations, a computer program includes, in part or wholly, one or more web applications, one or more mobile applications, one or more standalone applications, one or more web browser plugins, extensions, add-ins, or add-ons, or a combination thereof.

[0056] Unless otherwise defined, technical terms used herein have the same meaning as commonly understood by those skilled in the art to which the subject matter of the invention pertains. As used in this specification and the appended claims, the singular forms 'one', 'one', and 'above' include plural designations unless the context clearly indicates otherwise. Any reference to 'or' in this specification is intended to include 'and / or' unless otherwise specified.

[0057] Numerous embodiments have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the embodiments disclosed herein. Although preferred embodiments of the present disclosure have been illustrated and described herein, it will be apparent to those skilled in the art that such embodiments are provided merely as examples. Numerous changes, variations, and substitutions will now occur to those skilled in the art without departing from the described system. It should be understood that various alternatives to the embodiments described herein may be used to implement the described system.

[0058] Furthermore, the separation or integration of various system modules and components in the aforementioned implementations should not be understood as requiring such separation or integration in all implementations, and it should be understood that the described components and systems may generally be integrated together into a single product or packaged into multiple products. Accordingly, the foregoing description of exemplary implementations does not define or limit the present disclosure. Other modifications, substitutions, and variations are also possible without departing from the spirit and scope of the present disclosure.

[0059] The following paragraphs provide various examples of the implementations disclosed in this specification.

[0060] Clause 1. A method comprising: receiving image data of a space from an imaging device - said space is represented as a model -; determining a difference metric for a part of said space based on said image data and the current state of said model; updating said model to an updated model based on said image data and said difference metric; and generating a virtual representation of said space based on said updated model.

[0061] Clause 2. A method according to Clause 1, further comprising the step of determining the difference metric for the said part of the said space according to a semantic difference calculation method.

[0062] Clause 3. The method of Clause 2, wherein the semantic difference calculation method defines a plurality of metrics for feature points from the part of the space and compares the feature points in the image data with the feature points defined by the model.

[0063] Clause 4. In Clause 3, the semantic difference calculation method compares the feature points within the image data with the feature points defined by the model according to the nearest neighbor segmentation difference.

[0064] Clause 5. A method comprising, in any one of Clauses 1 to 4, further comprising the step of updating the model to an updated model using a subset of the image data based on the difference metric and a threshold, wherein the subset of the image data comprises the portion of the space and is determined according to the difference metric.

[0065] Clause 6. A method in Clause 5, wherein the subset of the image data is incorporated into the updated model according to a weight value that reflects the delta between the difference metric and the threshold.

[0066] Clause 7. A method in which, in Clause 5 or Clause 6, the subset of the image data is determined based on the location and / or radius of the portion of the space.

[0067] Clause 8. A method comprising, in any one of Clauses 1 to 7, further comprising the step of generating the model based on an initial set of image data of the space received from the imaging device.

[0068] Clause 9. A method in which, in any one of Clauses 1 through 8, the model is a neural radiation field (NeRF) model.

[0069] Clause 10. A method comprising, in any one of Clauses 1 to 9, further providing the virtual representation of the space to a user device for display.

[0070] Clause 11. The method of Clause 10, further comprising: identifying incomplete sections of the model; determining a prompt for collecting additional information based on the incomplete sections of the model and the image data; and providing the prompt to the user device.

[0071] Clause 12. The method of Clause 11, further comprising the step of updating the incomplete sections of the model using image data collected from a similar environment.

[0072] Clause 13. A method according to any one of Clauses 10 to 12, wherein the user device comprises an augmented reality (AR) / virtual reality (VR) headset or a mobile device.

[0073] Clause 14. A method in which, in any one of Clauses 10 to 12, the user device comprises the imaging device.

[0074] Clause 15. A method comprising, in any one of Clauses 1 through 14, further a step of generating the virtual representation of the space based on a point cloud or light field determined according to the updated model.

[0075] Clause 16. A method in which, in any one of Clauses 1 through 15, the virtual representation comprises a biocular rendering of the space.

[0076] Clause 17. A method in which, in any one of Clauses 1 through 16, the space comprises a room or partition of a larger environment.

[0077] Clause 18. A system comprising a processor, and a memory storing instructions that, when executed by said processor, cause said system to perform the method of any one of Clauses 1 through 17.

[0078] Clause 19. A non-transient computer-readable medium comprising instructions that, when executed by an electronic processor, cause a computing system to perform the method of any one of Clauses 1 through 17.

Claims

Claim 1 A method comprising: receiving image data of a space represented as a model from an imaging device; determining a difference metric for a part of the space based on the image data and the current state of the model; updating the model to an updated model based on the image data and the difference metric; and generating a virtual representation of the space based on the updated model. Claim 2 A method according to claim 1, further comprising the step of determining the difference metric for the portion of the space according to a semantic difference calculation method. Claim 3 In paragraph 2, the semantic difference calculation method defines a plurality of metrics for feature points from the part of the space and compares the feature points in the image data with the feature points defined by the model. Claim 4 In paragraph 3, the semantic difference calculation method compares the feature points within the image data with the feature points defined by the model according to the nearest neighbor segmentation difference. Claim 5 A method according to any one of claims 1 to 4, further comprising the step of updating the model to an updated model using a subset of the image data based on the difference metric and threshold, wherein the subset of the image data includes the portion of the space and is determined according to the difference metric. Claim 6 A method according to claim 5, wherein the subset of the image data is incorporated into the updated model according to a weight value that reflects the delta between the difference metric and the threshold. Claim 7 A method according to claim 5 or 6, wherein the subset of the image data is determined based on the location and / or radius of the portion of the space. Claim 8 A method according to any one of claims 1 to 7, further comprising the step of generating the model based on an initial set of image data of the space received from the imaging device. Claim 9 A method according to any one of claims 1 to 8, wherein the model is a neural radiation field (NeRF) model. Claim 10 A method comprising, in any one of claims 1 to 9, further providing the virtual representation of the space to a user device for display. Claim 11 A method according to claim 10, further comprising: identifying incomplete sections of the above model; determining a prompt for collecting additional information based on the incomplete sections of the above model and the image data; and providing the prompt to the user device. Claim 12 A method according to claim 11, further comprising the step of updating the incomplete sections of the model using image data collected from a similar environment. Claim 13 A method according to any one of claims 10 to 12, wherein the user device comprises an augmented reality (AR) / virtual reality (VR) headset or a mobile device. Claim 14 A method according to any one of claims 10 to 12, wherein the user device comprises the imaging device. Claim 15 A method comprising, in any one of claims 1 to 14, further a step of generating a virtual representation of the space based on a point cloud or light field determined according to the updated model. Claim 16 A method according to any one of claims 1 to 15, wherein the virtual representation includes a biocular rendering of the space. Claim 17 A method according to any one of claims 1 to 16, wherein the space comprises a room or partition of a larger environment. Claim 18 A system comprising a processor, and a memory storing instructions that, when executed by said processor, cause said system to perform the method of any one of claims 1 to 17. Claim 19 As a non-transient computer-readable medium, A non-transient computer-readable medium comprising instructions that, when executed by an electronic processor, cause a computing system to perform the method of any one of claims 1 through 17.

Images