Computational efficient method for computing a synthesized representation of a 3D environment

Abstract

Methods and apparatus for providing a representation of an environment, for example in an XR system, and any suitable computer vision and robotics applications. The representation of the environment can include one or more planar features. The representation of the environment can be provided by jointly optimizing planar parameters of the planar features and sensor poses observing the planar features. The joint optimization can be based on a reduced matrix and a reduced residual vector instead of a Jacobian matrix and an original residual vector.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority and benefit to the application filed on May 11, 2020, entitled “COMPUTATIONALLY EFFICIENT METHODFOR COMPUTING A COMPOSITE REPRESENTATION OF A 3D ENVIRONMENT”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application generally relates to providing a representation of an environment, for example, in cross-reality (XR) systems, autonomous vehicles, or other computer vision systems that include movable sensors. Background Technology

[0004] Systems that use sensors to acquire information about a 3D environment are used in a variety of scenarios, such as cross-reality (XR) systems or autonomous vehicles. These systems may use sensors, such as cameras, to acquire information about the 3D environment. Sensors can provide numerous observations of the environment that can be integrated into a representation of the environment. For example, when an autonomous vehicle is moving, its sensors can acquire images of the 3D environment from different poses. Each image can provide another observation of the environment, offering more information about a portion of the environment in a previous image or about a new part of the environment.

[0005] Such a system can synthesize a synthetic representation of a 3D environment by combining information obtained from multiple observations. New images can be added to the synthetic representation of the environment, filling in information based on the pose of the image. Initially, the pose of the image can be estimated, for example, based on the output of an internal sensor indicating the motion or relevance of features of the sensor that acquired the image to features already incorporated into the synthetic representation of the environment from previous images. However, errors in these estimation techniques may accumulate over time.

[0006] To compensate for accumulated errors, adjustments to the synthetic representation can be performed periodically. Such adjustments may require reshaping the poses of images already combined into the synthetic representation so that features in different images representing the same object in a 3D environment are better aligned. In some scenarios, the aligned features might be feature points. In other scenarios, the features might be planes based on the content identifiers of the images. In such scenarios, adjustments may need to be made to the relative poses of each image and the relative positions of the planes detected by the synthetic representation.

[0007] Such adjustments can provide an accurate representation of a 3D environment that can be used in any of a variety of ways. In XR systems, for example, a computer can control a human user interface to create cross-reality (XR) environments, where some or all of the XR environment, as perceived by the user, is generated by the computer. These XR environments can be virtual reality (VR), augmented reality (AR), and mixed reality (MR) environments, where some or all of the XR environment may be generated by the computer using data that partially describes the environment. For example, this data could describe virtual objects that can be rendered in a way that the user feels or perceives as part of the physical world, allowing the user to interact with the virtual objects. Because the data is rendered and presented through a user interface device (such as a head-mounted display), the user can experience these virtual objects. The data can be displayed to the user, or it can control audio played to the user, or it can control a touch (or haptic) interface, allowing the user to experience the tactile sensation of the virtual objects as felt or perceived by the user.

[0008] XR systems can be useful for many applications, covering areas such as scientific visualization, medical training, engineering design and prototyping, remote operation and telepresence, and personal entertainment. Compared to VR, AR and MR involve one or more virtual objects associated with real-world objects. The experience of interacting with real objects greatly enhances the user experience of XR systems and opens doors to a variety of applications that provide realistic and easily understood information about how the physical world can be altered.

[0009] To realistically render virtual content, XR systems can construct a representation of the physical world surrounding the user. For example, this representation can be constructed by processing images acquired using sensors on a wearable device that forms part of the XR system. In such a system, the user can perform an initialization routine by looking around the room or other physical environment where they intend to use the XR system, until the system obtains enough information to construct a representation of that environment. As the system operates and the user moves within or to other environments, sensors on the wearable device may acquire additional information to expand or update the representation of the physical world. Summary of the Invention

[0010] Various aspects of this application relate to methods and apparatus for providing representations of an environment, for example, in cross-reality (XR) systems, autonomous vehicles, or other computer vision systems that include movable sensors. The techniques described herein can be used together, individually, or in any suitable combination.

[0011] Some embodiments relate to a method of operating a computing system to generate a representation of an environment. The method includes: obtaining information captured by a sensor, the sensor-captured information including a first number of images; providing an initial representation of the environment, the initial representation including a first number of initial poses and initial parameters based at least in part on a second number of planar features of the first number of images; for each of the second number of planar features in each pose corresponding to one or more observations of the planar features, calculating a matrix indicating the one or more observations of the planar features, and decomposing the matrix into two or more matrices, the two or more matrices including a matrix having simplified rows compared to the first matrix; and calculating a first number of refined poses and refined parameters of the second number of planar features based at least in part on the matrix having simplified rows. The representation of the environment includes the first number of refined poses and refined parameters of the second number of planar features.

[0012] Some embodiments relate to a method of operating a computing system to generate a representation of an environment. The method includes: obtaining information captured by a sensor, the sensor-captured information including a first number of images; providing an initial representation of the environment, the initial representation including a first number of initial poses and initial parameters based at least in part on a second number of planar features of the first number of images; for each of the second number of planar features in each pose corresponding to one or more observations of the planar features, calculating a matrix having a third number of rows, the third number being less than the number of the one or more observations of the planar features; and calculating a first number of refined poses and refined parameters of the second number of planar features based at least in part on the matrix having the third number of rows. The representation of the environment includes the first number of refined poses and refined parameters of the second number of planar features.

[0013] The foregoing overview is provided by way of illustration and is not intended to be limiting. Attached Figure Description

[0014] The accompanying drawings are not intended to be drawn to scale. In the drawings, every identical or nearly identical component shown in each figure is represented by the same numbers. For clarity, not every component may be labeled in every drawing. In the drawings:

[0015] Figure 1 This is a schematic diagram illustrating an example of a simplified augmented reality (AR) scene according to some embodiments.

[0016] Figure 2This is a schematic diagram of an exemplary simplified AR scene illustrating exemplary use cases of an XR system according to some embodiments.

[0017] Figure 3 This is a schematic diagram illustrating the data flow of a single user in an AR system configured to provide a user with an experience of AR content interacting with the physical world, according to some embodiments.

[0018] Figure 4 This is a schematic diagram illustrating an exemplary AR display system for displaying virtual content to a single user according to some embodiments.

[0019] Figure 5A This is a schematic diagram illustrating a user rendering AR content while the user is moving in a physical world environment, according to some embodiments of an AR display system.

[0020] Figure 5B This is a schematic diagram illustrating viewing optical components and accompanying parts according to some embodiments.

[0021] Figure 6A This is a schematic diagram illustrating an AR system using a world reconstruction system according to some embodiments.

[0022] Figure 6B This is a schematic diagram illustrating components of an AR system that maintains a model of a connected world according to some embodiments.

[0023] Figure 7 This is a schematic diagram of a tracing graph formed by a device traversing paths through the physical world, according to some embodiments.

[0024] Figure 8A This is a schematic diagram illustrating observations of various planes in different orientations according to some embodiments.

[0025] Figure 8B This illustrates, according to some embodiments, the calculation including... Figure 8A Observed at the posture Figure 8A A schematic diagram of the geometric entities involved when refining the pose and planar parameters of the plane environment.

[0026] Figure 9 This is a flowchart illustrating a method for representing an environment according to some embodiments.

[0027] Figure 10 This illustrates the use of sensor-captured information according to some embodiments. Figure 9 A schematic diagram of a portion of the representation of the environment calculated using this method. Detailed Implementation

[0028] This document describes methods and apparatuses for providing environmental representations, such as in XR systems, and for any suitable computer vision and robotics applications. The inventors have recognized and understand methods and apparatuses for representing potentially complex environments, such as rooms with many objects therein, and for reducing time, computational costs, and memory usage. In some embodiments, an accurate representation of the environment can be provided by computational techniques that reduce the amount of information processed when adjusting a synthetic representation of a 3D environment to reduce accumulated errors when constructing a synthetic representation based on multiple observations over time. These techniques can be applied to systems that represent environments via a plane, which is detected in a representation of the environment constructed by combining images of the environment acquired at multiple poses. Such techniques can be based on a reduced matrix and a reduced residual vector instead of the Jacobian matrix and the original residual vector.

[0029] The system can utilize computationally lighter processing and / or lower memory usage to provide an accurate representation of the 3D environment with lower latency, lower power consumption, less heat generation, lighter weight, or other advantages. XR systems are an example of systems that can be improved using such techniques. Leveraging low computational complexity, user devices for XR systems can be programmed into smartphones, for example.

[0030] In some embodiments, to provide a realistic XR experience to multiple users, the XR system must be aware of the users' physical environment in order to correctly correlate the position of virtual objects relative to real objects. The XR system can construct a representation of the environment in which virtual objects can be displayed. This representation can be created based on information collected using sensors that are part of the XR device within the XR system. The XR device can be a head-mounted device with integrated displays and sensors, a handheld mobile device (e.g., a smartphone, smartwatch, tablet device, etc.), or other devices with sensors. However, the techniques described herein can be used on other types of devices with sensors, such as autonomous machines with sensors. Such devices may include one or more types of sensors, including, for example, image sensors, LiDAR cameras, RGBD cameras, infrared cameras, structured light sensors, ultrasonic sensors, or coherent light sensors. These devices may include one or more components that aid in collecting sensor information, such as infrared emitters, ultrasonic emitters, or structured light emitters. Alternatively or additionally, such devices may include sensors that aid in determining posture, such as gyroscope sensors, accelerometers, magnetometers, altimeters, proximity sensors, and GPS sensors.

[0031] In some embodiments, the representation of the environment may be a local map of the environment surrounding the XR device, and is created by the XR device by integrating information from one or more images collected while the device is running. In some embodiments, when the device first initiates scanning of the environment (e.g., starting a new session), the coordinate system of the local map may be bound to the device's position and / or orientation. The device's position and / or orientation may change from session to session.

[0032] Local maps may include sparse information that represents the environment based on a subset of features detected in information captured by sensors used in map formation. Additionally or alternatively, local maps may include dense information that represents the environment using information about surfaces in the environment, such as grids.

[0033] A subset of features in a map may include one or more types of features. In some embodiments, the subset of features may include point features, such as table corners that can be detected based on visual information. In some embodiments, the system may alternatively or additionally form a representation of the 3D environment based on higher-level features, such as planar features, such as planes. For example, a plane may be detected by processing depth information. The device may store planar features to replace or supplement feature points, but in some embodiments, storing planar features instead of corresponding point features can reduce the size of the map. For example, a plane may be stored in a map represented by the plane normal and the signed distance to the origin of the map's coordinate system. Conversely, the same structure may be stored as multiple feature points.

[0034] Besides potentially providing input for map creation, the information captured by sensors can also be used to track the movement of devices in their environment. Tracking allows XR systems to localize XR devices by estimating their pose relative to a reference frame established by the map. Localizing an XR device may require comparisons to find matches between a set of measurements extracted from images captured by the XR device and a set of features stored in an existing map.

[0035] This interdependence between map creation and device localization presents a significant challenge. Even with the simplification of representing surfaces using planes in a 3D environment, substantial processing may be required to accurately create the map and simultaneously locate the device. This processing must be completed rapidly as the device moves within the environment. On the other hand, XR devices may have limited computing resources, preventing them from moving within the environment with reasonable speed and flexibility.

[0036] This process may include jointly optimizing planar feature parameters and sensor pose in a map used to form a 3D environment detected in a plane. Conventional techniques for jointly optimizing planar feature parameters and pose can incur high computational costs and memory consumption. Since sensors can record different observations of planar features at different poses, joint optimization according to conventional methods may require solving very large nonlinear least squares problems, even for small-scale workspaces.

[0037] As used herein, “optimization” (and similar terms) does not need to produce a perfect or theoretically optimal solution. Instead, optimization may result from processes that reduce error metrics in the solution. Error reduction processes can be performed until a solution with sufficiently low error is reached. Such processes can be performed iteratively, and iterations can be performed until an exit criterion indicating sufficiently low error is detected. The exit criterion may be an indication that the process has converged to a solution, which can be detected by the percentage of error from iterations below a threshold and / or may be a predetermined number of iterations and / or other criteria indicating that a solution with low error has been identified.

[0038] This paper describes techniques for efficient graph optimization, with an XR system used as an example of a system that can optimize maps using these techniques. In some embodiments, information about each plane in the map is collected in each of a plurality of images acquired from corresponding poses. Observations of each plane can be made at each pose where the image used to create the 3D representation is captured. This information indicating one or more observations of a plane can be formatted as a matrix. In conventional methods of map optimization, this matrix might be a Jacobian matrix, and optimization can be based on calculations performed using that Jacobian matrix.

[0039] According to some embodiments, optimization can be performed based on a decomposition matrix, which may be mathematically related to a matrix used in conventional methods, but may have simplified rows compared to the matrix. The XR system can compute refinement parameters for planar features and refinement poses based on a matrix from the decomposition matrix with simplified rows, which reduces computational cost and memory storage. Refinement parameters for planar features and refinement poses can be computed such that the projection error of the observation of the planar feature from the corresponding refinement pose to the corresponding planar feature is minimized.

[0040] The technologies described herein can be used with or alone in a variety of devices and in a variety of environments, including wearable, portable, or autonomous devices with limited computing resources. In some embodiments, the technologies may be implemented by one or more servers that form part of an XR system.

[0041] Exemplary System

[0042] Figure 1 and Figure 2 A scene is shown that incorporates virtual content displayed as part of the physical world. For illustrative purposes, the AR system is used as an example of an XR system. Figures 3-6B An exemplary AR system is shown, including one or more processors, memory, sensors, and a user interface, which can operate according to the techniques described herein.

[0043] refer to Figure 1 The text describes an outdoor AR scene 354 where AR users see a physical world park-like setting 356 characterized by people, trees, and buildings in the background, as well as a concrete platform 358. In addition to these items, AR users also perceive that they "see" a robot statue 357 standing on the physical world concrete platform 358, and a flying cartoonish avatar 352 that appears to be an incarnation of Bumblebee, even though these elements (e.g., avatar 352 and robot statue 357) do not exist in the physical world. Due to the extreme complexity of human visual perception and the nervous system, producing AR technologies that facilitate a comfortable, natural, and rich presentation of virtual image elements among other virtual or physical world image elements is challenging.

[0044] Such AR scenarios can be realized using a system that constructs a physical world map based on tracking information. This system allows users to place AR content in the physical world, determine the location of the AR content on the physical world map, and save the AR scene so that, for example, the placed AR content can be reloaded to be displayed in the physical world during different AR experience sessions, and multiple users can share the AR experience. The system can construct and update a digital representation of the physical world surface around the user. This representation can be used to render virtual content so that it appears to be completely or partially occluded by physical objects between the user and the rendering location of the virtual content, to place virtual objects in physical-based interactions, and for virtual character path planning and navigation, or for other operations using information about the physical world.

[0045] Figure 2 Another example of an indoor AR scene 400 illustrating an exemplary use case of an XR system according to some embodiments is depicted. Exemplary scene 400 is a living room, featuring walls, a bookshelf on one side of the wall, a floor lamp in a corner of the room, a floor, a sofa, and a coffee table on the floor. In addition to these physical items, the user of the AR technology also perceives virtual objects such as images on the wall behind the sofa, birds flying through the door, a deer peeking out from the bookshelf, and decorations in the form of a windmill placed on the coffee table.

[0046] For images on a wall, AR technology needs information not only about the wall surface but also about objects and surfaces in the room (such as the shape of lamps) to occlude the image for proper rendering of virtual objects. For birds, AR technology needs information about all objects and surfaces around the room to render the bird with realistic physics to avoid objects and surfaces or bounce off them upon collision. For deer, AR technology needs information about surfaces such as the floor or coffee table to calculate where to place the deer. For windmills, the system can identify objects detached from the table and determine if they are movable, while the corner of a bookshelf or wall can be identified as fixed. Such distinctions can be used in determining which parts of the scene are used or updated in each of various operations.

[0047] Virtual objects can be placed within previous AR experience sessions. When a new AR experience session begins in the living room, the AR technology needs to accurately display the virtual object in its previously placed location and make it realistically visible from different viewpoints. For example, a windmill should be displayed as standing on a book, not drifting above a table in different locations without books. Such drifting can occur if the user's location in the new AR experience session is not accurately positioned within the living room. As another example, if the user views the windmill from a different viewpoint than when it was placed, the AR technology needs to correctly display the corresponding side of the windmill.

[0048] A scene can be presented to a user via a system comprising multiple components, including a user interface that can stimulate one or more user senses, such as vision, sound, and / or touch. Additionally, the system may include sensors that can measure parameters of the physical portions of the scene, including the user's position and / or movement within those physical portions. Further, the system may include one or more computing devices with associated computer hardware such as memory. These components may be integrated into a single device or distributed across multiple interconnected devices. In some embodiments, some or all of these components may be integrated into a wearable device.

[0049] Figure 3 An AR system 502, configured to provide an experience of interacting with AR content and the physical world 506 according to some embodiments, is depicted. The AR system 502 may include a display 508. In the illustrated embodiment, the display 508 may be worn by a user as part of a head-mounted device, allowing the user to wear the display over their eyes like goggles or glasses. At least a portion of the display may be transparent, allowing the user to observe a perspective reality 510. Perspective reality 510 may correspond to a portion of the physical world 506 within the current viewpoint of the AR system 502, which may correspond to the user's viewpoint when the user wears a head-mounted device incorporating the AR system's display and sensors to obtain information about the physical world.

[0050] AR content can also be displayed on display 508, overlaid on perspective reality 510. To provide accurate interaction between AR content and perspective reality 510 on display 508, AR system 502 may include sensor 522 configured to capture information about the physical world 506.

[0051] Sensor 522 may include one or more depth sensors that output depth maps 512. Each depth map 512 may have multiple pixels, each of which may represent the distance to a surface in the physical world 506 in a specific direction relative to the depth sensor. Raw depth data may be obtained from the depth sensors to create the depth maps. Such depth maps can be updated as quickly as the depth sensors can form new images, which may be hundreds or thousands of times per second. However, the data may be noisy and incomplete, and holes are shown as black pixels on the illustrated depth map.

[0052] Sensor 522 may include other sensors, such as image sensors. Image sensors can acquire information such as monocular or stereo information, which can be processed to otherwise represent the physical world. In some embodiments, the system may include other sensors, such as one or more of the following: LiDAR cameras, RGBD cameras, infrared camera sensors, visible spectrum camera sensors, structured light emitters and / or sensors, infrared light emitters, coherent light emitters and / or sensors, gyroscope sensors, accelerometers, magnetometers, altimeters, proximity sensors, GPS sensors, ultrasonic transmitters and detectors, and tactile interfaces. Sensor data can be processed in world reconstruction component 516 to create a mesh representing the connected portions of objects in the physical world. Metadata about such objects, including, for example, color and surface texture, can be similarly acquired by the sensors and stored as part of the world reconstruction. Metadata about the location of devices including the system can be determined or inferred based on sensor data. For example, magnetometers, altimeters, GPS sensors, etc., can be used to determine or infer the location of devices. The location of devices can have different degrees of granularity. For example, the accuracy of the device's position determination can vary from coarse (e.g., accurate to a sphere with a diameter of 10 meters), to fine (e.g., accurate to a sphere with a diameter of 3 meters), ultra-fine (e.g., accurate to a sphere with a diameter of 1 meter), to ultra-high precision (e.g., accurate to a sphere with a diameter of 0.5 meters).

[0053] The system can also acquire information about the user's head pose (or "pose") relative to the physical world. In some embodiments, the system's head pose tracking component can be used to calculate the head pose in real time. The head pose tracking component can represent the user's head pose in a coordinate system with six degrees of freedom, including, for example, translations on three vertical axes (e.g., forward / backward, up / down, left / right) and rotations about three vertical axes (e.g., pitch, yaw, and roll). In some embodiments, sensor 522 may include an inertial measurement unit that can be used to calculate and / or determine head pose 514. For example, head pose 514 for a depth map may indicate the current viewpoint of the sensor that captures the depth map with six degrees of freedom, but head pose 515 may be used for other purposes, such as relating image information to a specific part of the physical world, or relating the position of a display worn on the user's head to the physical world.

[0054] In some embodiments, head pose information can be derived in ways other than the IMU, such as from an object in an analyzed image. For example, the head pose tracking component can calculate the relative position and orientation of the AR device to a physical object based on visual information captured by a camera and inertial information captured by the IMU. The head pose tracking component can then calculate the AR device's head pose, for example, by comparing the calculated relative position and orientation of the AR device with a physical object having features of the physical object. In some embodiments, this comparison can be made by identifying features in images captured by one or more sensors in sensor 522 that are stable over time, such that changes in the position of these features in the captured images over time can be correlated with changes in the user's head pose.

[0055] The technology used to operate XR systems can provide XR scenarios for a more immersive user experience. In such systems, XR devices can estimate head pose at a frequency of 1 kHz with low utilization of computing resources. For example, such a device can be configured with four video graphics array (VGA) cameras operating at 30 Hz, an inertial measurement unit (IMU) operating at 1 kHz, the computing power of a single advanced RISC machine core, less than 1 GB of memory, and less than 100 Mbps of network bandwidth. The technology described herein can be used to reduce the processing required to generate and maintain maps and estimate head pose, as well as to provide and use data with low computational overhead. XR systems can calculate their pose based on matched visual features. Hybrid tracking is described in U.S. Patent Application Publication No. 2019 / 0188474 and is incorporated herein by reference in its entirety.

[0056] In some embodiments, as a user moves throughout the physical world with an AR device, the AR device can construct a map from feature points identified in successive images from a series of captured image frames. While each image frame can be acquired from different poses as the user moves, the system can adjust the orientation of features in each successive image frame to match the orientation of the initial image frame by matching features of successive image frames with those of previously captured image frames. Translation of successive image frames, such that points representing the same features will be matched with corresponding feature points from previously collected image frames, can be used to align each successive image frame to match the orientation of previously processed image frames. Frames in the resulting map may have a common orientation established when the first image frame was added to the map. This map, with its set of feature points in a common reference frame, can be used to determine the user's pose in the physical world by matching features from the current image frame to the map. In some embodiments, this map may be referred to as a tracking map.

[0057] Alternatively or additionally, a map of the 3D environment surrounding the user can be constructed by identifying planes or other surfaces based on image information. The positions of these surfaces from image to image can be correlated to create a representation. Techniques for effectively optimizing such a map, as described herein, can be used to form this map. Such a map can be used to locate virtual objects relative to the physical world, or for different or additional functions. For example, a pane-based 3D representation can be used for head pose tracking.

[0058] In addition to enabling the tracking of user poses within the environment, this map allows other components of the system (such as world reconstruction component 516) to determine the position of physical objects relative to the user. World reconstruction component 516 can receive depth maps 512 and head poses 514 from sensors, as well as any other data, and integrate this data into reconstruction 518. Reconstruction 518 may be more complete and less noisy than the sensor data. World reconstruction component 516 can update reconstruction 518 using spatial and temporal averaging of sensor data from multiple viewpoints over time.

[0059] Reconstruction 518 can include a representation of the physical world in one or more data formats, including, for example, voxels, meshes, planes, etc. Different formats can represent alternative representations of the same part of the physical world, or they can represent different parts of the physical world. In the example shown, on the left side of Reconstruction 518, the part of the physical world is rendered as a global surface; on the right side of Reconstruction 518, the part of the physical world is rendered as a mesh.

[0060] In some embodiments, the map maintained by the head pose component 514 may be sparse relative to other maps that may be maintained in the physical world. Instead of providing locations of surfaces and possibly other features, a sparse map may indicate the locations of points of interest and / or structures, such as corners or edges. In some embodiments, the map may include image frames, such as those captured by sensor 522. These frames may be reduced to features that may represent points of interest and / or structures. Information about the pose of the user from which the frame was acquired may also be stored as part of the map, in conjunction with each frame. In some embodiments, each image acquired by the sensor may or may not be stored. In some embodiments, the system may process the images as they are collected by the sensors and select a subset of image frames for further computation. Selection may be based on one or more criteria that limit information but ensure the map contains useful information. The system may add new image frames to the map, for example, based on overlap with previously added image frames, or based on image frames containing a sufficient number of features determined to likely represent stationary objects. In some embodiments, selected image frames or feature sets from selected image frames may be used as keyframes for the map, which can be used to provide spatial information.

[0061] In some embodiments, the amount of data processed when building a map can be reduced, such as by constructing a sparse map with a set of mapping points and keyframes and / or by dividing the map into tiles to enable tile-by-tile updates. Mapping points may be associated with points of interest in the environment. Keyframes may include information selected from data captured from cameras. U.S. Patent Application Publication No. 2020 / 0034624 describes the determination and / or evaluation of localization maps, the entire contents of which are hereby incorporated by reference.

[0062] AR system 502 can integrate sensor data from multiple viewpoints in the physical world over time. The sensor's pose (e.g., position and orientation) can be tracked as the device including the sensor moves. Since the sensor's frame pose is known and how it relates to other poses, each of these multiple viewpoints in the physical world can be fused together into a single composite reconstruction of the physical world, which can be used as an abstraction layer for a map and provides spatial information. By using spatial and temporal averaging (i.e., averaging data from multiple viewpoints over time) or any other suitable method, the reconstruction may be more complete and less noisy than the original sensor data.

[0063] exist Figure 3In the illustrated embodiment, the map represents a portion of the physical world in which a user of a single wearable device resides. In this scenario, the head pose associated with a frame in the map can be represented as a local head pose, indicating an orientation relative to an initial orientation used for a single device at the start of the session. For example, when the device is turned on or otherwise operated to scan the environment to construct a representation of that environment, the head pose can be tracked relative to the initial head pose.

[0064] Combined with content representing that part of the physical world, a map may include metadata. For example, metadata may indicate the time of acquisition of sensor information used to form the map. Metadata may also, or additionally, indicate the location of the sensor at the time of acquiring the information used to form the map. Location may be represented directly, such as with information from a GPS chip, or indirectly, such as by means of a wireless (e.g., Wi-Fi) signature indicating the strength of signals received from one or more wireless access points while sensor data is being collected, and / or by means of an identifier of the wireless access point to which the user equipment is connected at the time sensor data is collected, such as a BSSID.

[0065] Reconstruction 518 can be used for AR functions, such as generating a surface representation of the physical world for occlusion handling or physics-based processing. This surface representation may change as the user moves or objects in the physical world change. Aspects of Reconstruction 518 can be used, for example, by component 520, which generates a varying global surface representation in world coordinates, which can be used by other components.

[0066] AR content can be generated based on this information, such as through AR application 504. For example, AR application 504 could be a game program that performs one or more functions based on information about the physical world, such as visual occlusion, physics-based interaction, and environmental reasoning. It can perform these functions by querying data in different formats from the reconstruction 518 generated by the world reconstruction component 516. In some embodiments, component 520 can be configured to update its output when the representation in the region of interest of the physical world changes. For example, the region of interest can be set to approximate a portion of the physical world near the system user, such as a portion within the user's field of view, or projected (predicted / determined) into the user's field of view.

[0067] AR application 504 can use this information to generate and update AR content. The virtual portion of the AR content can be combined with perspective reality 510 and displayed on display 508 to create a realistic user experience.

[0068] In some embodiments, an AR experience can be provided to a user via an XR device, which may be a wearable display device. This XR device may be part of a system that includes remote processing and / or remote data storage, and / or, in some embodiments, other wearable display devices worn by other users. For simplicity of illustration, Figure 4 An example of a system 580 (hereinafter referred to as "system 580") including a single wearable device is shown. System 580 includes a head-mounted display device 562 (hereinafter referred to as "display device 562"), and various mechanical and electronic modules and systems supporting the functionality of display device 562. Display device 562 may be coupled to a frame 564 that may be worn by a display system user or observer 560 (hereinafter referred to as "user 560") and configured to position display device 562 in front of user 560's eyes. According to various embodiments, display device 562 may be a sequential display. Display device 562 may be monocular or binocular. In some embodiments, display device 562 may be... Figure 3 Example of display 508 in the image.

[0069] In some embodiments, speaker 566 is coupled to frame 564 and positioned adjacent to the ear canal of user 560. In some embodiments, another speaker (not shown) is positioned adjacent to another ear canal of user 560 to provide stereo / shape-shifting sound control. Display device 562 is operatively coupled to local data processing module 570, such as via wired lead or wireless connection 568, which can be mounted in various configurations, such as being fixedly attached to frame 564, fixedly attached to a helmet or hat worn by the user, embedded in headphones, or otherwise detachably attached to user 560 (e.g., in a backpack configuration or a belt-coupled configuration).

[0070] The local data processing module 570 may include a processor and digital memory, such as non-volatile memory (e.g., flash memory), both of which can be used to assist in the processing, caching, and storage of data. The data includes: (a) data captured from or otherwise attached to the user 560 by sensors (which may be operatively coupled to frame 564, for example), such as image capture devices (e.g., cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, radios, and / or gyroscopes; and / or (b) data acquired and / or processed using the remote processing module 572 and / or the remote data repository 574, possibly for transmission to the display device 562 after such processing or retrieval.

[0071] In some embodiments, the wearable device can communicate with remote components. The local data processing module 570 can be operatively coupled to the remote processing module 572 and the remote data repository 574 via communication links 576 and 578 (such as via wired or wireless communication links), such that these remote modules 572 and 574 are operatively coupled to each other and can be used as resources of the local data processing module 570. In a further embodiment, instead of or supplementing the remote data repository 574, the wearable device can access a cloud-based remote data repository and / or server. In some embodiments, the head pose tracking component described above can be at least partially implemented in the local data processing module 570. Figure 3 The world reconstruction component 516 can be implemented at least partially in the local data processing module 570. For example, the local data processing module 570 can be configured to execute computer-executable instructions to generate a map and / or a physical world representation based at least partially on at least a portion of the data.

[0072] In some embodiments, processing can be distributed across local and remote processors. For example, local processing can be used to build a map (e.g., a tracking map) on the user's device based on sensor data collected using sensors on the user's device. Such a map can be used by applications on the user's device. Furthermore, previously created maps (e.g., canonical maps) may be stored in a remote data repository 574. Where suitable stored or persistent maps are available, they can be used to replace or supplement tracking maps created locally on the device. In some embodiments, the tracking map can be mapped to a stored map, such that a correspondence is established between the tracking map and the canonical map, where the tracking map may be oriented relative to the wearable device's location when the user turns on the system, while the canonical map may be oriented relative to one or more persistent features. In some embodiments, persistent maps may be loaded onto the user's device to allow the user's device to render virtual content without latency associated with the scan location to build a tracking map of the user's entire environment based on sensor data acquired during the scan. In some embodiments, the user device can access remote persistent maps (e.g., stored in the cloud) without needing to download persistent maps to the user's device.

[0073] In some embodiments, spatial information can be transferred from the wearable device to a remote service, such as a cloud service configured to localize the device to a stored map maintained on a cloud service. According to one embodiment, location processing can occur in the cloud, matching the device location against an existing map (such as a canonical map) and returning a transformation that links virtual content to the wearable device's location. In such embodiments, the system can avoid transferring maps from a remote resource to the wearable device. Other embodiments can be configured for both device-based and cloud-based localization, for example, to enable functionality where network connectivity is unavailable or the user chooses not to enable cloud-based localization.

[0074] Alternatively or additionally, the tracking map can be merged with previously stored maps to expand or improve the quality of those maps. The process of determining whether a suitable previously created environment map is available and / or merging the tracking map with one or more stored environment maps can be performed in the local data processing module 570 or the remote processing module 572.

[0075] In some embodiments, the local data processing module 570 may include one or more processors (e.g., a graphics processing unit (GPU)) configured to analyze and process data and / or image information. In some embodiments, the local data processing module 570 may include a single processor (e.g., a single-core or multi-core ARM processor), which limits the computational budget of the local data processing model 570 but enables smaller devices. In some embodiments, the world reconstruction component 516 may generate a physical world representation in real time on a non-predefined space using a computational budget smaller than that of a single advanced RISC machine (ARM) core, allowing access to the remaining computational budget of a single ARM core for other purposes, such as mesh extraction.

[0076] The processing of optimized 3D environment maps, as described herein, can be performed on any processor in the system. However, the reduced computation and memory required by the optimizations described herein make it possible to perform such operations quickly and with low latency on a local processor that is part of the wearable device.

[0077] In some embodiments, the remote data repository 574 may include a digital data storage facility, which may be available via the Internet or other network configurations in a “cloud” resource configuration. In some embodiments, all data is stored and all computations are performed in the local data processing module 570, allowing for fully autonomous use from the remote module. In some embodiments, all data is stored and all or most computations are performed in the remote data repository, allowing for smaller devices. For example, world reconstruction may be stored entirely or partially in this repository 574.

[0078] In embodiments where data is remotely stored and accessed via a network, the data can be shared by multiple users of the augmented reality system. For example, user devices can upload their tracking maps to augment the database of environment maps. In some embodiments, tracking map uploads occur at the end of a user session with the wearable device. In some embodiments, tracking map uploads can occur continuously, semi-continuously, intermittently, at predefined times, after a predefined period following a previous upload, or when triggered by an event. Tracking maps uploaded by any user device can be used to expand or improve previously stored maps, whether based on data from that user device or any other user device. Similarly, persistent maps downloaded to a user device may be based on data from that user device or any other user device. In this way, high-quality environment maps may be readily available to users to improve their experience with the AR system.

[0079] In a further embodiment, persistent map downloads can be limited and / or avoided based on localization performed on remote resources (e.g., in the cloud). In such a configuration, the wearable device or other XR device passes feature information coupled to gesture information (e.g., device positioning information when sensing features represented in the feature information) to a cloud service. One or more components of the cloud service can match the feature information with a corresponding stored map (e.g., a canonical map) and generate a transformation between the coordinate systems of the tracking map maintained by the XR device and the canonical map. Each XR device that localizes its tracking map relative to the canonical map can accurately render virtual content relative to the location specified in the canonical map based on its own tracking.

[0080] In some embodiments, the local data processing module 570 is operatively coupled to the battery 582. In some embodiments, the battery 582 is a removable power source, such as a counter battery. In other embodiments, the battery 582 is a lithium-ion battery. In some embodiments, the battery 582 includes an internal lithium-ion battery that can be charged by the user 560 during non-operational periods of the system 580 and a removable battery, allowing the user 560 to operate the system 580 for extended periods without being tied to a power source to charge the lithium-ion battery, or without having to shut down the system 580 to replace the battery.

[0081] Figure 5AThe illustration depicts a user 530 wearing an AR display system that renders AR content as the user 530 moves within a physical world environment 532 (hereinafter referred to as "environment 532"). Information captured by the AR system along the user's movement path can be processed into one or more tracking maps. The user 530 positions the AR display system at location 534, and the AR display system records environmental information of the connected world relative to location 534 (e.g., digital representations of real objects in the physical world, which can store and update changes in real objects in the physical world). This information can be combined with images, features, directional audio input, or other desired data and stored as a pose. Location 534 is aggregated into data input 536, for example, as part of the tracking map, and processed at least by a connected world module 538, which can, for example, be processed by... Figure 4 The processing is performed on the remote processing module 572. In some embodiments, the connected world module 538 may include a head pose component 514 and a world reconstruction component 516, such that the processed information can be combined with other information about physical objects used when rendering virtual content to indicate the location of objects in the physical world.

[0082] The connected world 538 at least partially determines where and how the AR content 540 can be placed in the physical world, as determined from the data input 536. The AR content is “placed” in the physical world by presenting a representation of the physical world and the AR content via a user interface, wherein the AR content is rendered as if it interacts with objects in the physical world, and objects in the physical world are presented as if the AR content obscures the user’s view of these objects when appropriate. In some embodiments, the shape and position of the AR content 540 can be determined by appropriately selecting portions of a fixed element 542 (e.g., a table) from a reconstruction (e.g., reconstruction 518). As an example, the fixed element could be a table, and virtual objects could be positioned such that they appear to be on that table. In some embodiments, the AR content can be placed within a structure in a field of view 544, which can be the current field of view or an estimated future field of view. In some embodiments, the AR content can be persisted relative to a model 546 (e.g., a mesh) of the physical world.

[0083] As depicted, fixed element 542 serves as a proxy (e.g., a digital copy) for any fixed element that may be stored within the connected world module 538 in the physical world, allowing user 530 to perceive content on fixed element 541 without the system needing to map to fixed element 542 each time user 530 views it. Therefore, fixed element 542 can be a mesh model from a previous modeling session, or determined by a single user but still stored by connected world module 538 for future reference by multiple users. Thus, connected world module 538 can recognize environment 532 from a previously mapped environment and display AR content without requiring user 530's device to map all or part of environment 532, saving computational steps and cycles, and avoiding any delays in rendered AR content.

[0084] A physical world mesh model 546 can be created by the AR display system, and appropriate surfaces and metrics for interacting with and displaying AR content 540 can be stored by the connected world module 538 for future retrieval by user 530 or other users without requiring a complete or partial re-creation of the model. In some embodiments, data input 536 is input such as geolocation, user ID, and current activity to indicate to the connected world module 538 which of one or more fixed elements 542 is available, which AR content 540 was last placed in a fixed element 542, and whether that same content is being displayed (such AR content is "persistent" content regardless of whether the user is observing a particular connected world model).

[0085] Even in embodiments where objects are considered fixed (e.g., a dining table), the connected world module 538 can periodically update these objects in the physical world model to account for the possibility of changes in the physical world. The model of fixed objects may be updated very infrequently. Other objects in the physical world may be moving or otherwise not considered fixed (e.g., a kitchen chair). To render a realistic AR scene, the AR system may update the positions of these non-fixed objects at a much higher frequency than it would for updating fixed objects. To accurately track all objects in the physical world, the AR system can acquire information from multiple sensors, including one or more image sensors.

[0086] Figure 5B This is a schematic diagram of the optical assembly 548 and its accompanying components. In some embodiments, two eye-tracking cameras 550, pointed at the user's eye 549, detect measurements of the user's eye 548, such as eye shape, eyelid occlusion, pupil direction, and bright spots on the user's eye 549.

[0087] In some embodiments, one of the sensors may be a depth sensor 551, such as a time-of-flight sensor, that transmits signals to the world and detects reflections of those signals from nearby objects to determine the distance to a given object. For example, the depth sensor can quickly determine whether an object has entered the user's field of view due to the movement of those objects or changes in the user's posture. However, information about the position of objects in the user's field of view can be collected alternatively or additionally by other sensors. For example, depth information can be obtained from a stereo vision image sensor or an all-optical sensor.

[0088] In some embodiments, world camera 552 records a view larger than its periphery to draw and / or otherwise create a model of environment 532 and detect input that may affect AR content. In some embodiments, world camera 552 and / or camera 553 may be grayscale and / or color image sensors that can output grayscale and / or color image frames at fixed time intervals. Camera 553 may further capture images of the physical world within the user's field of view at specific times. Pixels of the frame-based image sensor may be resampled even if their values ​​remain unchanged. Each of world camera 552, camera 553, and depth sensor 551 has a corresponding field of view of 554, 555, and 556 to collect data from and record the physical world scene, such as the physical world environment 532 depicted in FIG. 34A.

[0089] The inertial measurement unit 557 can determine the motion and orientation of the observation optics 548. In some embodiments, the inertial measurement unit 557 can provide an output indicating the direction of gravity. In some embodiments, each component is operatively coupled to at least one other component. For example, the depth sensor 551 is operatively coupled to the eye-tracking camera 550 as confirmation of the measurement adjustment of the actual distance seen by the user's eye 549.

[0090] It should be understood that the observation optics 548 may include some of the components shown in FIG. 34B, and may include components that replace or supplement the components shown. For example, in some embodiments, the observation optics 548 may include two, instead of four, world cameras 552. Alternatively or additionally, cameras 552 and 553 do not need to capture visible light images of their entire field of view. The observation optics 548 may include other types of components. In some embodiments, the observation optics 548 may include one or more dynamic vision sensors (DVS) whose pixels can respond asynchronously to relative changes in light intensity exceeding a threshold.

[0091] In some embodiments, the observation optics 548 may not include a depth sensor 551 based on time-of-flight information. For example, in some embodiments, the observation optics 548 may include one or more all-optical cameras whose pixels can capture light intensity and the angle of incident light, from which depth information can be determined. For example, the all-optical camera may include an image sensor covered with a transmission diffraction mask (TDM). Alternatively or additionally, the all-optical camera may include an image sensor comprising angle-sensitive pixels and / or phase-detection autofocus pixels (PDAF) and / or microlens arrays (MLA). Replacing or supplementing the depth sensor 551, such a sensor can serve as a source of depth information.

[0092] It should also be understood that Figure 5B The configuration of the components is provided as an example. The observation optics 548 may include components with any suitable configuration that can be set to provide the user with the maximum practical field of view for a particular set of components. For example, if the observation optics 548 has a world camera 552, the world camera may be placed in the central region of the observation optics rather than on the side.

[0093] Information from sensors in the observation optics 548 can be coupled to one or more processors in the system. The processors can generate renderable data to allow a user to perceive virtual content interacting with objects in the physical world. This rendering can be implemented in any suitable manner, including generating image data depicting both physical and virtual objects. In other embodiments, physical and virtual content can be depicted in a scene by adjusting the opacity of a display device on which the user views the physical world. The opacity can be controlled to create the appearance of virtual objects and also prevent the user from seeing objects in the physical world occluded by the virtual objects. In some embodiments, the image data may consist only of modifiable virtual content such that, when viewed through a user interface, the virtual content is perceived by the user as realistically interacting with the physical world (e.g., clipping content to explain occlusion).

[0094] The position of the content displayed on the viewing optics 548 to create the impression of an object at a specific location may depend on the physical characteristics of the viewing optics. Furthermore, the user's head posture relative to the physical world and the direction of the user's gaze may affect where the content displayed at a specific location on the viewing optics will appear in the physical world. Sensors, as described above, can collect this information and / or provide information that can be calculated based on it, allowing a processor receiving sensor input to calculate where the object should be rendered on the viewing optics 548 to create the desired appearance for the user.

[0095] Regardless of how content is presented to the user, a model of the physical world can be used to correctly calculate the characteristics of virtual objects that may be affected by physical objects, including the shape, position, motion, and visibility of virtual objects. In some embodiments, the model may include a reconstruction of the physical world, for example, a reconstruction 518.

[0096] The model can be created based on data collected from sensors on a user's wearable device. Although in some embodiments, the model can be created based on data collected by multiple users, which can be aggregated on a computing device remote from all users (and may be "in the cloud").

[0097] The model can be created at least partially by a world reconstruction system, such as... Figure 6A More detailed description Figure 3 The world reconstruction component 516 may include a sensing module 660 that can generate, update, and store representations of a portion of the physical world. In some embodiments, the sensing module 660 may represent a portion of the physical world within the reconstruction range of a sensor as a plurality of voxels. Each voxel may correspond to a 3D cube of a predetermined volume in the physical world and includes surface information indicating the presence of a surface in the volume represented by the voxel. Voxels may be assigned values ​​indicating whether their corresponding volume has been determined to contain a surface including a physical object, whether it has been determined to be empty, or whether it has not yet been measured by a sensor and therefore its value is unknown. It should be understood that the values ​​indicating voxels determined to be empty or unknown do not need to be explicitly stored, as voxel values ​​can be stored in computer memory in any suitable manner, including not storing information for voxels determined to be empty or unknown.

[0098] In addition to generating information for persistent world representation, the perception module 660 can also identify and output indications of changes in the area surrounding the AR system user. Such indications of change can trigger updates to volumetric data stored as part of the persistent world, or trigger other functions, such as triggering the generation of AR content to update the AR content component 604.

[0099] In some embodiments, the sensing module 660 can identify changes based on a signed distance function (SDF) model. The sensing module 660 can be configured to receive sensor data, such as a depth map 660a and a head pose 660b, and then fuse the sensor data into an SDF model 660c. The depth map 660a can directly provide SDF information, and the image can be processed to derive SDF information. SDF information represents the distance to the sensor used to capture this information. Since those sensors can be part of a wearable unit, the SDF information can represent the physical world from the perspective of the wearable unit and therefore from the user's perspective. The head pose 660b allows the SDF information to be correlated with voxels in the physical world.

[0100] In some embodiments, the sensing module 660 can generate, update, and store representations of portions of the physical world within the sensing range. The sensing range can be determined at least in part based on the sensor's reconstructed range, and the reconstructed range can be determined at least in part based on limitations of the sensor's observation range. As a specific example, an active depth sensor operating with active IR pulses can reliably operate over a distance range, thereby creating an observation range for the sensor that can range from a few centimeters or tens of centimeters to several meters.

[0101] The world reconstruction component 516 may include additional modules that can interact with the perception module 660. In some embodiments, the persistent world module 662 may receive a representation of the physical world based on data acquired by the perception module 660. The persistent world module 662 may also include representations of the physical world in various formats. For example, volume metadata 662b of voxels, as well as meshes 662c and planes 662d, may be stored. In some embodiments, other information, such as depth maps, may be stored.

[0102] In some embodiments, the representation of the physical world, such as Figure 6A The representation shown can provide relatively dense information about the physical world, such as a tracking map based on feature points as described above, compared to a sparse map.

[0103] In some embodiments, the perception module 660 may include modules that generate representations for the physical world in various formats, including, for example, a mesh 660d, a plane, and semantics 660e. Representations for the physical world may be stored across local and remote storage media. The representations for the physical world may be described in different coordinate systems depending on, for example, the location of the storage medium. For example, a representation of the physical world stored in a device may be described in a coordinate system local to the device. The representations for the physical world may have pairs stored in the cloud. The pairs in the cloud may be described in a coordinate system shared by all devices in the XR system.

[0104] In some embodiments, these modules may generate a representation based on data within the perception range of one or more sensors at the time of representation generation, as well as data captured over previously time periods and information in the persistent world module 662. In some embodiments, these components may operate on depth information captured by a depth sensor. However, the AR system may include a vision sensor and may generate such a representation by analyzing monocular or binocular visual information.

[0105] In some embodiments, these modules can operate on regions of the physical world. When the sensing module 660 detects a change in the physical world within that sub-region, those modules may be triggered to update that sub-region of the physical world. For example, such a change can be detected by detecting a new surface or other criterion in the SDF model 660c, such as changing the value of a sufficient number of voxels representing the sub-region.

[0106] The world reconstruction component 516 may include component 664 that can receive a representation of the physical world from the perception module 660. Information about the physical world may be extracted by these components, for example, from a user request from an application. In some embodiments, information may be pushed to the user component, such as indications of changes in the representation of the physical world within a pre-identified area or within the perception range. Component 664 may include, for example, a game program and other components that perform processing for visual occlusion, physics-based interaction, and environmental reasoning.

[0107] In response to a query from component 604, perception module 660 can send one or more formats of representation for the physical world. For example, when component 664 indicates that the purpose is for visual occlusion or physical-based interaction, perception module 660 can send a representation of a surface. When component 664 indicates that the purpose is for environmental reasoning, perception module 660 can send a mesh, plane, and semantic representation of the physical world.

[0108] In some embodiments, the sensing module 660 may include a component that formats information to provide to component 664. An example of such a component could be a ray-projection component 660f. For example, using a component (e.g., component 664), information about the physical world can be queried based on a specific viewpoint. The ray-projection component 660f can select one or more representations of physical world data within the field of view from that viewpoint.

[0109] As understood from the foregoing description, the perception module 660 or another component of the AR system can process data to create a 3D representation of a portion of the physical world. The data to be processed can be processed by: culling portions of the 3D reconstructed volume at least in part based on camera frustum and / or depth images; extracting and persisting planar data; capturing, persisting, and updating 3D reconstructed data in blocks that allow local updates while maintaining adjacency consistency; providing occlusion data to applications that generate such scenes, wherein the occlusion data is derived from a combination of one or more depth data sources; and / or performing multi-level mesh simplification. The reconstruction may contain data of varying complexity, including, for example, raw data (such as real-time depth data), fused volumetric data (such as voxels), and computational data (such as meshes).

[0110] In some embodiments, the components of a connected world model can be distributed, with some parts executing locally on the XR device and others executing remotely, such as on a network-connected server or otherwise in the cloud. The allocation of processing and storage of information between the local XR device and the cloud can impact the functionality and user experience of the XR system. For example, reducing processing on the local device by distributing processing to the cloud can enable longer battery life and reduce heat generated on the local device. However, distributing too much processing to the cloud can produce unwanted latency that leads to an unacceptable user experience.

[0111] Figure 6B A distributed component architecture 600 configured for spatial computing according to some embodiments is depicted. The distributed component architecture 600 may include connected world components 602 (e.g., Figure 5A The components include PW 538, Lumin OS 604, API 606, SDK 608, and Application 610. Lumin OS 604 may include a Linux-based kernel with custom drivers compatible with XR devices. API 606 may include an application programming interface that grants access to the spatial computing features of XR devices (e.g., Application 610). SDK 608 may include a software development kit that allows the creation of XR applications.

[0112] One or more components in architecture 600 can create and maintain a model of a connected world. In this example, sensor data is collected on a local device. Processing of this sensor data can be performed partly locally on the XR device and partly in the cloud. PW 538 may include an environment map created at least in part based on data captured by AR devices worn by multiple users. During an AR experience session, individual AR devices (such as those combined above) Figure 4 The wearable device described may create tracking maps, which is one type of map.

[0113] In some embodiments, the device may include components for constructing sparse and dense maps. The tracking map may serve as a sparse map and may include the head pose of the AR device scanning the environment, as well as information about objects detected within that environment at each head pose. Those head poses may be maintained locally for each device. For example, the head pose on each device may be relative to an initial head pose when the device is turned on for its session. Thus, each tracking map may be local to the device that created it and may have its own coordinate system defined by its own local coordinate system. However, in some embodiments, the tracking map on each device may be formed such that a coordinate in its local coordinate system is aligned with the direction of gravity, as measured by its sensors (such as inertial measurement unit 557).

[0114] Dense maps may include surface information, which may be represented by grids or depth information. Alternatively or additionally, dense maps may include higher-level information derived from surface or depth information, such as the location and / or characteristics of planes and / or other objects.

[0115] In some embodiments, the creation of dense maps may be independent of the creation of sparse maps. For example, the creation of dense and sparse maps may be performed in separate processing pipelines within an AR system. For example, separate processing may enable the generation or processing of different types of maps at different rates. For example, a sparse map may be refreshed at a faster rate than a dense map. However, in some embodiments, the processing of dense and sparse maps may be related, even if performed in different pipelines. For example, changes in the physical world revealed in a sparse map may trigger updates to a dense map, and vice versa. Furthermore, even if created independently, the maps may be used together. For example, a coordinate system derived from a sparse map can be used to define the position and / or orientation of objects in a dense map.

[0116] Sparse and / or dense maps can be persisted for reuse by the same device and / or shared with other devices. Such persistence can be achieved by storing the information in the cloud. AR devices can send tracking maps to the cloud for merging, for example, with an environment map selected from previously stored persistent maps in the cloud. In some embodiments, a selected persistent map can be sent from the cloud to the AR device for merging. In some embodiments, persistent maps can be oriented relative to one or more persistent coordinate systems. Such maps can be used as canonical maps because they can be used by any of multiple devices. In some embodiments, a model of a connected world can include one or more canonical maps or be created from one or more canonical maps. Devices, even if they perform some operations based on their device-local coordinate system, can still use the canonical maps by determining the transformation between their device-local coordinate system and the canonical maps.

[0117] A canonical map can originate from a tracking map (TM) (e.g., TM 1102 in Figure 31A), which can be promoted to a canonical map. The canonical map can be persisted so that once the transformation between the local coordinate system of the device accessing the canonical map and the coordinate system of the canonical map is determined, the device accessing the canonical map can use the information in the canonical map to determine the location of objects represented in the canonical map in the physical world around the device. In some embodiments, the TM can be a sparse head pose map created by the XR device. In some embodiments, a canonical map can be created when the XR device sends one or more TMs to a cloud server for merging with additional TMs captured by the XR device at different times or by other XR devices.

[0118] In embodiments where a tracking map is formed on a local device and one coordinate of the local coordinate system is aligned with gravity, this orientation relative to gravity can be preserved when creating the canonical map. For example, a tracking map can be promoted to a canonical map when it is submitted for merging and does not overlap with any previously stored maps. Other tracking maps with orientations relative to gravity may also be subsequently merged with the canonical map. Merging can be performed to ensure that the resulting canonical map retains its orientation relative to gravity. For example, if the coordinates of each gravity-aligned map are not aligned with each other with a sufficiently tight tolerance, the two maps may not be merged, regardless of the correspondence of feature points in those maps.

[0119] Standard maps, or other maps, can provide information about portions of the physical world represented by data processed to create the corresponding maps. Figure 7 An exemplary tracking map 700 according to some embodiments is depicted. The tracking map 700 can provide a plan view 706 of corresponding physical objects in the physical world, represented by points 702. In some embodiments, map points 702 can represent features of physical objects that may include multiple features. For example, each corner of a table can be a feature represented by points on the map. Features can be derived from processed images, such as those acquired using sensors in a wearable device in an augmented reality system. For example, features can be derived by processing image frames output from sensors to identify features based on large gradients or other suitable criteria in the image. Further processing may limit the number of features in each frame. For example, the processing may select features that might represent persistent objects. One or more heuristics may be applied to this selection.

[0120] The tracking map 700 may include data about point 702 collected by the device. For each image frame with data points included in the tracking map, a pose may be stored. The pose may represent the orientation from which the image frame was captured, such that feature points within each image frame can be spatially correlated. The pose can be determined using positioning information, such as that derived from sensors on the wearable device (e.g., IMU sensors). Alternatively or additionally, the pose may be determined by matching the image frame with other image frames depicting overlapping portions of the physical world. The relative pose between two frames can be calculated by finding such positional correlations, which can be accomplished by matching subsets of feature points in the two frames. The relative pose can be adapted to the tracking map because, when the construction of the tracking map is initiated, the map may be relative to a device-local coordinate system established based on the device's initial pose.

[0121] Not all feature points and image frames collected by the device can be retained as part of the tracking map, as much of the information collected by the sensors may be redundant. Instead, only certain frames can be added to the map. Those frames can be selected based on one or more criteria, including, for example, the degree of overlap with image frames already in the map, the number of new features they contain, or a quality metric for the features in the frame. Image frames not added to the tracking map may be discarded or used to modify the location of features. As a further alternative, all or most of the image frames represented as a set of features can be retained, but a subset of those frames can be designated as keyframes for further processing.

[0122] Keyframes can be processed to generate keyframes 704. Keyframes can be processed to generate a set of 3D feature points and saved as keyframes 704. For example, such processing might require comparing image frames simultaneously exported from two cameras to stereoscopically determine the 3D positions of feature points. Metadata can be associated with these keyframes and / or keyframes, such as pose.

[0123] Environmental maps can be in any of a variety of formats, depending on, for example, where they are stored, including, for example, local storage and remote storage on an AR device. For instance, a map in remote storage can have a higher resolution than a map in local storage on a wearable device with limited memory. To send a higher-resolution map from remote storage to local storage, the map can be downsampled or otherwise converted to a suitable format, such as by reducing the number of poses and / or feature points stored for each region of the physical world in the map. In some embodiments, a slice or portion of a high-resolution map from remote storage can be sent to local storage, wherein the slice or portion is not downsampled.

[0124] When a new tracking map is created, the database of environment maps can be updated. To determine which environment among the potentially numerous environment maps in the database will be updated, the update may include effectively selecting one or more environment maps stored in the database associated with the new tracking map. The selected one or more environment maps may be ranked by relevance, and one or more of the highest-ranking maps may be selected for processing to merge the higher-ranking selected environment maps with the new tracking map to create one or more updated environment maps. When the new tracking map represents a part of the physical world that does not exist in a pre-existing environment map to be updated, the tracking map may be stored in the database as a new environment map.

[0125] Techniques for efficiently processing maps with flat surfaces

[0126] Figure 7 A map based on feature points is shown. Some systems can create maps based on identified surfaces (such as planes) rather than individual points. In some embodiments, instead of containing map points 702, the map may contain planes. For example, such a map might contain a plane representing a tabletop, rather than feature points representing the corners of the table and a set of possible points on the surface.

[0127] The process of creating / updating such a map may involve computing features (such as position and orientation) of planes in images acquired from various poses. However, the determination of the pose may depend on the estimated features of previously detected planes. Therefore, creating and updating the map may require jointly optimizing estimated features describing the planes and sensor poses, from which images are captured for estimating the features of the planes. This joint optimization can be referred to as plane bundle adjustment.

[0128] The inventors have recognized and understood that conventional methods for planar bundle adjustment of maps with one or more planar features can lead to large-scale nonlinear least squares problems, resulting in high computational costs and memory consumption. Unlike bundle adjustment typically performed on maps represented by feature points, where a single record from a sensor can produce a single observation of a point feature, planar features can be objects at infinity, and an image from a sensor can provide multiple observations of the planar features. Each observation of a planar feature can constrain the planar parameters of the planar feature and the pose parameters of its pose, leading to large-scale nonlinear least squares problems even for small workspaces.

[0129] The inventors have recognized and understood the planar bundle adjustment that reduces computing costs and memory usage. Figure 8A This is a schematic diagram illustrating observations of various planes in different poses according to some embodiments. In the example shown, there are M planes and N sensor poses. The rotation and translation of the i-th pose are denoted as R. i ∈SO3 and ti ∈R 3 Suppose that the j-th plane has a parameter as π. j =[n j ;d j ], where n is the plane normal with ||n||² = 1, and d is the negative distance from the origin of the coordinate system to the plane. The measurement results of the j-th plane at the i-th pose are a set of K defined as follows. ij Points:

[0130]

[0131] Each p ijk ∈P ij Constraints are provided for the i-th pose and the j-th plane, such as... Figure 8B As shown. Residual δ ijk From p ijk to plane π j The signed distance can be written as:

[0132] δ ijk =n j ·(R i p ijk +t i )+d j (7)

[0133] With t i Different, rotation R i and plane parameter π j It has additional constraints. For example, R i It can be parameterized using quaternions, angle axes, or Euler angles. Plane parameter π j It can be represented by homogeneous coordinates, nearest-point parameterization, or minimal parameterization based on quaternions. Regardless of the specific parameterization, this example defines θ. i →R(θ i ) and ω j →π(ω j This allows for arbitrary parameterization of rotational and planar parameters. θ i and t i Related to the sensor pose, it can be combined into ρ i =[θ i ;t i ]. ρ i It can have 6 or 7 unknowns that change with respect to the parameterization of the rotation matrix (7 for quaternions and 6 for the minimal representation of the rotation, such as angle axis parameterization). ω j It can have 3 or 4 unknowns (3 for the smallest representation of the plane, and 4 for the homogeneous coordinates of the plane). Using these notations, the residual δ ijk It is ρi and ω j The function.

[0134] Planar bundle adjustment may be achieved by jointly refining all ρ by minimizing the following nonlinear least squares problem. i (i≠1) and ω j Question:

[0135]

[0136] Here, the first pose ρ1 is fixed during optimization to rigidly anchor the coordinate system.

[0137] The Jacobian matrix of the plane bundle adjustment can be computed and provided to algorithms for solving least squares problems (e.g., the Levenberg-Marquardt (LM) algorithm and the Gaussian-Newton algorithm). The observations of the j-th plane at the i-th pose are the point set P. ij It can be derived from the point set P. ij Derive the Jacobian matrix J ij The Jacobian matrix J can be any J ij The stacking. Assume that in P ij K exists in ij One point. J ij (i≠1) has the following form:

[0138]

[0139] δ ijk , and The partial derivatives can be based on the residual δ in (7). ijk It can be calculated in the form of . It can be defined as follows:

[0140]

[0141] It should be noted that R as defined above i The element is θ i The function, and d j and n j The element is ω j The function. p ijk It can represent the k-th measurement result of the j-th plane at the i-th pose. jik y ijk and z ijk p can represent ijk Position in a coordinate system, which can be a sensor coordinate system, a device coordinate system, or a standard coordinate system. Residual δ ijk This can be calculated by substituting (10) into (7) and extending it. This yields:

[0142]

[0143] (11) can be written as:

[0144] δ ijk =c ijk ·v ij (12)

[0145] Among them, c ijk and v ij It is a 13-dimensional vector as follows:

[0146] c ijk =[x ijk y ijk , z ijk x ijk y ijk , z ijk x ijk y ijk , z ijk ,1,1,1,1] T ,

[0147]

[0148] c ijk The elements in the p are from observation p ijk Or 1, which is a constant. On the other hand, v ij The element in is ρ i and ω j The function.

[0149] δ ijk The partial derivatives can be obtained by assuming ρ i Having n ρ One unknown and ω j Having n ω It is calculated using several unknowns. It can be defined as:

[0150]

[0151] Assume ζ ij d It is ζ ij The d-th element. According to (12), δ ijk Relative to ζ ij d The partial derivatives have the following form:

[0152]

[0153] in, It is a 13-dimensional vector whose elements are v ij The element relative to ξ ijd The partial derivatives. According to (15), It has the following form:

[0154]

[0155] V ρi It can be a 13×6 or 13×7 matrix (13×7 for quaternions, and 13×6 for the smallest representation of a rotation matrix). Similarly, It has the following form:

[0156]

[0157] V ωj It can be a 13×3 or 13×4 matrix (13×3 is used for the minimum representation of the plane, and 13×4 is used for the homogeneous coordinates of the plane).

[0158] Jacobian matrix J ij It can be calculated based on the following:

[0159]

[0160] Where, the k-th row c ijk It is defined in (13). C ij It is the size K ij A 13×13 matrix. Jacobian matrix J ij By substituting (16) and (17) into (9) and using C in (18) ij The definition of calculation is as follows. This produces the following result:

[0161]

[0162] Assume P 1j It comes from the j-th plane ω in the first pose ρ1. j The measurement results. Since ρ1 may be fixed during optimization, therefore, from P 1j The derived Jacobian matrix J 1j It can have the following forms:

[0163]

[0164] C ij It can be rewritten in the following form:

[0165] C ij =Q ij M ij (21) Among them, M ij It has a size of 4×13 and Q T ij Q ij=I4, where I4 is a 4×4 identity matrix. For example, c in (13) ijk As defined, x ijk y ijk z ijk And 1 is copied several times to form c ijk Therefore, in C ij Of the 13 columns, only 4 are separate columns, which contain the constants 1 and P. ij The x, y, and z coordinates of a point within the space. It is represented as:

[0166]

[0167] C ij Column 13 is E ij A copy of the four columns in E. ij It can be decomposed into:

[0168] E ij =Q ij U ij (23) Where Q T ij Q ij =I4, and U ij It is an upper triangular matrix. Q ij Having size K ij ×4, and U ij It has a size of 4×4. Fine QR decomposition can be used because point K... ij The number is usually much greater than 4.

[0169] Fine QR decomposition can reduce computation time. ij It can be divided according to its columns as follows:

[0170]

[0171] Substituting (24) into (22) produces the following result:

[0172]

[0173] Comparing (22) and (23) yields the following results:

[0174]

[0175]

[0176] Because of C ij The column is E ij A copy of the column, according to c in (13) ijk The form and E in (22) ij Definition, C ij It can be written as:

[0177] C ij =[x ij y ij , z ij x ij y ij , z ij x ij y ij , z ij ,1,1,1,1] (27)

[0178] Substituting (26) into (27) produces the following result:

[0179]

[0180] C ij Factorization can be used to significantly reduce computational costs. Although a fine QR decomposition is described in the example, other decomposition methods can be used, including, for example, singular value decomposition (SVD).

[0181] It can be based on C ij Calculate J ij The reduced Jacobian matrix J r ij C ij It can be decomposed into C ij =Q ij M ij As described above.

[0182]

[0183] Reduced Jacobian matrix J r ij Having a Jacobian matrix J ij Fewer lines, because M ij It is more than C ij A much smaller matrix. C ij Having size K ij ×13. Conversely, M ij It has a size of 4×13. Typically, K ij Much greater than 4.

[0184] J ij and J r ij They can be superimposed to form the Jacobian matrix J and the reduced-order Jacobian matrix J shown below for the cost function (8). r :

[0185]

[0186] In algorithms for solving least squares problems, Jr can replace J to calculate J.T J. For planar beam adjustment, J T J = J rT J r J and J r It is based on J as defined in (30) ij and J r ij The block vector. According to block matrix multiplication:

[0187]

[0188] For i≠1, using the expression in (19), J T ij J ij It has the following form:

[0189]

[0190] Similarly, using the expression in (29), J r ij T J r ij It has the following form:

[0191]

[0192] Substitute (21) into And use fact Q T ij Q ij =I4,

[0193]

[0194] Similarly,

[0195]

[0196]

[0197]

[0198] For i = 1, according to (20), for J T 1j J 1j The only nonzero term is V. T ωj C T 1j C 1j V ωj On the other hand, according to (29), J r 1j T J r1j It has only one corresponding non-zero term V T ωj C T 1j C 1j V ωj Similar to the derivation in (34), we obtain:

[0199]

[0200] In summary, using (34), (35), and (36), J T ij J ij =J r ij T J r ij According to (31), therefore J T J = J rT J r .

[0201] P ij K in ij The residual vector at each point can be defined as: According to (12), δ ij It can be written as:

[0202] δ ij =C ij v ij (37)

[0203] δ ij The reduced residual vector δ r ij It can be defined as:

[0204]

[0205] Superimpose all δ ij and δ r ij The residual vector δ and the reduced residual vector δ r It has the following form:

[0206]

[0207] Reduced residual vector δ r It can replace the residual vector δ in algorithms for solving least squares problems. For planar bundle adjustment, j T δ=J rT δ r J, J r , δ and δ r These are elements J as defined in (30) and (39), respectively.ij J r ij δ ij and δ r ij Block vectors. Applying block matrix multiplication:

[0208]

[0209] For i≠1, use J in (19) ij J in (29) r ij The representation of , and δ in (37) ij and δ in (38) r ij The expression, J T ij δ ij and J r ij T δ r ij It has the following form:

[0210]

[0211] Substitute (21) into And use fact Q T ij Q ij =I4,

[0212]

[0213] Similarly,

[0214]

[0215] For i=1, substitute (20) and (37) into J. T 1j δ 1j And apply block matrix multiplication, J T 1j δ 1j The only nonzero term is V. T ω1j C T 1j C 1j v 1j On the other hand, substituting (29) and (38) into J r 1j T δ r 1j J r 1j T δr 1j It has only one non-zero term V T ω 1j M T 1j M 1j v1 j Similar to the derivation in (42),

[0216]

[0217] In summary, according to (42), (43) and (44), J T ij δ ij =J r ij T δ ij According to (40), J T δ=J rT δ.

[0218] It should be understood that in some embodiments, a system using a reduced-order Jacobian matrix can compute the reduced-order Jacobian matrix based on sensor data or other information, without forming a Jacobian matrix or repeatedly computing it to achieve the reduced matrix form described herein.

[0219] For planar bundle adjustment, the reduced-order Jacobian matrix Jr and the reduced residual vector δ r J and δ in (4) can be used to calculate the step size in the algorithm for solving the least squares problem, and J r and δ r Each block J in r ij and δ ij r has 4 rows. The algorithm for solving the least squares problem uses (4) to calculate the step size for each iteration. Since J T J = J rT J r J T δ=J rT δ, (J rT J r +λI)ξ=J rT δ equals (J) T J+λI)ξ=J T δ. Therefore, J r and δ r It can replace J and δ in calculating the step size in algorithms for solving least squares problems. According to J in (29) r ij and δ in (38) r ij Definition, J rij and δ r ij The number of rows and M ij The number of rows is the same, M ij It has 4 rows. Therefore, J r and δ r It has 4 rows. Therefore, no matter how many points are in P... ij In the middle, simplified J r ij and δ r ij It has a maximum of 4 rows. This significantly reduces the computational cost of algorithms for solving least squares problems. The additional cost here is the computation of C. ij =Q ij M ij Because of C ij It remains constant during iteration, so it is only computed once before the iteration.

[0220] In some embodiments, planar beam adjustment can obtain initial predictions of N poses and M planar parameters, as well as measurement results {P}. ij Planar beam adjustment can be applied to each p. ijk ∈P ij Compute matrix block c ijk And stack the matrix blocks to C ij As (16). Planar beam adjustment can make C ij Decomposed into an orthogonal matrix Q ij and the upper triangular matrix M ij As (19). Upper triangular matrix M ij Algorithms for solving least squares problems (e.g., the Levenberg-Marquardt (LM) algorithm and the Gaussian-Newton algorithm) can be provided to compute the reduced-order Jacobian matrix block J in (27). r ij The reduced residual block δ in (36) r ij These blocks can be stacked to form a reduced-order Jacobian matrix J. r As the (28) reduced residual vector δ r As (37). The algorithm for solving the least squares problem can compute the refined pose and plane parameters until convergence.

[0221] In some embodiments, planar features can be combined with other features such as point features in 3D reconstruction. In the combined cost function derived from multiple features, the Jacobian matrix from the planar cost has the form J as in (28), and the residual vector also has the form δ as in (37). Therefore, in bundle adjustment with mixed features, the reduced-order Jacobian matrix J in (28) rThe reduced residual vector δ in (37) can replace the Jacobian matrix and the original residual vector.

[0222] The representation of the environment can be calculated using planar bundle adjustment. Figure 9 This is a flowchart illustrating a method 900 for providing an environment according to some embodiments. Figure 10 This is a schematic diagram illustrating a portion 1000 of a representation of an environment calculated using method 900 based on sensor-captured information according to some embodiments.

[0223] Method 900 can begin by obtaining (action 902) information captured by one or more sensors in a corresponding pose. In some embodiments, the obtained information may be a visual image and / or a depth image. In some embodiments, the image may be a keyframe of a combination of multiple images. Figure 10 The example illustrates three sensors: sensor 0, sensor 1, and sensor 2. In some embodiments, the sensors may belong to a device that may have a device coordinate system with an origin O. The device coordinate system may represent the position and orientation when the device first initiates a scan of the environment used for the session. Each sensor may have a corresponding sensor coordinate system. Each sensor may capture images taken in a corresponding pose ρ. i-1 ρ i and ρ i+1 One or more images, such as images 1002A, 1002B, and 1002C. In some embodiments, the coordinate system with origin O may represent a coordinate system shared by one or more devices. For example, the sensor may belong to three devices sharing a coordinate system.

[0224] Method 900 may include providing a first representation of the environment (action 904). The first representation may include initial predictions of feature parameters for features extracted from information captured by sensors and initial predictions of corresponding poses. Figure 10 In the example, a first plane 1004 is observed in images 1002A and 1002B. The first plane 1004 can be defined by the plane parameter π. j =[n j ;d j [This is represented by an image]. Images 1002A and 1002B respectively observe multiple points P in plane 1004. (i-1)j and P ij A portion of the first plane 1004 can be observed through images 1002A and 1002B. A portion of the first plane 1004 can also be observed through either image 1002A or image 1002B alone. The second plane 1006 is observed in images 1002B and 1002C. The second plane 1006 can be determined by the plane parameter π. j+1 =[n (j+1) ;d (j+1)[This is represented by an image]. Images 1002B and 1002C respectively observe multiple points P in plane 1006. i(j+1) and P (i+1)j A portion of the second plane 1006 can be observed solely through images 1002B and 1002C. A portion of the second plane 1006 can be observed solely through image 1002B or solely through image 1002C. Point features 1008 are also observed in images 1002B and 1002C. The first representation may include information for pose ρ. i-1 ρ i and ρ i+1 The initial prediction, the plane parameters for the first plane 1004 and the second plane 1006, and the point feature parameters for the point feature 1008.

[0225] Initial predictions can be based on images from which features are first observed. For Figure 10 Example, posture ρ i-1 The initial prediction of the planar parameters for the first plane 1004 can be based on image 1002A. Pose ρ i The initial predictions for the planar parameters of the second plane 1006 and the feature parameters of the point features 1008 can be based on image 1002B. Pose ρ i+1 The initial prediction can be based on image 1002C.

[0226] As features are observed from subsequent images, the initial prediction can be refined to reduce drift and improve rendering quality. Method 900 may include actions 906 and 908 to compute a refined pose and refined feature parameters, which may include refined plane parameters and refined point feature parameters. Action 906 may be performed for each plane in the corresponding pose. Figure 10 In the examples, respectively for those in pose ρ i-1 and ρ i The first plane 1004 executes action 906. Also, respectively for those in posture ρ i and ρ i+1 The second plane 1006 performs action 906. Action 906 may include calculating a matrix (e.g., C) (action 906A). ij As (16), the matrix indicates the corresponding plane (e.g., plane j) in the corresponding pose (e.g., pose i). Action 906 may include decomposing the matrix (action 906B) into two or more matrices, including a matrix with simplified rows compared to the matrix (e.g., an upper triangular matrix M). ij As (19)).

[0227] Method 900 may include providing (action 910) a second representation of the environment having refined pose and refined feature parameters. Method 900 may include determining (action 912) whether new information has been observed, such that actions 902 to 910 should be repeated.

[0228] Therefore, while some aspects of certain embodiments have been described, it should be understood that various changes, modifications, and improvements are readily apparent to those skilled in the art.

[0229] As an example, embodiments are described in conjunction with augmented reality (AR) environments. It should be understood that some or all of the techniques described herein can be applied in MR environments, or more generally in other XR environments, in VR environments, and in any other computer vision and robotics applications.

[0230] As another example, embodiments are described in conjunction with devices such as wearable devices. It should be understood that some or all of the technologies described herein can be implemented via networks (such as the cloud), discrete applications and / or devices, and any suitable combination of networks and discrete applications.

[0231] Such changes, modifications, and improvements are intended to be part of this disclosure and are intended to be within the spirit and scope of this disclosure. Furthermore, while advantages of this disclosure have been indicated, it should be understood that not every embodiment of this disclosure will include every described advantage. Some embodiments may not implement any features described herein and in some instances as advantages. Therefore, the foregoing description and figures are by way of example only.

[0232] The embodiments described above in this disclosure can be implemented in any of a variety of ways. For example, embodiments can be implemented using hardware, software, or a combination thereof. When implemented in software, the software code can execute on any suitable processor or set of processors, whether provided in a single computer or distributed among multiple computers. Such a processor can be implemented as an integrated circuit, wherein one or more processors in the integrated circuit components include commercially available integrated circuit components nominally known in the art, such as CPU chips, GPU chips, microprocessors, microcontrollers, or coprocessors. In some embodiments, the processor can be implemented in custom circuitry, such as an ASIC, or in semi-custom circuitry resulting from configuring a programmable logic device. As yet another alternative, the processor can be part of a larger circuitry or semiconductor device, whether commercially available, semi-custom, or custom. As a specific example, some commercially available microprocessors have multiple cores, such that one or a subset of those cores can constitute a processor. However, the processor can be implemented using circuitry of any suitable format.

[0233] Furthermore, it should be understood that a computer can be implemented in any of a variety of forms, such as a rack-mount computer, desktop computer, laptop computer, or tablet computer. In addition, a computer can be embedded in a device that is not typically considered a computer but has suitable processing capabilities, including a personal digital assistant (PDA), a smartphone, or any other suitable portable or fixed electronic device.

[0234] Furthermore, a computer may have one or more input and output devices. Among other things, these devices can be used to present a user interface. Examples of output devices that can be used to provide a user interface include a printer or display screen for visual presentation of output, and a speaker or other sound-generating device for auditory presentation of output. Examples of input devices that can be used for a user interface include a keyboard, and pointing devices such as a mouse, touchpad, and digitizing tablet. As another example, a computer may receive input information via speech recognition or in other audio formats. In the illustrated embodiments, the input / output devices are shown as physically separate from the computing device. However, in some embodiments, the input and / or output devices may be physically integrated into the same unit as the processor or other components of the computing device. For example, the keyboard may be implemented as a soft keyboard on a touchscreen. In some embodiments, the input / output devices may be completely disconnected from the computing device and functionally integrated via a wireless connection.

[0235] Such computers can be interconnected in any suitable form via one or more networks, including local area networks (LANs) or wide area networks (WANs), such as corporate networks or the Internet. Such networks can be based on any suitable technology and can operate according to any suitable protocol, and may include wireless networks, wired networks, or fiber optic networks.

[0236] Furthermore, the various methods or processes outlined in this paper can be encoded as software that can be executed on one or more processors employing any of a variety of operating systems or platforms. Moreover, such software can be written using any of many suitable programming languages ​​and / or programming or scripting tools, and can also be compiled into executable machine language code or intermediate code that executes on a framework or virtual machine.

[0237] In this respect, the present disclosure may be implemented as a computer-readable storage medium (or multiple computer-readable media) (e.g., computer memory, one or more floppy disks, optical discs (CDs), optical discs, digital video discs (DVDs), magnetic tape, flash memory, circuit configurations in field-programmable gate arrays or other semiconductor devices, or other tangible computer storage media) encoding one or more programs that, when executed by one or more computers or other processors, perform methods implementing the various embodiments of the present disclosure discussed above. As apparent from the foregoing examples, a computer-readable storage medium may retain information for a sufficient time to provide computer-executable instructions in a non-transitory form. Such a computer-readable storage medium or media may be transportable, such that one or more programs stored thereon may be loaded onto one or more different computers or other processors to implement aspects of the present disclosure discussed above. As used herein, the term "computer-readable storage medium" covers only computer-readable media that can be considered an article of manufacture (i.e., an article of manufacture) or a machine. In some embodiments, the present disclosure may be implemented as a computer-readable medium rather than a readable computer storage medium, such as for propagating signals.

[0238] The terms "program" or "software" are used in a general sense herein to refer to any type of computer code or set of computer-executable instructions that can be used to program a computer or other processor to implement the aspects of this disclosure as discussed above. Furthermore, it should be understood that, according to one aspect of this embodiment, one or more computer programs that perform the methods of this disclosure during execution do not need to reside on a single computer or processor, but can be distributed in a modular manner across multiple different computers or processors to implement the aspects of this disclosure.

[0239] Computer-executable instructions can take many forms, such as program modules, and are executed by one or more computers or other devices. Typically, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. Generally, in various embodiments, the functionality of program modules can be combined or distributed as desired.

[0240] Furthermore, data structures can be stored in any suitable form on a computer-readable medium. For ease of illustration, a data structure might be shown as having fields related by position within the data structure. Such relationships can also be implemented by allocating storage to fields that has a position in the computer-readable medium that conveys the relationship between the fields. However, any suitable mechanism can be used to establish relationships between information in the fields of a data structure, including by using pointers, tags, or other mechanisms that establish relationships between data elements.

[0241] The aspects of this disclosure can be used individually, in combination, or in various arrangements not specifically discussed in the foregoing embodiments, and therefore their application is not limited to the details and arrangements of the components set forth in the foregoing description or shown in the accompanying drawings. For example, an aspect described in one embodiment can be combined in any way with aspects described in other embodiments.

[0242] Furthermore, this disclosure can be implemented as a method for which examples have been provided. Actions performed as part of a method can be ordered in any suitable manner. Therefore, embodiments in which actions are performed in a different order than those shown can be constructed, which may include the simultaneous execution of some actions, even if shown as sequential actions in the exemplary embodiments.

[0243] The use of ordinal terms such as “first,” “second,” “third,” etc., to modify a claim element does not imply any priority, order of precedence, or sequence of actions of a method of execution relative to another claim element. Rather, it serves only as a label to distinguish one claim element with a certain name from another element with the same name (but using ordinal terms) to differentiate claim elements.

[0244] Furthermore, the wording and terminology used herein are for descriptive purposes and should not be considered restrictive. The use of “comprising,” “including,” “having,” “containing,” “involving,” and variations thereof in this document is intended to cover the items listed herein and their equivalents, as well as any additional items.

Claims

1. A method for operating a computing system to generate a representation of an environment, the method comprising: Obtain information captured by a sensor, the information captured by the sensor including a first number of images; Provide an initial representation of the environment, the initial representation including a first number of initial poses and initial parameters based at least in part on a second number of planar features of the first number of images; For each of the second number of planar features in each pose corresponding to one or more observed images including the planar features, Calculate the Jacobian matrix of the one or more observations indicating the planar features, and The Jacobian matrix is ​​decomposed into two or more matrices, the two or more matrices including a matrix with simplified rows compared to the Jacobian matrix; as well as By providing the matrix with simplified rows to an algorithm for solving the least squares problem, the first number of refinement poses and refinement parameters of the second number of planar features are calculated. The representation of the environment includes the first number of refined poses of the second number of planar features and the refined parameters.

2. The method according to claim 1, wherein, For each of the second number of planar features in each pose corresponding to one or more views of the image including the planar features, decomposing the Jacobian matrix into two or more matrices includes: Calculate the orthogonal matrix and the upper triangular matrix.

3. The method according to claim 2, wherein, The first number of refinement poses and refinement parameters for calculating the second number of planar features are at least partially based on the upper triangular matrix.

4. The method according to claim 1, wherein, By providing the matrix with simplified rows to an algorithm for solving a least-squares problem, the calculation of the first number of refinement poses and refinement parameters for the second number of planar features includes: The matrix block is computed at least in part based on the matrix having simplified rows.

5. The method according to claim 4, wherein, By providing the matrix with simplified rows to an algorithm for solving a least-squares problem, the calculation of the first number of refinement poses and refinement parameters for the second number of planar features includes: The matrix blocks are combined to form a reduced-order matrix.

6. The method according to claim 5, wherein, By providing the matrix with simplified rows to an algorithm for solving a least-squares problem, the calculation of the first number of refinement poses and refinement parameters for the second number of planar features includes: The reduced-order matrix is ​​provided to the algorithm for solving the least squares problem.

7. The method according to any one of claims 1 to 6, wherein, By providing the matrix with simplified rows to an algorithm for solving a least-squares problem, the calculation of the first number of refinement poses and refinement parameters for the second number of planar features includes: The block vector is computed based at least in part on the matrix having simplified rows.

8. The method according to claim 7, wherein, By providing the matrix with simplified rows to an algorithm for solving a least-squares problem, the calculation of the first number of refinement poses and refinement parameters for the second number of planar features includes: The block vectors are combined to form a reduced vector.

9. The method according to claim 8, wherein, By providing the matrix with simplified rows to an algorithm for solving a least-squares problem, the calculation of the first number of refinement poses and refinement parameters for the second number of planar features includes: The reduced vector is provided to the algorithm for solving the least squares problem.

10. The method according to claim 1, wherein, For each of the second number of planar features in each pose corresponding to one or more observations of the planar features, calculating the Jacobian matrix indicating the one or more observations of the planar features includes: For each of the one or more observations of the planar feature, calculate the matrix block indicating the observation, and The matrix blocks are combined into a matrix that indicates the one or more observations of the planar features.

11. A non-transitory computer-readable medium storing computer-executable instructions configured to, when executed by at least one processor, perform a method of operating a computing system to generate a representation of an environment, the method comprising: Obtain information captured by a sensor, the information captured by the sensor including a first number of images; Provide an initial representation of the environment, the initial representation including a first number of initial poses and initial parameters based at least in part on a second number of planar features of the first number of images; For each of the second number of planar features in each pose corresponding to one or more observed images including the planar features, Calculate the Jacobian matrix of the one or more observations that indicate the planar features; The Jacobian matrix is ​​decomposed into two or more matrices, the two or more matrices including a matrix with simplified rows compared to the Jacobian matrix; as well as By providing the matrix with the simplified rows to an algorithm for solving the least squares problem, the first number of refinement poses and refinement parameters of the second number of planar features are calculated. The representation of the environment includes the first number of refined poses of the second number of planar features and the refined parameters.

12. The non-transient computer-readable medium according to claim 11, wherein, Decomposing the matrix into two or more matrices includes: calculating the orthogonal matrix and the upper triangular matrix.

13. The non-transient computer-readable medium according to any one of claims 11 to 12, wherein, By providing the matrix with simplified rows to an algorithm for solving a least-squares problem, the calculation of the first number of refinement poses and refinement parameters for the second number of planar features includes: The matrix block is computed at least in part based on the matrix having simplified rows.

14. The non-transient computer-readable medium according to any one of claims 11 to 12, wherein, By providing the matrix with simplified rows to an algorithm for solving a least-squares problem, the calculation of the first number of refinement poses and refinement parameters for the second number of planar features includes: The matrix blocks are combined to form a reduced-order matrix.

15. The non-transient computer-readable medium according to claim 14, wherein, By providing the matrix with simplified rows to an algorithm for solving a least-squares problem, the calculation of the first number of refinement poses and refinement parameters for the second number of planar features includes: The reduced-order matrix is ​​provided to the algorithm for solving the least squares problem.

16. The non-transient computer-readable medium according to any one of claims 11 to 12, wherein, By providing the matrix with simplified rows to an algorithm for solving a least-squares problem, the calculation of the first number of refinement poses and refinement parameters for the second number of planar features includes: The block vector is computed based at least in part on the matrix having simplified rows.

17. The non-transient computer-readable medium according to claim 16, wherein, By providing the matrix with simplified rows to an algorithm for solving a least-squares problem, the calculation of the first number of refinement poses and refinement parameters for the second number of planar features includes: The block vectors are combined to form a reduced vector.

18. The non-transient computer-readable medium according to claim 17, wherein, By providing the matrix with simplified rows to an algorithm for solving a least-squares problem, the calculation of the first number of refinement poses and refinement parameters for the second number of planar features includes: The reduced vector is provided to the algorithm for solving the least squares problem.

19. The non-transient computer-readable medium according to claim 11, wherein, For each of the second number of planar features in each pose corresponding to one or more observations of the planar features, calculating the Jacobian matrix indicating the one or more observations of the planar features includes: For each of the one or more observations of the planar feature, calculate the matrix block indicating the observation, and The matrix blocks are combined into a matrix that indicates the one or more observations of the planar features.

Images