System and method for map construction
Through parallel processing and asynchronous parallel processing technology, the system delay problem existing in multi-component map construction and synchronization of AR/MR systems is solved, efficient map construction and synchronization is achieved, and system performance is improved.
Patent Information
- Application Number
- CN202210699679.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-07-23
- Filing Date
- 2019-07-23
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2039-07-23
AI Technical Summary
Existing AR/MR systems have system delay problems during the construction and synchronization of cross-multi-component maps, resulting in performance degradation. Especially under the limitations of system such as processing capabilities, memory, bandwidth and data sources, it is difficult to achieve efficient map construction and synchronous.
Using parallel processing and asynchronous parallel processing technology, by formatting map data into changes and utilizing the order of change, system components are allowed to process in parallel, reduce system delays, and minimize component downtime. At the same time, using independently transferable and manipulated map subunits, data transmission is optimized, bandwidth-constrained channels are adapted, and the sorting and application of state changes is achieved through the map manager and writer reader.
It effectively reduces system delay, improves the efficiency of map construction and synchronization, reduces component downtime, and improves the real-time performance and consistency of AR/MR systems.
Smart Images

Figure CN115097627B_ABST
Abstract
Description
[0001] This application is a divisional application of the Chinese patent application with application number 201980049058.6. The application date of the original application is July 23, 2019, the priority date is July 23, 2018, the date of entering the Chinese national phase is January 22, 2021, and the name of the invention is “System and method for map construction”. Technical Field
[0002] The present disclosure relates to a multi-component map building and map building synchronization system, and a method of building and synchronizing map building across multi-component maps using the system. Background Art
[0003] Modern computing and display technologies have facilitated the development of "mixed reality" (MR) systems for so-called "virtual reality" (VR) or "augmented reality" (AR) experiences, in which digitally reproduced images, or portions thereof, are presented to a user in a manner that appears to be, or can be perceived to be, real. VR scenes typically involve the presentation of digital or virtual image information, but are opaque to actual real-world visual input. AR scenes typically involve the presentation of digital or virtual image information as an enhancement to the visualization of the real world around the user (i.e., transparency to the real-world visual input). Thus, AR scenes involve the presentation of digital or virtual image information with transparency to real-world visual input.
[0004] MR systems typically generate and display color data, which increases the realism of the MR scene. Many of these MR systems display color data by sequentially projecting sub-images of different (e.g., primary) colors or "fields" (e.g., red, green, and blue) corresponding to a color image in rapid succession. Projecting the color sub-images at a sufficiently high rate (e.g., 60 Hz, 120 Hz, etc.) may convey a smooth color MR scene in the user's mind.
[0005] Various optical systems generate images, including color images, at various depths to display MR (VR and AR) scenes. Some such optical systems are described in U.S. Utility Patent Application Serial No. 14 / 555,585, filed on November 27, 2014, the contents of which are expressly and entirely incorporated herein by reference as if fully set forth herein.
[0006] A head-mounted display device implementing AR / MR provides simultaneous viewing of real and virtual objects. With an "optical see-through" display, a user can see through transparent (or translucent) elements in the display system to directly view light from real objects in the environment. The transparent element (often called a "combiner") superimposes light from the display on the user's real-world view, where the light from the display projects an image of the virtual content into a see-through view of real objects in the environment. A camera can be mounted on the head-mounted display device to capture images or video of the scene the user is viewing.
[0007] Current optical systems, such as those in AR / MR systems, render virtual content optically. The content is "virtual" because it does not correspond to a real physical object located at a corresponding location in space. Instead, the virtual content exists only in the brain (e.g., optical center) of the user of the head-mounted display device when stimulated by a beam of light directed toward the user's eyes.
[0008] Some AR / MR systems use maps of real-world environments (e.g., topological maps, geometric maps, sparse point maps, etc.) to generate a traversable world model for AR / MR scenes. The traversable world model facilitates interactions involving real-world objects in the real-world environment, such as (e.g., displaying virtual objects on a real-world sofa). Sparse maps are basic building blocks for calculating visual postures relative to a "stable" coordinate system. Environmental features become reference points in a 3D sparse map. The position of the reference point relative to the sensor defines the position or posture of the sensor. Sparse maps can be built using data from multiple system components. Sparse maps allow tracking to facilitate the permanence of virtual objects ("pixel sticks") in the real-world environment. Some methods related to AR / MR map construction involve positioning AR / MR devices using topological maps to determine the position and posture of users and / or AR / MR devices. Some methods related to AR / MR map construction involve using geometric maps to model real-world objects of AR / MR devices to allow more accurate and realistic presentation of virtual objects. Some AR / MR systems use simultaneous localization and mapping techniques to generate and / or update (possibly dynamically) a map of the real-world environment.
[0009] As an example, positioning allows a user wearing a head mounted display device to view virtual objects on a display while walking around the area where the virtual objects appear. Using the updated position and pose data, virtual objects can be rendered for each viewpoint, giving the user the perception that they are walking around objects that occupy real space. If the head mounted display device is used to present multiple virtual objects, measurements of head pose can be used to render the scene to match the user's dynamically changing position and pose and provide an enhanced sense of immersion.
[0010] Many methods and techniques can be used to generate, modify and / or optimize maps. Some of these techniques involve new data. One group of such new data techniques is collectively referred to as tracking, and includes updating map point statistics, which may trigger map point deletions. Another new data technique is visual inertial odometry ("VIO"), which includes applying inertial constraints to a sparse point graph. Another group of new data techniques is collectively referred to as "map building", which includes inserting new keyrig frames ("keyframes" or "rigframes"), map points, and observations. Another group of new data techniques is collectively referred to as relocalization, which includes adding new orb features and bag-of-words ("BoW") indexes. Another group of new data techniques is collectively referred to as map merging, which includes merging sparse map data from different AR / MR sessions. Another group of new data techniques is collectively referred to as joint dense optimization, which includes merging dense map data.
[0011] Other map building techniques involve optimization of existing maps. One group of such optimization techniques is collectively referred to as bundle adjustment, which includes modifying sparse map geometry and online calibration. Another group of optimization techniques is collectively referred to as loop closure, which includes modifying the geometry of sparse maps.
[0012] Various methods and techniques related to AR / MR map construction are described in U.S. utility patent applications Serial No. 14 / 705,983 filed on May 7, 2015 (Attorney Docket No. ML.200V7.310) and Serial No. 14 / 704,844 filed on May 5, 2015 (Attorney Docket No. ML.20020.304), the contents of which are expressly and entirely incorporated herein by reference as if fully set forth herein.
[0013] As described above, map building facilitates AR / MR system functionality. Synchronizing maps across components of a multi-component AR / MR system (e.g., a head-mounted display ("wearable") and a belt- or body-worn processing component ("Power Pack") facilitates a coherent and consistent AR / MR scene. Map synchronization involves certain system limitations, such as processing power, memory, bandwidth, data sources, latency, and multiple operating systems, processors, and architectures. Summary of the invention
[0014] In one embodiment, a computer-implemented method for updating a point map on a system having first and second hardware components that are communicatively coupled includes: a first component performs a first process on a point map in a first state to generate a first change. The method also includes a second component performing a second process on the point map in the first state to generate a second change. The method further includes the second component applying the second change to the point map in the first state to generate a first updated point map in the second state. In addition, the method includes the first component sending the first change to the second component. In addition, the method includes the second component applying the first change to the first updated point map in the second state to generate a second updated point map in a third state.
[0015] In one or more embodiments, the method further includes the second component sending the second change to the first component. The method further includes: the first component applies the second change to the dot graph in the first state to generate a first updated dot graph in the second state. In addition, the method includes: the first component applies the first change to the first updated dot graph in the second state to generate a second updated dot graph in the third state. The method may also include: the first component generates a change order based on the first change and the second change, and the first component applies the second change to the dot graph in the first state before applying the first change to the updated dot graph in the second state based on the change order. The first component applying the second change to the dot graph in the first state may include: the first component generates a pre-changed first state. The first component applying the first change to the first updated dot graph in the second state may include: the first component generates a pre-changed second state. The second component applying the second change to the dot graph in the first state may include: the second component generates a pre-changed first state, and the second component applying the first change to the first updated dot graph in the second state may include: the second component generates a pre-changed second state.
[0016] In one or more embodiments, the method also includes the first component generating a change order based on the first change and the second change, and the first component sending the change order to the second component. The method may also include: the second component applies the second change to the point map in the first state before applying the first change to the updated point map in the second state based on the change order. The first processing can be selected from the group consisting of: tracking, map point deletion, VIO, map construction, insertion of new keyframes / key binding frames, insertion of new map points, insertion of new observations, bundle adjustment, modification of the geometry of sparse maps, online calibration, repositioning, adding new orb features; adding new BoW indexes; loop closure; modifying the geometry of sparse maps; map merging; and joint dense optimization.
[0017] In one or more embodiments, the spot map corresponds to a physical location where the system is located. The physical location may be a room or a cell of a room. The spot map may be a portion of a larger spot map.
[0018] In one or more embodiments, the first component is a head mounted audiovisual display component of the mixed reality system, and the second component is a torso mounted processing component of the mixed reality system. The method may also include the second component reconciling the first change and the second change before applying the second change to the point map in the first state.
[0019] In one or more embodiments, the first component and the second component are communicatively coupled to the third component. The method also includes the third component performing a third process on the point map in the first state to generate a third change. The method further includes the first component and the second component sending the first change and the second change to the third component, respectively. In addition, the method includes: the third component applies the third change to the second updated point map in the third state to generate a third updated point map in a fourth state. The first component can be a head-mounted audio-visual display component of the mixed reality system, the second component can be a torso-mounted processing component of the mixed reality system, and the third component can be a cloud-based processing component. The method can also include the second component receiving the first change from the first component before generating the second change. The second component can include a removable medium for data storage.
[0020] In another embodiment, a mixed reality system includes a head mounted audio-visual display component and a torso mounted processing component communicatively coupled to the display component. The display component is configured to perform a first process on a point map in a first state to generate a first change, and send the first change to the processing component. The processing component is configured to perform a second process on the point map in the first state to generate a second change, apply the second change to the point map in the first state to generate a first updated point map in the second state, and apply the first change to the first updated point map in the second state to generate a second updated point map in a third state.
[0021] In one or more embodiments, the processing component is further configured to send a second change to the display component. The display component is also configured to apply the second change to the dot graph in the first state to generate a first updated dot graph in the second state, and apply the first change to the first updated dot graph in the second state to generate a second updated dot graph in the third state. The display component may also be configured to generate a change order based on the first change and the second change, and apply the second change to the dot graph in the first state before applying the first change to the updated dot graph in the second state based on the change order.
[0022] In one or more embodiments, the display component is further configured to generate a change order based on the first change and the second change, and send the change order to the processing component. The processing component may also be configured to apply the second change to the point map in the first state before applying the first change to the updated point map in the second state based on the change order. The first processing can be selected from the group consisting of: tracking, map point deletion, VIO, map construction, insertion of new keyframes / key binding frames, insertion of new map points, insertion of new observations, bundle adjustment, modification of the geometry of sparse maps, online calibration, repositioning, adding new orb features; adding new BoW indexes; loop closure; modifying the geometry of sparse maps; map merging; and joint dense optimization.
[0023] In one or more embodiments, the spot map corresponds to a physical location where the system is located. The physical location may be a room or a cell of a room. The spot map may be part of a larger spot map.
[0024] In one or more embodiments, the processing component is further configured to reconcile the first change and the second change before applying the second change to the dot graph in the first state. The display and processing component can be communicatively coupled to the online component. The display and processing component can be configured to send the first change and the second change to the online component, respectively. The online component can be configured to perform a third process on the dot graph in the first state to generate a third change, and apply the third change to the second updated dot graph in the third state to generate a third updated dot graph in a fourth state.
[0025] In one or more embodiments, the display component includes a display component map manager, which is configured to generate a change order, apply changes based on the change order, and send the change order to the processing component. The display component also includes a display component change writer, which is configured to send a first change to the processing component. The display component further includes a display component change reader configured to receive the change. The display component map manager can also be configured to generate a pre-change state. The display component map manager can include a change buffer.
[0026] In one or more embodiments, the processing component includes a processing component map manager, which is configured to receive the change order and apply the change based on the change order. The processing component also includes a processing component change writer, which is configured to send a second change to the display component. The processing component further includes a processing component change reader configured to receive the change. The processing component map manager may also be configured to generate a pre-change state. The processing component map manager may include a change buffer. The processing component may include a removable medium for data storage.
[0027] In another embodiment, a computer-implemented method for updating a point map on a system having a head-mounted audiovisual display assembly and a torso-mounted processing assembly communicatively coupled to the display assembly includes: the processing assembly dividing the point map into a plurality of point map subunits. The method also includes: the processing assembly sending a first subset of the plurality of point map subunits to the display assembly. The method further includes: in response to movement of the display assembly, the processing assembly sending a second subset of the plurality of point map subunits to the display assembly.
[0028] In one or more embodiments, the method further includes: the display component stores a first subset of the plurality of dot graph subunits; the display component detects motion thereof; and the display component sends motion data to the processing component. The method further includes: the display component deletes a portion of the first subset of the plurality of dot graph subunits based on the motion; the display component receives a second subset of the plurality of dot graph subunits; and the display component stores the second subset of the plurality of dot graph subunits. The portion of the first subset of the plurality of dot graph subunits and the second subset of the plurality of dot graph subunits may have substantially the same size.
[0029] In one or more embodiments, a second subset of the plurality of dot pattern subunits corresponds to a direction of motion, and a portion of the first subset of the plurality of dot pattern subunits corresponds to a second direction opposite to the direction of motion. The plurality of dot pattern subunits may be a 3D array. The 3D array may be a 3×3×1 array, a 10×10×1 array, or a 6×6×3 array.
[0030] In another embodiment, a mixed reality system includes a head-mounted audiovisual display assembly and a torso-mounted processing assembly communicatively coupled to the display assembly. The processing assembly is configured to divide a dot map into a plurality of dot map subunits, send a first subset of the plurality of dot map subunits to the display assembly, and send a second subset of the plurality of dot map subunits to the display assembly in response to movement of the display assembly. The display assembly is configured to store the first subset of the plurality of dot map subunits, detect movement thereof, send movement data to the processing assembly, delete a portion of the first subset of the plurality of dot map subunits based on the movement, receive a second subset of the plurality of dot map subunits, and store the second subset of the plurality of dot map subunits.
[0031] In one or more embodiments, a portion of a first subset of the plurality of dot pattern subunits and a second subset of the plurality of dot pattern subunits have substantially the same size. The second subset of the plurality of dot pattern subunits may correspond to a direction of motion, and a portion of the first subset of the plurality of dot pattern subunits may correspond to a second direction opposite to the direction of motion. The plurality of dot pattern subunits may be a 3D array. The 3D array may be a 3×3×1 array, a 10×10×1 array, or a 6×6×3 array.
[0032] In another embodiment, a system for communicating data across a USB cable includes a writer, a reader, and a data structure including an array of cells. Each cell of the array includes an owner indicator, a status indicator, and cell data.
[0033] In one or more embodiments, the cell array includes a plurality of map cells and at least one transition cell. The system may also include a cell list including a plurality of pointers corresponding to each cell of the map and a count of pointers or cells of the map.
[0034] In another embodiment, a method for communicating data across a USB cable using a system including a writer, a reader, and a data structure including an array of cells, each cell of the array including an owner indicator, a status indicator, and cell data, the method including initializing the data structure by setting all owner indicators to "no owner" and setting all status indicators to "INVALD". The method also includes identifying cells to be transmitted across the USB cable. The method further includes setting the owner indicator of the cell to "comm2". In addition, the method includes setting the status indicator of the cell to "USB". In addition, the method includes transmitting the cell across the USB cable. The method also includes setting the owner indicator of the cell to "MAP" after transmitting the cell across the USB cable. The method further includes setting the status indicator of the cell to "MAP" after transmitting the cell across the USB cable.
[0035] In one or more embodiments, the method further includes the system reading a cell list including a plurality of pointers corresponding to each cell having an owner indicator set to "MAP" and a count of pointers to a plurality of cells having an owner indicator set to "MAP". The method may also include writing a fill request array. The method may further include invalidating a cell by setting the owner indicator of the cell to "no owner" and setting the status indicator of the cell to "INVALD". In addition, the method may include sending a cell by setting the owner indicator of the cell to "comm2" and setting the status indicator of the cell to "USB".
[0036] Additional and other objects, features, and advantages of the present disclosure are described in the detailed description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] The accompanying drawings illustrate the design and utility of various embodiments of the present disclosure. It should be noted that the drawings are not drawn to scale and that elements of similar structure or function are represented by similar reference numerals throughout the drawings. In order to better understand how to obtain the above and other advantages and purposes of the various embodiments of the present disclosure, the present disclosure briefly described above will be described in more detail by reference to specific embodiments of the present disclosure shown in the accompanying drawings. Understanding that these drawings depict only typical embodiments of the present disclosure and therefore should not be considered as limiting the scope thereof, the present disclosure will be described and explained with additional specificity and detail through the use of the accompanying drawings, in which:
[0038] Figure 1-4 A VR / AR / MR system and its subsystems according to some embodiments are schematically depicted.
[0039] Figure 5 Depicted is a focal plane for a VR / AR / MR system according to some embodiments.
[0040] Figure 6 Depicted is a simplified view of a traversable world model in accordance with some embodiments.
[0041] Figure 7 A method of rendering using a traversable world model according to some embodiments is described.
[0042] Figure 8 A topology diagram is depicted in accordance with some embodiments.
[0043] Fig. 9 Depicted is a high-level flow diagram of a positioning process using a topology map in accordance with some embodiments.
[0044] Fig.10 Depicted is a geometric graph that is a connection between various keyframes according to some embodiments.
[0045] Fig.11 Depicted is a topology graph layered on top of a geometry graph in accordance with some embodiments.
[0046] Fig.12 Depicted is a high-level flow diagram of a process for performing wave propagation beam adjustment in accordance with some embodiments.
[0047] Fig.13 Depicted are a map point, a keyframe, and a rendered line from the map point to the keyframe as seen through a virtual keyframe in accordance with some embodiments.
[0048] Fig.14 Depicted is a high-level flow diagram of a process for finding a map point based on rendering rather than searching, according to some embodiments.
[0049] Fig.15A map building system including multiple AR / MR systems according to some embodiments is schematically depicted.
[0050] Fig.16 Depicted is a workflow for an asynchronous map building method using a multi-component map building system according to some embodiments.
[0051] 17A to 17D Several exemplary loop operations are depicted in accordance with some embodiments.
[0052] Fig.18 Depicted is a workflow for an asynchronous map building method using a multi-component map building system with map management according to some embodiments.
[0053] Fig.19 An AR / MR system for map building and map synchronization according to some embodiments is schematically depicted.
[0054] Fig. 20 A wearable map manager for an AR / MR system for map building and map synchronization generating a sequence of changes according to some embodiments is schematically depicted.
[0055] Fig.21 A powered package map manager for an AR / MR system for map building and map synchronization is schematically depicted applying a plurality of received state changes based on an order of changes in accordance with some embodiments.
[0056] Fig. 22 A wearable map manager for an AR / MR system for map building and map synchronization is schematically depicted that processes a plurality of received state changes and generates an order of changes in accordance with some embodiments.
[0057] Fig.23 A powered package map manager for an AR / MR system for map building and map synchronization is schematically depicted processing a plurality of received state changes based on an order of changes in accordance with some embodiments.
[0058] Fig.24 and Fig.25 A method of cell streaming in a multi-component map building system according to some embodiments is schematically depicted.
[0059] Fig.26 and Fig. 27 Depicted are data structures for zero-copy data transfer in accordance with some embodiments.
[0060] Fig.28 is a block diagram that schematically depicts an illustrative computing system in accordance with some embodiments. DETAILED DESCRIPTION
[0061] Various embodiments of the present disclosure are directed to systems, methods and articles of manufacture for map building and map synchronization in a multi-component system in a single embodiment or in multiple embodiments.Other objects, features and advantages of the present disclosure are described in the detailed description, drawings and claims.
[0062] Various embodiments will now be described in detail with reference to the accompanying drawings, which are provided as illustrative examples of the present disclosure so that those skilled in the art can practice the present disclosure. It should be noted that the following drawings and examples are not meant to limit the scope of the present disclosure. In the case where known components (or methods or processes) can be used to partially or completely implement certain elements of the present disclosure, only those parts of these known components (or methods or processes) necessary for understanding the present disclosure will be described, and the detailed description of other parts of such known components (or methods or processes) will be omitted, so as not to obscure the present disclosure. In addition, various embodiments encompass the current and future known equivalents of the components referred to herein by way of example.
[0063] The head mounted audio-visual display system and the user's physical contact modulation between the system can be implemented independently of the AR / MR system, but some embodiments are described below with respect to the AR / MR system for illustrative purposes only. The contact modulation system described herein can also be used in the same manner as a VR system.
[0064] Summary of Problems and Solutions
[0065] Map building is fundamental to many AR / MR system functions, and map synchronization between components across a multi-component AR / MR system helps achieve stable map building of coherent and consistent AR / MR scenes. System limitations (such as processing power, memory, bandwidth, data sources, component latency, and multiple operating systems, processors, and architectures) can cause overall system latency during map building, which can negatively impact AR / MR system performance.
[0066] For example, in some wearable devices, tracking consumes a significant portion of the map data and associated processing, and map building generates the majority of the map data produced on the wearable devices. In addition, some wearable forward-facing cameras ("FOVs") provide data for map building. Therefore, it is preferred to run latency-critical jobs and jobs that use FOV camera data on wearable devices. However, wearable devices have limited memory and processing budgets / capabilities. For example, many wearable devices cannot store an entire sparse point map for a large real-world area / space. Similarly, many wearable devices cannot run large geometry optimizations (e.g., global bundle adjustment, loop closures, etc.) in a practical amount of time.
[0067] A power pack typically has more memory (including some permanent memory, such as an SD card) and processing power than a wearable device. Therefore, the power pack has access to more map data (including sparse point map data for large real-world areas and dense point map data for limited real-world areas) and has the processing power to run large geometric optimizations (e.g., global bundle adjustment, loop closure, etc.). The power pack may even have enough processing power to run optimizations involving dense point map data. However, map data can be transmitted from the wearable device to the power pack via a link (e.g., USB), and updated maps can be transmitted from the power pack to the wearable device via a link. Links such as USB may have limited bandwidth that may result in unacceptable system latency. For example, it is preferred to perform latency-sensitive operations (e.g., operations related to head posture) at the wearable device without having to be processed at the power pack.
[0068] Some AR / MR systems also include cloud-based storage and processing components. The cloud component has access to a large amount of persistent storage and processing power. The cloud component can also access sparse and dense point maps of the same real-world area / environment generated earlier by different systems or the same system. Data from different systems can be stored as map grids on the cloud, allowing larger-scale optimizations. Therefore, the cloud component can perform processing-intensive optimizations that involve sparse point and dense point map data from multiple sessions. However, the network connection between the cloud component and the power pack and wearable device is not always available. Therefore, it is preferred to perform latency-sensitive operations (e.g., operations related to head posture) with the participation of the cloud component. In some embodiments, a fourth layer of map construction may exist on top of the cloud component layer.
[0069] In addition, these AR / MR system components involve different platforms, architectures, and organizational structures. For example, these system components (wearable devices, power packs, clouds) can have different processor architectures, memory hierarchies, and storage models (e.g., Myriad / Streaming Hybrid Architecture Vector Engine (“SHAVE”) for wearable devices, NVidia / ARM for power packs, PCs for clouds). System components can also have different framework environments / operating systems (e.g., minotaur for wearable devices, malleus for power packs, various OSs for clouds). In addition, system components can have different code bases, release processes, etc. (e.g., especially between wearable devices and power packs to the cloud). These component differences can further complicate map synchronization.
[0070] Embodiments described herein include a map data processing pipeline that allows parallel processing of all system components by formatting processing results as changes and utilizing a change order. The processing pipeline reduces system latency and allows system components to perform optimally matched jobs while minimizing downtime of system components while waiting for serial processing of map data. Embodiments also include a map data format / structure that includes independently transferable and manipulable map subunits. The map data format / structure allows a wearable device to operate on a subset of map subunits while a power pack stores multiple map subunits that define a sparse point map of a real-world area. Embodiments further include a data format / structure for transmitting data across a bandwidth-constrained channel that includes ownership and status information. The map data format / structure allows more efficient data transmission across such bandwidth-constrained channels (e.g., USB).
[0071] Illustrative VR, AR and / or MR Systems
[0072] The following description relates to illustrative VR, AR, and / or MR systems that can be practiced with contact modulation systems. However, it should be understood that the embodiments are also applicable to applications in other types of display systems (including other types of VR, AR, and / or MR systems), and therefore, the embodiments are not limited to the illustrative systems disclosed herein.
[0073] exist Figures 1 to 4 Components of a VR, AR, and / or MR virtual image system 100 are depicted in FIG. The virtual image generation system 100 includes a frame structure 102 worn by an end user 50; a display subsystem 110 carried by the frame structure 102 so that the display subsystem 110 is positioned in front of the eyes of the end user 50; and a speaker 106 carried by the frame structure 102 so that the speaker 106 is positioned adjacent to the ear canal of the end user 50 (optionally, another speaker (not shown) is positioned adjacent to another ear canal of the end user 50 to provide stereo / shapeable sound control). The display subsystem 110 is designed to present a light pattern to the eyes of the end user 50 that can be comfortably perceived as an enhancement of physical reality with a high level of image quality and three-dimensional perception, and is capable of presenting two-dimensional content. The display subsystem 110 presents a series of frames at a high frequency that provide a perception of a single coherent scene.
[0074] In the illustrated embodiment, the display subsystem 110 employs an "optical see-through" display through which a user can directly view light from real objects via a transparent (or translucent) element. The transparent element (often referred to as a "combiner") superimposes the light from the display on the user's view of the real world. To this end, the display subsystem 110 includes a partially transparent display. The display is positioned in the field of view of the end user 50 between the eyes of the end user 50 and the surrounding environment, so that direct light from the surrounding environment is transmitted through the display to the eyes of the end user 50.
[0075] In the illustrated embodiment, the image projection assembly provides light to a partially transparent display to be combined with direct light from the surrounding environment and transmitted from the display to the eyes of the user 50. The projection subsystem can be a fiber scanning based projection device, and the display can be a waveguide based display, into which the scanned light from the projection subsystem is injected to produce, for example, an image at a single optical viewing distance closer than infinity (e.g., arm's length), images at multiple discrete optical viewing distances or focal planes, and / or image layers stacked at multiple viewing distances or focal planes to represent a volumetric 3D object. The layers in the light field can be stacked closely enough together to appear continuous to the human visual subsystem (i.e., one layer is within the confusion cone of an adjacent layer). Additionally or alternatively, picture elements can be blended across two or more layers to increase the perceived continuity of transitions between layers in the light field, even if the layers are more sparsely stacked (i.e., one layer is outside the confusion cone of an adjacent layer). The display subsystem 110 can be monocular or binocular.
[0076] The virtual image generation system 100 may also include one or more sensors (not shown) mounted on the frame structure 102 to detect the position and movement of the head 54 of the end user 50 and / or the position and inter-eye distance of the eyes of the end user 50. Such sensors may include image capture devices (such as cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, radios, and / or gyroscopes. Many of these sensors operate under the assumption that the frame 102 to which the sensors are fixed is substantially fixed to the user's head, eyes, and ears in that order.
[0077] The virtual image generation system 100 may also include a user orientation detection module. The user orientation module detects the instantaneous position of the head 54 of the end user 50 (e.g., via a sensor coupled to the framework 102), and may predict the position of the head 54 of the end user 50 based on the position data received from the sensor. Detecting the instantaneous position of the head 54 of the end user 50 helps determine the specific real object that the end user 50 is looking at, thereby providing an indication of a specific virtual object to be generated relative to the real object, and further providing an indication of the position of the virtual object display. The user orientation module may also track the eyes of the end user 50 based on the tracking data received from the sensor.
[0078] The virtual image generation system 100 may also include a control subsystem, which may take any of a variety of forms. The control subsystem includes a plurality of controllers, such as one or more microcontrollers, microprocessors or central processing units (CPUs), digital signal processors, graphics processing units (GPUs), other integrated circuit controllers, such as application specific integrated circuits (ASICs), programmable gate arrays (PGAs) (e.g., field PGAs (FPGAs)), and / or programmable logic controllers (PLUs).
[0079] The control subsystem of the virtual image generation system 100 may include a central processing unit (CPU), a graphics processing unit (GPU), one or more frame buffers, and a three-dimensional database for storing three-dimensional scene data. The CPU may control the overall operation, while the GPU may render frames (i.e., convert the three-dimensional scene into a two-dimensional image) based on the three-dimensional data stored in the three-dimensional database and store the frames in the frame buffer. One or more additional integrated circuits may control the reading in and / or out of frames from the frame buffer and the operation of the image projection component of the display subsystem 110.
[0080] The various processing components of the virtual image generation system 100 may be physically contained in distributed subsystems. Figures 1 to 4 As shown in , the virtual image generation system 100 may include a local processing and data module 130 that is operably coupled to the display subsystem 110 and the sensor, such as by wired leads or wireless connections 136. The local processing and data module 130 may be mounted in a variety of configurations, such as fixedly attached to the frame structure 102 ( Figure 1 ), fixedly attached to a helmet or hat 56 ( Figure 2 ), removably attached to the torso 58 of the end user 50 ( Figure 3 ), or removably attached to the hip 60 of the end user 50 in a belt-coupled style configuration ( Figure 4The virtual image generation system 100 may also include a remote processing module 132 and a remote data repository 134 that are operably coupled to the local processing and data module 130, such as through wired leads or wireless connections 138, 140, so that these remote modules 132, 134 are operably coupled to each other and can be used as resources for the local processing and data module 130.
[0081] The local processing and data module 130 may include an energy-efficient processor or controller, and digital memory (such as flash memory), both of which may be used to assist in processing, caching, and storing data captured from sensors and / or data acquired and / or processed using the remote processing module 132 and / or remote data repository 134, which may be communicated to the display subsystem 110 after such processing or acquisition. The remote processing module 132 may include one or more relatively powerful processors or controllers configured to analyze and process data and / or image information. The remote data repository 134 may include a relatively large-scale digital data storage facility that may be available via the Internet or other networking configuration in a "cloud" resource configuration. In some embodiments, all data is stored and all calculations are performed in the local processing and data module 130, thereby allowing completely autonomous use from any remote module.
[0082] The couplers 136, 138, 140 between the various components described above may include one or more wired interfaces or ports for providing wired or optical communications, or one or more wireless interfaces or ports such as via RF, microwave, and IR for providing wireless communications. In some implementations, all communications may be wired, while in other implementations, all communications may be wireless. In further implementations, the choice of wired and wireless communications may be different from Figures 1 to 4 Therefore, the specific choice of wired or wireless communication should not be considered limiting.
[0083] In some embodiments, the user orientation module is included in the local processing and data module 130, while the CPU and GPU are included in the remote processing module. In alternative embodiments, the CPU, GPU, or portions thereof may be included in the local processing and data module 130. The 3D database may be associated with a remote data repository 134 or provided locally.
[0084] Some VR, AR, and / or MR systems use multiple volume phase holograms, surface relief holograms, or light-guiding optical elements embedded with depth plane information to generate images that appear to originate from corresponding depth planes. In other words, a diffraction pattern or diffractive optical element ("DOE") can be embedded in a light-guiding optical element ("LOE"; e.g., a planar waveguide) or printed / embossed on a light-guiding optical element such that as collimated light (a light beam with a substantially flat wavefront) is substantially totally internally reflected along the LOE, it intersects the diffraction pattern at multiple locations and is directed toward the user's eyes. The DOE is configured so that the light emitted from the LOE converges so that they appear to originate from a specific depth plane. A focusing lens ("condenser") can be used to generate the collimated light.
[0085] For example, a first LOE can be configured to deliver collimated light to the eye that appears to originate from an optically infinite depth plane (0 diopters). Another LOE can be configured to deliver collimated light that appears to originate from a distance of 2 meters (1 / 2 diopters). Another LOE can be configured to deliver collimated light that appears to originate from a distance of 1 meter (1 diopters). By using stacked LOE assemblies, it will be appreciated that multiple depth planes can be created, with each LOE being configured to display an image that appears to originate from a particular depth plane. It will be appreciated that a stack can include any number of LOEs. However, at least N stacked LOEs are required to generate N depth planes. In addition, N, 2N, or 3N stacked LOEs can be used to generate RGB color images at N depth planes.
[0086] In order to present 3D virtual content to the user, the VR, AR and / or MR system projects images of the virtual content into the user's eyes so that they appear to originate from various depth planes in the Z direction (i.e., orthogonally away from the user's eyes). In other words, the virtual content can not only change in the X and Y directions (i.e., in a 2D plane orthogonal to the central visual axis of the user's eyes), but can also appear to change in the Z direction, so that the user may perceive the object to be very close or at an infinite distance or any distance in between. In other embodiments, the user can perceive multiple objects at different depth planes at the same time. For example, the user can see a virtual dragon appearing from infinity and running towards the user. Alternatively, the user can simultaneously see a virtual bird at a distance of 3 meters from the user and a virtual coffee cup at arm's length (approximately 1 meter) from the user.
[0087] The multi-plane focusing system creates the perception of variable depth by projecting an image on some or all of multiple depth planes at respective fixed distances from the user's eyes in the Z direction. Figure 5 It should be understood that the multi-plane focusing system can be used at a fixed depth plane 150 (e.g., Figure 5Frames are displayed at six depth planes 150 shown in FIG. 1 . Although an MR system may include any number of depth planes 150, an exemplary multi-plane focusing system has six fixed depth planes 150 in the Z direction. When generating virtual content (one or more of the six depth planes 150), a 3D perception is created so that the user can perceive one or more virtual objects at different distances from the user's eyes. Figure 5 As shown in , assuming that the human eye is more sensitive to objects that are closer than objects that appear to be farther away, a depth plane 150 that is closer to the eye is generated. In other embodiments, the depth planes 150 can be placed equidistant from each other.
[0088] The depth plane position 150 may be measured in diopters, which is a unit of optical power equal to the reciprocal of the focal length measured in meters. For example, in some embodiments, depth plane 1 may be 1 / 3 diopters apart, depth plane 2 may be 0.3 diopters apart, depth plane 3 may be 0.2 diopters apart, depth plane 4 may be 0.15 diopters apart, depth plane 5 may be 0.1 diopters apart, and depth plane 6 may represent infinity (i.e., 0 diopters apart). It should be understood that other embodiments may generate depth planes 150 at other distances / diopters. Thus, when virtual content is generated at strategically placed depth planes 150, the user is able to perceive virtual objects in three dimensions. For example, when displayed in depth plane 1, the user may perceive a first virtual object as being close to him, while another virtual object appears at infinity at depth plane 6. Alternatively, virtual objects may be displayed first at depth plane 6, then at depth plane 5, and so on, until the virtual objects appear very close to the user. It should be understood that the above example is significantly simplified for illustrative purposes. In another embodiment, all six depth planes may be focused at a specific focal distance away from the user. For example, if the virtual content to be displayed is a coffee cup half a meter away from the user, all six depth planes may be generated at various cross-sections of the coffee cup, thereby providing the user with a highly granular 3D view of the coffee cup.
[0089] In some embodiments, the VR, AR, and / or MR system may be used as a multi-plane focusing system. In other words, all six LOEs may be illuminated simultaneously, so that images that appear to originate from six fixed depth planes are generated in rapid succession, and the light source rapidly delivers image information to LOE 1, then LOE 2, then LOE 3, etc. For example, a portion of a desired image may be injected at time 1, which includes an image of the sky at optical infinity, and the LOEs that maintain collimation of the light (e.g., from Figure 5 Then, an image of a closer branch can be injected at time 2, and a depth plane configured to create a depth plane that appears to originate from 10 meters away (e.g., Figure 5 The image of the pen can then be injected at time 3, and the LOE configured to create an image that appears to originate from a depth plane 1 meter away can be utilized. This type of paradigm can be repeated in rapid temporal sequence (e.g., 360 Hz) so that the user's eyes and brain (e.g., visual cortex) perceive the input as all parts of the same image.
[0090] VR, AR and / or MR systems can project images (i.e., by diverging or converging light beams) that appear to originate from various locations along the Z axis (i.e., depth planes) to generate images for a 3D experience / scene. As used in this application, light beams include, but are not limited to, directional projections of light energy (including visible and invisible light energy) radiated from a light source. Images that appear to originate from various depth planes are generated to conform to the vergence and accommodation of the user's eyes to the image, and to minimize or eliminate vergence-accommodation conflicts.
[0091] Exemplary traversable world model
[0092] Figure 6 Components of a traversable world model 2000 are shown in accordance with some embodiments. As a user 2001 traverses an environment, the user's personal AR system 2010 captures information (e.g., images, position information, position and orientation information, etc.) and saves the information via gesture-tagged images. In the illustrated embodiment, an image of an object 2020 (similar to a table) may be taken, and map points 2004 may be collected based on the captured image. This forms the core of the traversable world model, as shown by a plurality of keyframes (e.g., cameras) 2002 that have captured information about the environment.
[0093] like Figure 6 As shown in , there may be multiple keyframes 2002 that capture information about the space at any given point in time. For example, a keyframe could be another user's AR system capturing information from a particular viewpoint. Another keyframe could be a room-based camera / sensor system that is capturing images and points 2004 through a fixed viewpoint. By triangulating images and points from multiple viewpoints, the position and orientation of real objects in 3D space can be determined.
[0094] In some embodiments, the traversable world model 2008 is a combination of raster images, point clouds and descriptor clouds, and polygon / geometry definitions (referred to herein as parametric geometry). All of this information is uploaded to and retrieved from the cloud, with portions of it corresponding to specific spaces that a user may enter. Figure 6As shown in , the traversable world model also contains a number of object recognizers 2012 that work on the cloud or on the user's personal system 2010 to recognize objects in the environment based on point and pose tagged images captured through various keyframes of multiple users. Essentially, by continuously capturing information about the physical world through multiple keyframes 2002, the traversable world is always growing and can be negotiated (continuously or as needed) to determine how to render virtual content relative to existing physical objects in the real world. By collecting information from the user's environment, a traversable world 2006 is built / enhanced and can be "delivered" to one or more AR users simultaneously or in the future.
[0095] Asynchronous communication is established between the user's corresponding individual AR system and a cloud-based computer (e.g., a server computer). In other words, the user's personal AR system continuously updates information about the user's surroundings to the cloud, and also receives information about the traversable world from the cloud. Therefore, not every AR user has to capture images and identify objects based on the captured images, and having an asynchronous system can make the system more efficient. While new information is updated to the cloud, existing information about a portion of the world is automatically transmitted to the personal AR system. It should be understood that the traversable world model exists not only in the cloud or other forms of networked computing or peer-to-peer systems, but can also exist in the user's personal AR system.
[0096] In some embodiments, the AR system can use different levels of resolution for local components (e.g., computing components 130 such as a power pack) and remote components (e.g., cloud-based computers 132). This is because remote components (e.g., resources residing on a cloud server) are typically more computationally powerful than local components. The cloud-based computer can select data collected by many different personal AR systems and / or one or more spatial or room-based sensor systems and use this information to add to the traversable world model. The cloud-based computer can aggregate only the optimal (e.g., most useful) information into the persistent world model. In other words, redundant information and / or information of less than optimal quality can be processed in a timely manner to avoid degrading the quality and / or performance of the system.
[0097] Fig.10 An example method 2100 for interacting with a traversable world model according to some embodiments is shown. At 2102, a user's personal AR system can detect the user's position and orientation in the world. In some embodiments, as will be described in further detail below, the position can be derived from a topological map of the system. In some embodiments, the position can be derived from GPS or any other positioning tool. It should be understood that the personal AR system can constantly access the traversable world.
[0098] In another embodiment (not shown), a user may request access to another user's space, prompting the system to access that portion of the traversable world and associated parameter information corresponding to that other user. Thus, there may be many triggering conditions for traversable worlds. However, at the simplest level, it should be understood that the traversable world is constantly being updated and accessed by multiple user systems, thereby constantly adding and receiving information from the cloud.
[0099] Following the example above, at 2104, the system may draw a radius representing a physical area around the user that conveys both the user's location and intended direction based on the user's known location. Next, at 2106, the system may retrieve a piece of navigable world based on the user's intended location. In some embodiments, a piece of navigable world may contain information from a geometric map of the space acquired through previous keyframes and captured images and data stored in the cloud. At 2108, the AR system uploads information from the user's environment into the navigable world model. At 2110, based on the uploaded information, the AR system renders the navigable world associated with the user's location to the user's personal AR system.
[0100] This information enables virtual content to meaningfully interact with the user's real surroundings in a coherent way. For example, a virtual "monster" can be rendered as a specific building originating from the real world. Or, in another example, a user can leave a virtual object relative to the physical coordinates of the real world, so that a friend (also wearing an AR system) finds the virtual object at the same physical coordinates. In order to achieve such capabilities (and more capabilities), it is very important for the AR system to constantly access the traversable world to obtain and upload information. It should be recognized that the traversable world contains a persistent digital representation of the real space, which is crucially used in rendering virtual and / or digital content relative to the real coordinates of the physical space. It should be understood that the AR system can maintain coordinates of the real world and / or the virtual world. In some embodiments, a third party can maintain a map of the real world (e.g., coordinates), and the AR system can consult the map to determine one or more parameters in order to render virtual content related to real objects in the world.
[0101] It should be understood that the traversable world model itself does not render the content displayed to the user. Rather, it is a high-level concept that dynamically obtains and updates a persistent digital representation of the real world in the cloud. In some embodiments, the derived geometry information is loaded onto a game engine, which then renders the content associated with the traversable world. Thus, regardless of whether a user is in a particular space, that particular space has a digital representation in the cloud that any user can access. The piece of the traversable world may contain information about the physical geometry of the space and the image of the space, information about the various avatars that occupy the space, information about virtual objects, and other miscellaneous information.
[0102] As described in further detail herein, one or more object recognizers may inspect or "crawl" a traversable world model, marking points that belong to parametric geometry. Parametric geometry, points, and descriptors may be packaged into a traversable world model to allow low latency delivery or communication of information corresponding to a portion of the physical world or environment. In some embodiments, an AR system may implement a two-level structure in which a traversable world model allows for fast gesture processing in a first level, but within the framework is a second level (e.g., FAST features). In some embodiments, the second level structure may improve resolution by performing a three-dimensional (3D) feature map build on a frame-to-frame basis.
[0103] Topology
[0104] An integral part of a traversable world model is the creation of maps of very tiny areas of the real world. For example, in order to render virtual content related to physical objects, very detailed positioning is required. Such positioning cannot be achieved solely through GPS or traditional location detection technology. For example, an AR system may not only need the coordinates of the physical location of the user, but may also need to know exactly which room of the building the user is in, for example. Based on this information, the AR system can obtain data for the room (e.g., the specific geometry of the real objects in the room, map points of the room, geometric information of the room, etc.) to appropriately display virtual content relative to the identified real objects in the room. However, at the same time, this precise granular positioning must be performed in a cost-effective manner so that excessive resources are consumed unnecessarily.
[0105] To this end, the AR system can use a topological map for positioning purposes, rather than using GPS or obtaining a detailed geometric map created from extracted point and gesture tagged images (e.g., geometric points may be too specific and therefore most costly). In some embodiments, the topological map is a simplified representation of physical spaces in the real world and the relationships between various spaces that can be easily accessed from the cloud and presents only a fingerprint of the space. More details about the topological map are provided further below.
[0106] In some embodiments, the AR system can layer a topology map on top of a traversable world model, e.g., to locate nodes. The topology map can layer various types of information on top of the traversable world model, e.g., point clouds, images, objects in space, global positioning system (GPS) data, Wi-Fi data, histograms (e.g., a color histogram of a room), received signal strength (RSS) data, etc. This allows various layers of information (e.g., more detailed layers of information that interact with higher-level layers) to be placed in context with each other so that they can be easily accessed. This information can be thought of as fingerprint data; in other words, it is designed to be specific enough to be unique to a location (e.g., a particular room).
[0107] As discussed above, in order to create a complete virtual world that can be reliably transferred between various users, the AR system captures different types of information about the user's surroundings (e.g., map points, features, pose marker images, objects in the scene, etc.). This information is processed and stored in the cloud so that it can be retrieved as needed. As previously mentioned, the traversable world model is a combination of raster images, point clouds, and descriptor clouds, as well as polygon / geometry definitions (referred to herein as parametric geometry). Therefore, it should be understood that the large amount of information captured by the user's personal AR system allows for high quality and accuracy in creating the virtual world.
[0108] In other words, because various AR systems (e.g., user-specific head-mounted systems, room-based sensor systems, etc.) are constantly capturing data corresponding to the immediate environment of the respective AR system, very detailed and accurate information about the real world at any point in time is known with a high degree of certainty. While this amount of information is extremely useful for a large number of AR applications, sorting through so much information to find the piece of navigable world that is most relevant to the user for localization purposes is highly inefficient and takes up precious bandwidth.
[0109] To this end, the AR system creates a topology map that essentially provides less granular information about a specific scene or a specific place. In some embodiments, the topology map can be derived through global positioning system (GPS) data, Wi-Fi data, histograms (e.g., color histograms of rooms), received signal strength (RSS) data, etc. For example, the topology map can be created by histograms (e.g., color histograms) of various rooms / areas / spaces and reduced to nodes on the topology map. For example, when a user walks into a room or space, the AR system can take a single image (or other information) and construct a color histogram of the image. It should be understood that, to some extent, the histogram of a specific space is mostly constant over time (e.g., the color of the wall, the color of the objects in the room, etc.). In other words, each room or space has unique features that are different from any other room or place. The unique histogram can be compared and identified with other histograms of other spaces / areas. Since the AR system knows which room the user is in, the remaining granular information can be easily accessed and downloaded.
[0110] Therefore, although the histogram will not contain specific information about all the features and points that the various cameras (keyframes) have captured, the system can immediately detect where the user is based on the histogram, and then fetch all the more specific geometric information associated with a specific room or place. In other words, instead of having to sort through the large amount of geometric and parametric information that comprises the navigable world model, the topological map provides a fast and efficient way to locate the AR user. Based on the positioning, the AR system fetches the keyframes and points that are most relevant to the identified location. For example, after the system determines that the user is in a conference room in a building, the system can then fetch all the keyframes and points associated with the conference room instead of searching all the geometric information stored in the cloud.
[0111] Reference now Figure 8 , presenting an example embodiment of a topology map 4300. As discussed above, the topology map 4300 can be a collection of nodes 4302 and connections 4304 (e.g., represented by connecting lines) between the nodes 4302. Each node 4302 represents a specific location (e.g., a conference room in an office building) with a different signature or fingerprint (e.g., GPS information, color histogram or other histogram, Wi-Fi data, RSS data, etc.), and the lines can represent the connectivity between them. It should be understood that connectivity may not be related to geographic connectivity, but may simply be shared devices or shared users. For example, a first user may have walked from a first node to a second node. The relationship can be represented by the connection between the nodes. As the number of AR users increases, the nodes and the connections between the nodes will also increase proportionally, providing more accurate information about each location.
[0112] Once the AR system has identified a node of the topology graph, the system can then retrieve a set of geometric information related to the node to determine how / where to display virtual content relative to real objects in that space. Therefore, layering the topology graph on the geometry graph is particularly helpful for localizing and efficiently retrieving only relevant information from the cloud.
[0113] In some embodiments, the AR system may represent two images captured by respective cameras of a portion of the same scene in a graph-theoretic context as first and second gesture-tagged images. It should be understood that in this context, a camera may refer to a single camera taking images of different scenes, or may be two different cameras. There is a certain strength of connection between the gesture-tagged images, which may be, for example, points in the field of view of the two cameras. In some embodiments, a cloud-based computer may construct a graph (e.g., with Figure 8 The total number of nodes and edges in a graph is much smaller than the total number of points in an image.
[0114] At a higher level of abstraction, other information monitored by the AR system can be hashed together. For example, a cloud-based computer can hash together one or more of Global Positioning System (GPS) location information, Wi-Fi location information (e.g., signal strength), a color histogram of the physical space, and / or information about physical objects surrounding the user. The more data points there are, the more likely it is that the computer has a statistically unique identifier for the space. In this case, space is a statistically defined concept.
[0115] As an example, an office can be a space represented as, for example, a large number of points and two dozen gesture-labeled images. The same space can be represented topologically as a graph with only a certain number of nodes (e.g., 5, 25, 100, 1000, etc.), which can be easily hashed again. Graph theory allows representing, for example, the connectivity of the algorithmically shortest path between two spaces.
[0116] Therefore, the system abstracts from specific geometry by converting it into a pose-tagged image with implicit topology. The system takes the abstraction level even higher by adding other pieces of information, such as color histogram profiles and Wi-Fi signal strength. This makes it easier for the system to recognize the user's actual real-world location without having to understand or process all the geometry associated with that location.
[0117] Fig. 9An example method 2400 for constructing a topology map is shown. First, at 2402, a user's personal AR system can capture an image from a first viewpoint of a particular location (e.g., the user walks into a room of a building and an image is captured from that viewpoint). At 2404, a color histogram can be generated based on the captured image. As previously described, the system can use any other type of identifying information (e.g., Wi-Fi data, RSS information, GPS data, number of windows, etc.), but a color histogram is used in this example for illustrative purposes.
[0118] Next, at 2406, the system runs a search to identify the user's location by comparing the color histogram to a database of color histograms stored in the cloud. At 2410, a decision is made to determine whether the color histogram matches an existing color histogram stored in the cloud. If the color histogram does not match any color histograms of the color histogram database, it can be stored as a node of the topology map (2414). If the color histogram matches an existing color histogram of the database, it is stored as a node in the cloud (2412). If the color histogram matches an existing color histogram in the database, the location is identified and the appropriate geometric information is provided to the personal AR system.
[0119] Continuing with the same example, the user may walk into another room or another location where the user's personal AR system takes another picture and generates another color histogram of another location. If the color histogram is the same as the previous color histogram or any other color histogram, the AR system identifies the user's location. If the color histogram is different from the stored histogram, another node is created on the topology map. In addition, since the first node and the second node were taken by the same user (or the same camera / same personal user system), the two nodes are connected in the topology map.
[0120] In addition to helping with positioning, topological maps can also be used to improve / repair errors and / or missing information in geometric maps. In some embodiments, topological maps can be used to find loop stress in geometric maps or geometric structures at a specific location. As discussed above, for any given location or space, images taken by one or more AR systems (multi-view images captured by one user's personal AR system or multiple users' AR systems) generate a large number of map points for a specific space. For example, a single room may correspond to thousands of map points captured by multiple viewpoints of various cameras (or one camera moved to various locations).
[0121] The AR system utilizes map points to identify objects as discussed above (via object recognizers) and add to the traversable world model in order to store a more comprehensive picture of the geometry of various objects in the real world. In some embodiments, the map points derived from the various keyframes can be used to triangulate the pose and orientation of the camera that captured the image. In other words, the collected map points can be used to estimate the pose (e.g., position and orientation) of the keyframe (e.g., camera) that captured the image.
[0122] However, it should be understood that given the large number of map points and keyframes, there will inevitably be some errors (e.g., stresses) in this calculation based on the keyframe positions of the map points. To address these stresses, the AR system can perform a bundle adjustment. The bundle adjustment allows the map points and keyframes to be refined or optimized to minimize the stresses in the geometry.
[0123] For example, Fig.10 As shown in , example geometries are presented. Fig.10 As shown in , the geometric graph can be a collection of keyframes 2502 that are all connected to each other. The keyframes 2502 can represent the viewpoints of various map points from which the geometric graph is derived. In the illustrated embodiment, each node of the geometric graph represents a keyframe (e.g., a camera), and the various keyframes are connected to each other by connecting lines 2504.
[0124] In the illustrated embodiment, the strength of the connection between different keyframes is represented by the thickness of the connection line 2504. Fig.10 As shown in , the connecting line between nodes 2502a and 2502b is depicted as a thicker connecting line 2504 than the connecting line between node 2502a and node 2502f. The connecting line between node 2502a and node 2502d is also depicted as thicker than the connecting line between 2502b and node 2502d. In some embodiments, the thickness of the connecting line represents the number of features or map points shared between them. For example, if the first keyframe and the second keyframe are close together, they can share a large number of map points (e.g., node 2502a and node 2502b), and therefore can be represented by a thicker connecting line. Of course, it should be understood that other ways of representing geometric diagrams can be used similarly.
[0125] For example, in another embodiment, the strength of the line can be based on the geographic proximity between keyframes. Fig.10As shown in , each geometric graph represents a number of keyframes 2502 and their connections to each other. Now, assuming that stress is identified in a particular point of the geometric graph, a bundle adjustment can be performed to relieve the stress by radially pushing the stress outward radially from the identified stress point 2506. As will be described in further detail below, the stress is radially pushed out from the wave 2508 (e.g., n=1, n=2, etc.) propagating from the stress point.
[0126] The following description shows an example method of performing wave propagation beam adjustment. It should be understood that all of the following examples only involve wave propagation beam adjustment, and other types of beam adjustments can be similarly used in other embodiments. First, identify specific stress points. Fig.10 In the illustrated embodiment of , the center (node 2502a) is considered the identified stress point. For example, the system may determine that the stress at a particular point of the geometric map is particularly high (e.g., residual error, etc.). The stress may be identified based on one of two reasons. First, a maximum residual error may be defined for the geometric map. If the residual error at a particular point is greater than the predefined maximum residual error, a bundle adjustment may be initiated. Second, in the case of loop stress, a bundle adjustment may be initiated, as will be further described below (when the topology map indicates that the map points are not aligned).
[0127] When stress is identified, the AR system distributes the error evenly by starting at the stress point and propagating the error radially through the network of nodes surrounding the particular stress point. For example, in the illustrated embodiment, the bundle adjustment may distribute the error as n=1 (one degree of separation from the identified stress point, node 2502a) around the identified stress point. In the illustrated embodiment, nodes 2502b-2502g are all part of the n=1 wave around the stress point, node 2502a.
[0128] In some cases, this may be sufficient. In other embodiments, the AR system may propagate the stresses even further and push the stresses out to n=2 (two degrees of separation from the identified stress point, node 2502a) or n=3 (three degrees of separation from the identified stress point, node 2502a) so that the stresses are pushed out radially step by step until the stresses are evenly distributed. Therefore, performing a bundle adjustment is an important method to reduce stresses in a geometric figure. Ideally, the stresses are pushed out to n=2 or n=3 to obtain better results.
[0129] In some embodiments, waves can be propagated in even smaller increments. For example, after the waves have been pushed out to n=2 around the stress point, a bundle adjustment can be performed in the region between n=3 and n=2 and propagated radially. By controlling the increments of the waves, this iterative wave propagation bundle adjustment process can be run on large amounts of data to reduce stress on the system. In an optional embodiment, because each wave is unique, nodes that have been touched by a wave (e.g., a bundle adjustment) can be colored so that the wave does not re-propagate on the adjusted part of the geometry. In another embodiment, the nodes can be colored so that synchronized waves can propagate from / originate from different points in the geometry.
[0130] As mentioned previously, layering the topology graph over the geometry of keyframes and map points can be particularly important for discovering loop closure stress. Loop closure stress refers to the differences between map points captured at different times that should be aligned but are not. For example, if a user walks around the block and returns to the same location, the map point derived from the location of the first keyframe and the map point derived from the location of the last keyframe inferred from the collected map points should ideally be the same. However, given the stress inherent in the calculation of poses (positions of keyframes) based on different map points, errors often occur and the system fails to recognize that the user has returned to the same location because the keypoints estimated from the first keyframe are not geometrically aligned with the map points derived from the last keyframe. This can be an example of loop closure stress.
[0131] To this end, the topology map can be used to find loop closure stress in the geometric map. Returning to the previous example, using the topology map in conjunction with the geometric map allows the AR system to identify loop closure stress in the geometric map because the topology map may indicate that the user has returned to the starting point (e.g., based on a color histogram). For example, referring to Fig.11 In the hierarchical graph 2600, the nodes of the topological graph (e.g., 2604a and 2604b) are layered on top of the nodes of the geometric graph (e.g., 2602a-2602f). Fig.11 As shown in , when the topology map is placed on top of the geometry map, the topology map may indicate that keyframe B (node 2602g) is the same as keyframe A (node 2602a). Based on this, loop closure stress can be detected, the system detects that keyframes A and B should be closer together in the same node, and then the system can perform a bundle adjustment. Therefore, after identifying the loop closure stress, the AR system can then use a bundle adjustment technique, such as the bundle adjustment technique described above, to perform a bundle adjustment on the identified stress point.
[0132] It should be appreciated that performing bundle adjustment based on the layering of the topology map and the geometry map ensures that the system only obtains the keyframes that need to be bundle adjusted, rather than obtaining all keyframes in the system. For example, if the AR system identifies the presence of loop stress based on the topology map, the system can simply obtain the keyframes associated with one or more specific nodes of the topology map and perform bundle adjustment only on those keyframes rather than on all keyframes of the geometry map. Again, this makes the system efficient and does not obtain unnecessary information that may unnecessarily cost the system resources.
[0133] Reference now Fig.12 , describes an example method 2700 for correcting loop stress based on a topological map. At 2702, the system can identify loop stress based on a topological map layered on top of a geometric map. At 2704, once loop stress has been identified, the system can retrieve a set of keyframes associated with a node of the topological map at which loop stress has occurred. After retrieval of the keyframe for that node of the topological map, the system can initiate a bundle adjustment at that point in the geometric map at 2706. At 2708, similar to Fig.10 In the technique shown in , stress propagates from the identified stress point and is distributed radially in the form of a wave to n=1 (then n=2, n=3, etc.).
[0134] Map Building
[0135] When mapping a virtual world, it is important to know all the features and points in the real world to accurately depict virtual objects in relation to the real world. To this end, as discussed above, map points captured from various head-mounted AR systems are continuously added to the traversable world model by adding new pictures that convey information about various points and features of the real world. As discussed above, based on the points and features, the pose and position of keyframes (e.g., cameras, etc.) can also be inferred. While this allows the AR system to collect a set of features (2D points) and map points (3D points), it may also be important to find new features and map points to render a more accurate version of the traversable world.
[0136] One way to find new map points and / or features can be to compare features of one image with features of another image. Each feature can have a label or feature descriptor attached to it (e.g., color, identifier, etc.). Comparing feature labels in one picture with another picture may be a way to uniquely identify natural features in the environment. For example, if there are two keyframes, each capturing about 500 features, then comparing the features of one keyframe with the other keyframe may help determine new map points. However, while this may be a viable solution in the case of only two keyframes, when there are multiple keyframes (each capturing millions of points), it becomes a very large search problem that requires a lot of processing power. In other words, if there are M keyframes, each with N unmatched features, then searching for new features will involve MN2 (O(MN2)) operations. Unfortunately, this is a very large search operation.
[0137] A new point finding method that can avoid such large search operations is to render instead of search. In other words, assuming that the positions of M keyframes are known and each keyframe has N points, the AR system can project lines (or cones) from the N features to the M keyframes to triangulate the 3D position of each 2D point. Now refer to Fig.13 , in this particular example, there are 6 keyframes 2802, and lines or rays are rendered (using a graphics card) from the 6 keyframes to a point 2804 derived from a corresponding keyframe. In some embodiments, a new 3D map point may be determined based on the intersection of the rendered lines. In other words, when two rendered lines intersect, the pixel coordinate of that particular map point in 3D space may be 2, rather than 1 or 0. Thus, the higher the intersection of the lines at a particular point, the higher the likelihood that there is a map point in 3D space that corresponds to a particular feature. In some embodiments, as Fig.13 The intersection method shown in can be used to find new map points in 3D space.
[0138] It should be understood that for optimization purposes, rather than rendering lines from the keyframes, a triangular pyramid may be rendered from the keyframes for more accurate results. The triangular pyramid is projected so that the rendered lines (e.g., 2804) to the N features represent the bisector of the triangular pyramid, and the sides of the pyramid are projected on either side of the Nth feature. In some embodiments, the half angle from the two sides may be defined by the pixel spacing of the camera, which is extended by the lens mapping function on either side of the Nth feature.
[0139] The interior of the cone can be occluded such that the bisector is the brightest and the edges on either side of the Nth feature can be set to 0. The camera buffer can be a sum buffer such that the bright spots can represent candidate locations for new features, taking into account both the camera resolution and the lens calibration. In other words, the projected cone rather than a straight line can help compensate for the fact that some keyframes are farther away than other keyframes that have already captured features at closer distances. In this method, the frustum rendered from a farther keyframe will be larger (and have a larger radius) than the frustum rendered from a closer keyframe. A sum buffer can be applied to determine the 3D map points (e.g., the brightest spots on the map can represent new map points).
[0140] In essence, the AR system can project rays or frustums from N mismatched features in M previous keyframes onto the texture of the M+1 keyframe, encoding the keyframe identifier and the feature identifier. The AR system can construct another texture based on the features in the current keyframe and mask the first texture with the second texture. All colors are candidate pairings for search constraints. This method advantageously converts O(MN2) search constraints to O(MN) rendering, followed by a minor O((<M)N(<<N)) search.
[0141] In another method, new map points can be determined by selecting a virtual keyframe from which the existing N features can be viewed. In other words, the AR system can select the virtual keyframe from which to view the map points. For example, the AR system can use the above keyframe projection but select a new "keyframe" based on the PCA (principal component analysis) of the normals of the M keyframes from which the {M, N} labels are sought (e.g., a PCA-derived keyframe will provide the optimal view from which to derive the labels).
[0142] Performing PCA on the existing M keyframes provides a new keyframe that is most orthogonal to the existing M keyframes. Therefore, positioning the virtual keyframe in the most orthogonal direction can provide the optimal viewpoint from which to find new map points in 3D space. Performing another PCA provides the next most orthogonal direction, and performing another PCA provides another orthogonal direction. Thus, it can be understood that performing 3 PCAs can provide the x, y, and z coordinates in the 3D space in which map points can be constructed based on the existing M keyframes with N features.
[0143] Fig.14An example method 2900 for determining a map point from M known keyframes is described. First, at 2902, the AR system obtains M keyframes associated with a particular space. As discussed above, the M keyframes refer to known keyframes that have captured the particular space. Next, at 2904, PCA of the normals of the keyframes is performed to find the most orthogonal direction of the M keyframes. It should be understood that PCA can produce three principals, each principal being orthogonal to the M keyframes. Next, at 2906, the AR system selects the principal that is smallest in 3D space and is also the most orthogonal to the views of all M keyframes.
[0144] At 2908, after the subject orthogonal to the keyframe has been identified, a virtual keyframe may be placed along the axis of the selected subject. In some embodiments, the virtual keyframe may be placed far enough away that its field of view includes all M keyframes.
[0145] Next, at 2910, the AR system can render a feature buffer such that rays (or cones) are rendered from each of the M keyframes to the Nth feature. The feature buffer can be a summed buffer such that bright spots (pixel coordinates where N lines have intersected) represent candidate locations for the N features. It should be understood that the same process described above can be repeated for all three PCA axes such that map points are found on the x, y, and z axes.
[0146] Next, at 2912, the system can store all the bright spots in the image as virtual "features". Next, at 2914, a second "label" buffer can be created at the virtual keyframe to stack lines (or cones) and save their {M, N} labels. Next, at 2916, a "mask radius" can be drawn around each bright spot in the feature buffer. It should be understood that the mask radius represents the angular pixel error of the virtual camera. The AR system can fill the resulting circle around each bright spot and mask the label buffer with the resulting binary image. In an optional embodiment, the circle can be filled by applying a gradient filter so that the center of the circle is bright, but the brightness gradually decreases to zero at the periphery of the circle.
[0147] In the now masked label buffer, the subject rays can be collected using the {M, N} tuple label for each triangle. It should be understood that if cones / triangles are used instead of rays, the AR system can only collect triangles where both edges of the triangle are captured inside the circle. Therefore, the mask radius basically acts as a filter that eliminates rays with poor conditions or rays with large divergence (e.g., rays at the edge of the field of view (FOV) or rays emitted from far away).
[0148] For optimization purposes, the tag buffer can be rendered with the same shading as previously used in the generated cones / triangles. In another optional optimization embodiment, the triangle density can be scaled from one to zero instead of checking the extents (sides) of the triangles. Thus, very divergent rays will effectively raise the noise floor in the occluded areas. Running a local threshold check inside the marker will trivially extract centroids only from rays that are completely inside the marker.
[0149] At 2918, the set of occluded / optimized rays m may be fed to a bundle adjuster to estimate and / or correct the position of the newly determined map point. It should be appreciated that the system is functionally limited by the size of the render buffer employed. For example, if the keyframes are widely separated, the rays / cones generated will be of lower resolution.
[0150] In an alternative embodiment, instead of using PCA analysis to find the orthogonal direction, a virtual keyframe can be placed at the location of one of the M keyframes. This may be a simpler and more efficient solution, since the M keyframes may have captured the space at the optimal resolution of the camera. If PCA is used to find the orthogonal direction to place the virtual keyframe, the above process is repeated by placing the virtual camera along each PCA axis and finding the map point in each axis.
[0151] In another example method of finding a new map point, the AR system can hypothesize a new map point. The AR system can take the first three principal components from the PCA analysis of the M keyframes. Next, a virtual keyframe can be placed at each principal. Next, the feature buffer can be accurately rendered at each of the three virtual keyframes exactly as described above. Because the principal components are orthogonal to each other by definition, rays drawn outward from each camera may collide with each other at some point in 3D space.
[0152] It should be understood that in some cases, there may be intersections of multiple rays. Therefore, there may now be N features in each virtual keyframe. Next, a geometric algorithm can be used to find the intersections between different rays. Because there may be N3 rays, this geometric algorithm can be a constant time algorithm. Masking and optimization can be performed in the same manner as described above to find map points in 3D space.
[0153] World Model Refinement
[0154] In some embodiments, the AR system can stitch separate small world model fragments into larger coherent fragments. This may happen at two levels: small model and large model. The small model corresponds to the local user level (e.g., on a computing component such as a waist pack). On the other hand, the large model corresponds to the large-scale or system-wide level for "whole world" modeling (e.g., a cloud system). This can be implemented as part of the concept of a traversable world model.
[0155] For example, a personal AR system worn by a first user captures information about a first office, while a personal AR system worn by a second user captures information about a second office that is different from the first office. The captured information can be passed to a cloud-based computer that ultimately builds a comprehensive and consistent representation of the real space sampled or collected by the various users walking around with the various AR devices. The cloud-based computer gradually builds a navigable world model through use over time. Different geographic locations are expected to be built, mainly concentrated in population centers, but eventually covering more rural areas.
[0156] As described above, the cloud-based computer can, for example, perform hash operations on GPS, Wi-Fi, room color histograms, and caches of all natural features in the room, and place pictures, and generate a topology map that is the topology of the connectivity of things. The cloud-based computer can use the topology to identify locations to stitch areas together. Alternatively, the cloud-based computer can use hash values of features (e.g., a topology map), for example, to identify a geometric configuration in one location that matches a geometric configuration in another location.
[0157] Exemplary Map Building System
[0158] Fig.15 Schematically depicting a map building system 200 including a first AR / MR system 220 and a second AR / MR system 230 according to some embodiments. The first AR / MR system 220 and the second AR / MR system 230 are communicatively coupled to a cloud processing component 210 via a first network 240 and a second network 242, respectively. Fig.15 2, but the various components of the map construction system 200 may be communicatively coupled to each other via more or fewer networks. Each of the first network 240 and the second network 242 may be, for example, a wireless or cellular network, a dedicated communication network (e.g., a mobile phone network), a local area network (LAN), a wide area network (WAN), and / or other technologies that enable one or more computing devices to communicate with each other.
[0159] Each of the first AR / MR system 220 and the second AR / MR system 230 has a power pack 222, 232 communicatively coupled to a wearable device 224, 234 using a corresponding USB connector 226, 236. The power pack 222, 232 can have an NVidia / ARM architecture, running a malleus system. The wearable device 224, 234 can have a Myriad / SHAVE architecture running a minotaur system. The cloud processing component 210 can have a PC architecture running a server (e.g., Windows) system.
[0160] Each of the first AR / MR system 220 and the second AR / MR system 230 can perform map building and map synchronization by itself. In addition, the first AR / MR system 220 and the second AR / MR system 230 can use data from other systems 230, 220 transmitted through the club processing component 210 to perform map building and map synchronization. Sharing data from multiple AR / MR systems 220, 230 increases the accuracy of map building. However, increasing the amount of input data for map building increases the processor load.
[0161] Example map construction / Map synchronization method
[0162] Map building is not static, but is constantly generated, modified, and optimized. In this way, the state of the map changes over time. In addition, factors such as data locality (i.e., the data sources for map building operations are various components such as wearable devices and power packs) and platform constraints (e.g., processing power and memory) make it more efficient to distribute certain workloads / jobs between various system components (such as wearable devices 224, power packs 222, and cloud processing components 210). The nature of the connection of the AR / MR system 220 and the communication delays in the connections between the various components (e.g., network 240 and USB 226) cause significant delays to achieve cross-component map synchronization. For example, in some AR / MR systems, the map is "locked" during each map-related operation, and then copies of the updated map are distributed across components to facilitate the next operation. For many AR / MR systems, this delay / lag is unacceptable. Map management takes into account different workloads related to the map at different times, as well as the synchronization delays of different map building system components.
[0163] Parallel processing between system components (such as SHAVES) can reduce system latency. However, the differences between the above system components complicate the implementation of parallel processing. Some of these complexities include the triggering of system components at different times, different processing speeds of system components, and different data sets on which system components operate. Asynchronous parallel processing involves various considerations, such as managing multiple readers and writers operating on the same map. Minimizing the number of concurrent processes can reduce system complexity, but a certain degree of asynchronous parallel processing improves map building system performance.
[0164] Asynchronous parallel map building workflow
[0165] Fig.16 A workflow for an asynchronous map building method 300 for a multi-component map building system according to some embodiments is depicted. Various map modification techniques involving new data and optimization can be performed in parallel using method 300. As described above, new data techniques can include tracking, VIO, map building, relocalization, map merging, and joint dense optimization. Map modification techniques involving optimization include bundle adjustment and loop closure. New data techniques are efficiently performed at system components that are new data sources. Optimization techniques require a lot of processing and can therefore be more efficiently performed at power packages. During optimization, special consideration needs to be given to persistent points and dense map building.
[0166] At time "t" 310, a bundle adjustment (optimization) operation 320 begins on the snapshot of the sparse graph. Because the bundle adjustment 320 is a processing intensive operation, the bundle adjustment 320 takes a relatively long amount of time ΔT 322 to complete. As discussed above, the processing intensive bundle adjustment 320 may be performed by a dynamic package.
[0167] During the execution of the bundle adjustment 320, another component (e.g., a wearable device) may perform map point deletion 330 (e.g., because a map point is determined to be an outlier). Map point deletion 330 is performed on subsequent snapshots of the sparse map beginning at time "t+1" 312. Because map point deletion 330 is a relatively less processing intensive operation, map point deletion 330 may be completed during the bundle adjustment 320. At time "t+2" 314, the wearable device has completed map point deletion 330.
[0168] After map point deletion 330 is completed, the system may perform other operations during bundle adjustment 320. Finally, at time "t+ΔT" 316, the bundle adjustment is complete. This workflow 300 allows multiple map-related operations (bundle adjustment 320, map point deletion 330, etc.) to be performed in parallel by different components of the mapping system, thereby reducing the amount of latency / lag in the map building system. The parallel execution of multiple map-related operations will be described in further detail below. In particular, if the map points being deleted are also part of a bundle adjustment operation, there may be a conflict between these two operations (bundle adjustment 320 and map point deletion 330).
[0169] Data conflicts during asynchronous parallel map construction
[0170] 17A to 17D Depicted are several example loop operations that may result in data conflicts with asynchronous parallel map construction. Fig.17A Six keyframes 410, 411, 412, 413, 414, 415, and 416 are depicted as points in map 401. A loop 418 is detected between keyframes 410 and 410'.
[0171] Fig. 17B It depicts closing loop 418 with fixed origin 410 by moving keyframes 422 , 423 , 424 , 425 , and 426 to generate a new map 402 .
[0172] Fig. 17C The loop 418 is depicted closed with a fixed origin 410. Due to the asynchronous workload, loop 438 is detected at a time corresponding to the generation of keyframe 433. Thus, loop 438 is closed by moving keyframes 432 and 433. One or more separate workloads generate keyframes 434, 435, and 436. The resulting map 403, which loops back at keyframe 433 and then generates keyframes 434, 435, and 436, has a similar Fig.17A and Fig. 17B The maps 401 and 402 are of very different shapes.
[0173] Fig.17D 440 while allowing the origin 410 to move to a new location 440 according to some embodiments. This loop closure can be performed during the generation of key binding frames 441, 442, 443, 444, 445, and 446 without generating unusual shapes. For example, Fig. 17C Compared with the map 403 in Fig.17D The map in 404 is closer Fig.17A and Fig. 17B Maps 401 and 402.
[0174] Map management design includes identifying and resolving / reconciling conflicts such as these and many other possible map building conflicts with asynchronous workloads. Resolution / reconciliation includes determining whether the initial state used to generate the state change is still valid and whether the state change should be applied. These conflicts are data and time related, further complicating the resolution / reconciliation process in map management.
[0175] Asynchronous parallel map building workflow with map management
[0176] Fig.18 Depicted is a workflow for an asynchronous map building method 500 for a multi-component map building system with a map manager according to some embodiments. The map building method 500 includes a unique timeline that is maintained by the map manager (described below) but is not necessarily represented as a time-indexed collection of states. Fig.18 Included are three states 510, 512, 514. A "+" operator 516 represents applying a change (eg, "Δ0" 522) to a previous state to generate a next state.
[0177] Changes are generated by various map operations in method 500. For example, map building operation 520 receives as input the state when map building operation 520 is performed (i.e., state zero 510) and external data 518. State zero 510 can be a special state where the map is empty. As with all map building operations in method 500, the output of map building operation 520 is not a new state, but a state change (i.e., Δ0 522). Similarly, bundle adjustment operation 530 receives as input the state when bundle adjustment operation 530 is performed (i.e., state one 512) and other data (not shown). The output of bundle adjustment operation 530 is another state change (i.e., Δ1 532). In addition, multiple other map operations 540 can output corresponding multiple state changes 542.
[0178] The "+" operation 516 applies various state changes (i.e., Δ0 522, Δ1 532, Δ ... 542) to the previous state to generate the next state. For example, state change Δ0 522 is applied to state zero 510 to generate state one 512. Similarly, state change Δ1 532 is applied to state one 512 to generate state two 514. State changes 522, 532, 542 are applied in the order of state progression as long as there are no conflicts. If a conflict between a state and a state change is detected, the conflict between the state and the state change is resolved / reconciled during the "+" operation 516. The conflict resolution is implementation-dependent because the type of conflict depends on the modification applied to the map.
[0179] System components read snapshots of the various states 510, 512, 514 and other data that may be needed by the various map building operations 520, 530, 540. Similarly, system components write state changes 522, 532, 542 that contain modifications to the map generated by the various map building operations 520, 530, 540. Soon after receiving the state changes 522, 532, 542, they are applied to create a new map state 512, 514. Various components that perform operations that depend on the map state can subscribe to the state changes.
[0180] Thus, a given map state may be determined by the initial state, and used for the order of state changes. For example, given an initial state zero 510, and state changes 522 and 532, a map state may be defined as state zero 510 "+" state change 522 "+" state change 532. Thus, a globally agreed upon initial map state and ordering of state changes allows different system components to perform map building operations and generate state changes while running on different platforms at different rates.
[0181] The map manager determines the initial map state and ordering of state changes, and sends / pushes / broadcasts / copies this map build data to all system components to allow asynchronous parallel map builds while maintaining consistent map builds across all components of the system. Each system component has a map manager that will ultimately have access to all state changes produced in the system. The ordered state changes serve as a virtual bus connecting all system components and the map managers within them.
[0182] Fig.19 An AR / MR system 600 for map building and map synchronization is depicted. The system 600 includes a wearable device 610 and a power pack 620 that communicate with each other and are coupled to a cloud processing component 640 via a USB communication link 630. Each of the wearable device 610 and the power pack 620 includes a map manager 612, 622, a reader 614, 624, and a writer 616, 626. Each of the map managers 612, 622, the readers 614, 624, and the writers 616, 626 are communicatively coupled to a USB communication link 630 that forms a virtual bus connecting all system components.
[0183] The readers 614, 624 receive data from the USB communication link 630 and other sources in their respective components 610, 620. The data received by the readers 614, 624 includes state changes. The readers 614, 624 send the received data to the map managers 612, 622.
[0184] The map manager 612, 622 processes the data received from the reader 614, 624. As described above, the map manager 612 in the wearable device 610 generates a change order 618 corresponding to the order of state changes starting from the initial state. The change order 618 is generated based on when the state changes are applied to the various states. The order is generally related to the time when the wearable map manager 612 receives the state changes. If more than one state change arrives at the wearable map manager 612 at the same time, the map manager 612 decides how to order the state changes written at the same time.
[0185] The map manager 612 in the wearable device 610 sends a change order 618 to the map manager 622 in the power pack 620. The change order 618 allows the system components 610, 620 and the virtual bus 630 to operate asynchronously, so that the map managers 612, 622 can receive state changes from different system components 610, 620 in a sequence different from the change order 618. The change order 618 allows the map managers 612, 622 to apply the received state changes in an agreed order to synchronize the map across the system components 610, 620.
[0186] In some embodiments, the reordering of state changes based on the change order 618 does not change the order of state changes generated by system components including a particular map manager relative to each other. In addition, the state changes can be modified to include only data related to the map in a particular system component (e.g., a smaller map stored in a wearable device as described below) to more efficiently utilize the virtual bus 630.
[0187] The map manager 612, 622 sends data to the writer 616, 626. Such data may include state changes, and in some embodiments, a change order 618. The writer 616, 626 in turn sends data to the virtual bus 630. The virtual bus 630 distributes the state changes between the system components 610, 620 in a change order 618, so that each component 610, 620 eventually receives all state changes and change orders 618. Because the map is not static, the system components 610, 620 constantly send map updates to each other across the virtual bus 630. The virtual bus 630 is also connected to a cloud processing component 640, which can retain its own map for use across multiple map building systems connected thereto. Like those in the other system components 610, 620, the cloud processing component 640 can have a map manager, a reader, and a writer (not shown).
[0188] Fig. 20A wearable map manager 710 is depicted that generates a change sequence 750 by analyzing a plurality of received state changes. The state changes include map building state changes 720, bundle adjustment state changes 730, and loop closure state changes 740. The state changes may be generated by different system components. For example, some map building state changes 720 may be generated by the wearable device, and the bundle adjustment state changes 730 and loop closure state changes 740 may be generated by the power pack. Each state change may be uniquely identified by a combined key that identifies the writer and a sequence count or timestamp.
[0189] For example, if each writer makes a state change, the combined key could be:
[0190] Mapping: MapDelta(0), MapDelta(1)
[0191] BundleAdjust: BundleAdjust(0), BundleAdjust(1)
[0192] LoopClosure: LoopClosure(0), LoopClosure(1)
[0193] The wearable map manager 710 analyzes the received state changes and generates a change order 750. Continuing with the example of the combined key listed above, the change order can be communicated as a single stream of state changes that also represents a global ordering of those state changes. An exemplary change order is "MapDelta(0), BundleAdjust(0), MapDelta(1), LoopClosure(0), BundleAdjust(1), LoopClosure(1)". Other map managers and systems eventually receive the same state changes in the change order 750 from the wearable map manager 710.
[0194] Fig.21A power pack map manager 810 is depicted applying a plurality of received state changes based on a received change sequence 850. The state changes include map building state changes 820, bundle adjustment state changes 830, and loop closure state changes 840. The state changes may be generated by different system components. For example, some map building state changes 820 may be generated by a wearable device, and bundle adjustment state changes 830 and loop closure state changes 840 may be generated by a power pack. The change sequence 850 may have been generated by a wearable map manager and received via a virtual bus. In order to apply the received state changes 820, 830, 840 in a sequence encoded in the received change sequence 850, the power pack map manager 810 includes a plurality of buffers 822, 832, 842 that store the received state changes 820, 830, 840 until they are called by the change sequence 850. If the next state change according to the change sequence 850 is available, the power pack map manager 810 will apply the next change. If the next change according to the change sequence 850 is not available, the power packet map manager 810 will wait for the next state change to be received and operate in an asynchronous manner. If the state change arrives before the change sequence 850, the power packet map manager 810 will wait for the change sequence 850 to arrive and operate in an asynchronous manner.
[0195] The "+" operation described above for applying state changes to state is configured to be substantially similar for various system components. This allows state changes to be distributed and applied across different system components. The "+" operation can be configured to be performed by wearable devices and power packs by removing the device's binary from the manifest. For cloud processing components, the "+" operation can be configured by removing the cloud binary from the manifest or an explicit version of the "+" operator, etc.
[0196] Fig. 22 A wearable map manager 910 is depicted that processes multiple received state changes. The state changes include tracking state changes 920, map building state changes 930, and bundle adjustment state changes 940. The state changes can be generated by different system components. For example, some tracking state changes 920 and map building state changes 930 can be generated by the wearable device, and the bundle adjustment state changes 940 can be generated by the power package. The wearable map manager 910 generates a change order 950 from the received state changes 920, 930, 940. In order to minimize the locking of the map, the wearable map manager 910 also performs reader jobs Pre-Track 924, Pre-Map 934, and Pre-Bund Adjust 944 to create snapshots of the map for the corresponding state changes.
[0197] Fig.23The power pack map manager 1010 is depicted processing a plurality of received state changes 1020 based on a received change order 1050. The state changes may be generated by different system components, and the change order may be generated by a wearable device. The power pack map manager 1010 generates a change order 1050 from the received state changes 1020, 1030, 1040. In order to minimize the locking of the map, the power pack map manager 1010 also performs a reader job pre-loop (Pre-Loop Cl) 1024, a pre-pose (Pre-Pose) 1034, and a pre-cell stream transmission (Pre-Cell Str) 1044 to create a map snapshot of the corresponding state change. The snapshot may include only the data required for a specific reader job. After taking the snapshot, the map manager may execute different reader jobs on the map. Generating the snapshot allows the actual map processing to be performed offline, thereby minimizing the amount of time the map is locked for processing.
[0198] The power packet map manager 1010 executes state changes 1020 based on the change order 1050 and triggers downstream jobs corresponding to previously executed reader jobs 1024, 1034, 1044 and the snapshots generated thereby. Triggering downstream jobs includes loopback 1026, track update 1036 and cell streaming 1046.
[0199] The above embodiments use a map building system including a map manager, state changes, and change order to facilitate asynchronous parallel map building to synchronize maps across system components. This reduces system latency / lag, thereby achieving improved real-time system performance.
[0200] Cell Stream Transmission
[0201] Fig.24 and Fig.25 A cell streaming method in a multi-component map building system is depicted. The method addresses the memory limitations of the wearable device by storing a smaller map 1100 in the wearable device and a larger map (not shown) (including the smaller map 1100) in a power pack or cloud processing component (not shown).
[0202] like Fig.24, the larger map has been divided into a plurality of map sub-units / cells 1110, 1112, and the smaller map 1100 includes a first subset of the plurality of map sub-units 1110, 1112. The smaller map 1100 is a 3×3×1 sub-unit 3D array, with each sub-unit 1110, 1112 having a size of approximately 7m×7m×7m. In other embodiments, the smaller map (not shown) may be a 10×10×1 sub-unit 3D array or a 6×6×3 3D sub-unit array. The smaller map 1100 also includes a central map sub-unit 1112 corresponding to the current location of the map building system / user.
[0203] Fig.25 The cell flow is shown as the map building system / user moves from the previous central map subunit 1112 to the updated central map subunit 1112'. Fig.25 The updated smaller map 1100' depicted in FIG. 1 includes a greater variety of map subunits 1110, 1112, 1112', 1114, 1116. These new map subunits 1112', 1114, 1116 are Fig.24 110, 1112 are identical. For example, each of the map subunits 1110, 1112, 1112', 1114, 1116 has a size of approximately 7m x 7m x 7m. When the map building system / user moves to the updated central map subunit 1112', the map subunit 1114 is more than one map subunit away from the map building system / user. Therefore, a portion of the first subset of the plurality of map subunits, the portion defining the map subunits 1114, is deleted from the memory of the wearable device. Fig.24 Smaller map 1100 in.
[0204] The wearable device also receives (e.g., from a power pack) a second subset of the plurality of map subunits 1116 such that the remaining map subunits 1110, 1112 from the first subset of the plurality of subunits and the second subset of the plurality of map subunits 1116 completely surround the updated central map subunit 1112'. As described above, by deleting the map subunit 1114, the second subset of the plurality of map subunits 1116 is stored in the memory of the wearable device becoming available.
[0205] Fig.24 and Fig.25The method shown in allows a wearable device to access map building subunits required for various map building functions while minimizing the demand on limited memory in the wearable device by unloading unneeded map building subunits before loading new map building subunits. The method maintains at least one map subunit in each direction in the XY plane from a central map unit. Thus, map subunits can be exchanged while maintaining certain map data used for map building functions (e.g., tracking) during the exchange process. Although Fig.24 and Fig.25 The movement depicted in is a diagonal movement, but any other movement in the XY plane can be followed by an update of the smaller map in the wearable device according to the above method.
[0206] use Fig.24 and Fig.25 The method shown in or other similar methods, the sparse map can be expressed as: SparseMap(t) = {Cell1(t), Cell2(t), ..., Cell N (t)}. As described above, a state change can be applied to a sparse map, resulting in SparseMap(t+1)=SparseMap(t) "+"StateChange(t). When the "+" operator has a distributive attribute, a state change can be applied to a sparse map as follows:
[0207] SparseMap(t)“+”State Change(t)={Cell1(t), Cell2(t),…,Cell N (t)}“+”State Change(t)
[0208] ={Cell1(t)“+”State Change(t), Cell2(t)“+”State Change(t),…,Cell N (t)“+”State Change(t)}
[0209] ={Cell1(t+1), Cell2(t+1),…, Cell N (t+1)}
[0210] =SparseMap(t+1).
[0211] This way, state changes can be applied at the individual level without having access to the state of all cells in the full sparse map (e.g., in wearable devices).
[0212] During cell streaming, a snapshot of the map subunit / cell is copied from the power pack to the wearable device. Due to the size of some maps, by the time the last map subunit is received at the wearable device, the first map subunit received by the wearable device may be out of date with respect to the map building process performed at the power pack. Using the per-cell state change approach, state changes can be applied to the first map units received by the wearable device to bring them up to date.
[0213] The cell streaming method described above also allows the wearable device to perform map-related functions while storing only a portion of a complete sparse map of the local environment. By streaming map sub-units from the power pack based on the map building system / user's movements, memory usage will be kept to a minimum in the wearable device while retaining enough map data for the wearable device to perform many map functions.
[0214] Zero-copy data transfer
[0215] Fig.26 and Fig. 27 Describes a data structure for zero-copy data transfer over a relatively low-bandwidth communication channel such as a USB connector. Zero-copy data transfer can be used Fig.24 and Fig.25 and the cell exchange mechanism described above to reduce latency in the USB communication channel between the wearable device and the power pack. Using the data structure shown and zero-copy data transfer, the USB communication channel can write data directly to the system component memory to be used in various map building operations. This is an improvement over current systems, in which data from the USB communication channel is first written to a buffer in the system component and then written from the buffer to the system component memory. In this way, zero-copy data transfer reduces memory usage and processor usage for copying data.
[0216] Zero-copy data transfer uses ownership and cell state data indicators / tags. Fig.26 As shown in the , the ownership of a particular cell can be "NONE", "MAP", or "USB". Similarly, the status of a particular cell can be "INVALID", "MAP", or "USB". A map can be an array of cells, each with a separate owner and status. The map can also include enough cells to cover transient situations when cells enter and leave the map.
[0217] Initially, a cell has a status of "NONE" and "INVALID". When a cell is transferred over a USB connection, its ownership is set to "USB" or "Comm2" and its status is set to "USB". When the transfer is complete and the cell is being used by the system, both its ownership and status are set to "MAP". Cells may be accessed through various map building operations / jobs using readers and writers. Only a single instance of a writer may be allowed in a map building system. Multiple instances of readers may be allowed in a map building system, but simultaneous reading and writing are not allowed.
[0218] When a particular map operation / job begins, the framework will pass readers and / or writers to the corresponding system components as appropriate. The cell list includes pointers to all cells currently owned by the map and a count of those cells. At the end of the map operation / job, the writer will populate the request array. The request array instructs the framework to perform one of three functions on each cell of the map. An invalid request marks the cell as ownership "NONE" and a status of "INVALID". A send request marks the cell to be transferred as ownership "USB" or "Comm2" and a status of "USB". After the transfer is complete, a send request marks the cell as ownership "NONE" and a status of "INVALID". A debug request sends a cell without marking it as "INVALID". In this way, the system component does not change the cell during debugging.
[0219] Ownership can be transferred between the USB communication channel ("USB") and the system components ("MAP") operating on the cells. Since ownership can be changed by simple pointer operations without copying the data in the cells, changing ownership of cells to be transferred is relatively inexpensive to the processor and memory.
[0220] System Architecture Overview
[0221] Fig.28 1 is a block diagram of an illustrative computing system 1200 suitable for implementing embodiments of the present disclosure. The computer system 1200 includes a bus 1206 or other communication mechanism for communicating information that interconnects subsystems and devices such as a processor 1207, system memory 1208 (e.g., RAM), static storage devices 1209 (e.g., ROM), disk drives 1210 (e.g., magnetic or optical), communication interfaces 1214 (e.g., a modem or Ethernet card), a display 1211 (e.g., a CRT or LCD), input devices 1212 (e.g., a keyboard), and cursor controls.
[0222] According to one embodiment of the present disclosure, the computer system 1200 performs specific operations by executing one or more sequences of one or more instructions contained in the system memory 1208 by the processor 1207. Such instructions can be read into the system memory 1208 from another computer-readable / usable medium (such as a static storage device 1209 or a disk drive 1210). In alternative embodiments, hard-wired circuits can be used instead of software instructions or in combination with software instructions to implement the present disclosure. Therefore, the embodiments of the present disclosure are not limited to any specific combination of hardware circuits and / or software. In one embodiment, the term "logic" shall mean any combination of software or hardware used to implement all or part of the present disclosure.
[0223] As used herein, the term "computer-readable medium" or "computer-usable medium" refers to any medium that participates in providing instructions to processor 1207 for execution. Such media can take many forms, including, but not limited to, non-volatile media and volatile media. Non-volatile media include, for example, optical or magnetic disks, such as disk drive 1210. Volatile media include dynamic memory, such as system memory 1208.
[0224] Common forms of computer readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tapes, any other physical media with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM (e.g., NAND flash, NOR flash), any other memory chip or cassette, or any other medium from which a computer can read.
[0225] In an embodiment of the present disclosure, the execution of the instruction sequence for practicing the present disclosure is performed by a single computer system 1200. According to other embodiments of the present disclosure, two or more computer systems 1200 coupled by a communication link 1215 (e.g., a LAN, PTSN, or wireless network) can cooperate with each other to execute the instruction sequence required to practice the present disclosure.
[0226] Computer system 1200 can send and receive messages, data and instructions, including programs, i.e., application code, via communication link 1215 and communication interface 1214. The received program code can be executed by processor 1207 upon receipt, and / or stored in disk drive 1210 or other non-volatile storage device for later execution. Database 1232 in storage medium 1231 can be used to store data accessible by system 1200 via data interface 1233.
[0227] The present disclosure includes methods that can be performed using subject equipment. The method may include the action of providing such suitable equipment. Such provision may be performed by a user. In other words, the "providing" action only requires the user to obtain, access, approach, locate, set, activate, power on or otherwise act to provide the necessary equipment in the subject method. The method described here can be performed in any order of logically possible narrated events and the order of narrated events.
[0228] Exemplary aspects of the present disclosure and details of material selection and manufacturing have been described above. As for other details of the present disclosure, they can be understood in conjunction with the patents and publications cited above and the knowledge generally known or understood by those skilled in the art. With respect to the method-based aspects of the present disclosure, the same applies in terms of additional actions that are generally or logically adopted.
[0229] In addition, although the present disclosure has been described with reference to several examples that optionally incorporate various features, the present disclosure is not limited to the examples described or indicated with respect to each variant of the present disclosure. Various changes may be made to the described disclosure and its equivalents (whether referenced herein or not included for the sake of brevity) without departing from the true spirit and scope of the present disclosure. In addition, where a range of values is provided, it is understood that each intermediate value between the upper and lower limits of the range and any other stated value or intermediate value in the stated range is included in the present disclosure.
[0230] Likewise, it is contemplated that any optional features of the described inventive variations may be set forth or claimed independently or in combination with any one or more features described herein. For a single item, the possibility of multiple identical items is included. More specifically, as used herein and in the claims associated therewith, the singular forms "one," "an," "said," and "the" include plural references unless otherwise specifically stated. In other words, the use of articles allows "at least one" of the subject items in the above description and claims associated with the present disclosure. It should also be noted that such claims may be written to exclude any optional elements. In this way, the statement is intended to serve as a prior basis for the use of exclusive terms such as "only," "only" and the like in conjunction with the narration of the claim elements, or for the use of "negative" limitations.
[0231] In the absence of such exclusive terms, the term "comprising" in the claims associated with the present disclosure shall allow for the inclusion of any additional elements, regardless of whether a given number of elements are recited in such claims, or the addition of features may be considered to modify the nature of the elements presented in such claims. Except as expressly defined herein, all technical and scientific terms used herein shall be given as broad a meaning as commonly understood while maintaining claim validity.
[0232] The breadth of the present disclosure is not limited to the examples and / or subject description provided, but rather is limited only by the scope of the claim language associated with the present disclosure.
[0233] In the foregoing description, the present disclosure has been described with reference to specific embodiments of the present disclosure. However, it will be apparent that various modifications and changes may be made thereto without departing from the broader spirit and scope of the present disclosure. For example, the above-described process flow is described with reference to a specific ordering of process actions. However, the order of many of the described process actions may be changed without affecting the scope or operation of the present disclosure. Therefore, the description and drawings should be considered illustrative rather than restrictive.
Claims
1. A computer-implemented method for updating a point map on a system having a first component, a second component, and a third component that are communicatively coupled, the method comprising: The first component performs a first process on the point map in a first map state to generate a first map change, wherein the first map state is a state of the point map; The second component performs a second process on the point map in the first map state to generate a second map change; The second component applies the second map change to the point map in the first map state to generate a first updated point map in a second map state, wherein the second map state is a state of the first updated point map; The first component sends the first map change to the second component; The second component applies the first map change to the first update point map in the second map state to generate a second update point map in a third map state, wherein the third map state is a state of the second update point map; The third component performs a third process on the point map in the first map state to generate a third map change; The first component and the second component send the first map change and the second map change to the third component respectively; as well as The third component applies the third map change to the second update point map in the third map state to generate a third update point map in a fourth map state.
2. The method according to claim 1, further comprising: The second component sends the second map change to the first component; The first component applies the second map change to the point map in the first map state to generate the first updated point map in the second map state; and The first component applies the first map change to the first update point map in the second map state to generate the second update point map in the third map state.
3. The method according to claim 2, further comprising: The first component generates a map change sequence based on the first map change and the second map change; as well as The first component applies the second map change to the point map in the first map state before applying the first map change to the updated point map in the second map state based on the map change order.
4. The method according to claim 2, wherein: The first component applying the second map change to the point map in the first map state includes: the first component generating a pre-changed first map state, and The first component applying the first map change to the first update point map in the second map state includes: the first component generating a pre-changed second map state.
5. The method according to claim 1, wherein: The second component applying the second map change to the point map in the first map state includes: the second component generating a pre-changed first map state, and The second component applying the first map change to the first update point map in the second map state includes: the second component generating a pre-changed second map state.
6. The method according to claim 1, further comprising: The first component generates a map change sequence based on the first map change and the second map change; as well as The first component sends the map change sequence to the second component.
7. The method according to claim 6, further comprising: The second component applies the second map change to the point map in the first map state before applying the first map change to the updated point map in the second map state based on the map change order.
8. The method according to claim 1, wherein: The first processing is selected from the group consisting of: tracking, map point deletion, VIO, map building, insertion of new keyframes / keybinding frames, insertion of new map points, insertion of new observations, bundle adjustment, modification of the geometry of sparse maps, online calibration, relocalization, adding new orb features; adding new BoW indexes; loop closure; modifying the geometry of sparse maps; map merging; and joint dense optimization.
9. The method according to claim 1, wherein: The point diagram corresponds to the physical location where the system is located.
10. The method according to claim 9, wherein: The physical location is a room.
11. The method according to claim 9, wherein: The physical location is a cell of a room.
12. The method according to claim 1, wherein: The dot plot is part of a larger dot plot.
13. The method according to claim 1, wherein: The first component is a head mounted audio-visual display component of a mixed reality system, and Wherein, the second component is a torso-mounted processing component of the mixed reality system. 14 . The method of claim 1 , further comprising the second component reconciling the second map change with the first map state before applying the second map change to the point map in the first map state.
15. The method according to claim 1, wherein: The first component is a head-mounted audio-visual display component of a mixed reality system, wherein the second component is a torso-mounted processing component of the mixed reality system, and Wherein, the third component is a cloud-based processing component. 16 . The method of claim 1 , further comprising the second component receiving the first map change from the first component before generating the second map change.
17. The method of claim 1, the second component comprising a removable medium for data storage.
18. The method according to claim 1, wherein: The second processing is selected from the group consisting of: tracking, map point deletion, VIO, map building, insertion of new keyframes / keybound frames, insertion of new map points, insertion of new observations, bundle adjustment, modification of sparse map geometry, online calibration, relocalization, adding new orb features, adding new BoW indexes, loop closure, modification of sparse map geometry, map merging, and joint dense optimization.
