Inertial Data Management for Extended Reality on Mobile Platforms

Abstract

The present disclosure relates to inertial data management for extended reality for mobile platforms. Specific implementations of the subject technology provide extended reality display devices that can be used on and / or off of mobile platforms. The present invention discloses systems and methods for separating the motion of the mobile platform from other motions of the device so that virtual content can be displayed without erroneous motion caused by the motion of the mobile platform. The subject technology can provide extended reality scenery in any suitable movable platform, such as in a car, watercraft, aircraft, train, or any other vehicle.

Description

[0001] This application is a divisional application of the Chinese invention patent application with application date of September 18, 2021, application number 202111097591.5, and invention name “Inertial Data Management for Extended Reality on Mobile Platforms”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 080,623, filed on September 18, 2020, entitled “Extended Reality For Moving Platforms,” the disclosure of which is hereby incorporated herein in its entirety. Technical Field

[0004] This specification generally relates to extended reality scenes. Background Art

[0005] An electronic device may display and modify content based on the orientation and / or motion of the device. However, determining the orientation and / or motion of a device, particularly a portable electronic device that moves freely within a physical environment, may be challenging in some cases. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Certain features of the subject technology are set forth in the appended claims. However, for purposes of explanation, several embodiments of the subject technology are set forth in the following drawings.

[0007] Figure 1A to Figure 1B Depicted are exemplary systems for various extended reality techniques according to one or more implementations.

[0008] Figure 2 An exemplary architecture is shown that may implement the subject technology in accordance with one or more specific implementations of the subject technology.

[0009] Figure 3 An example of a physical set of an electronic device including a mobile platform is shown in accordance with an implementation of the subject technology.

[0010] Figure 4 Examples are shown in which an electronic device according to an implementation of the subject technology moves with and relative to a mobile platform.

[0011] Figure 5 An example of anchoring virtual content to a mobile platform according to an implementation of the subject technology is shown.

[0012] Figure 6 An exemplary diagram of an electronic device operating when disposed on an aircraft in accordance with an implementation of the subject technology is shown.

[0013] Figure 7 Aspects of various simultaneous localization and mapping (SLAM) states of an electronic device in accordance with implementations of the subject technology are shown.

[0014] Figure 8 Shows a specific implementation of the subject technology Figure 7 Additional aspects of the third SLAM state.

[0015] Fig. 9 Shows a specific implementation of the subject technology Figure 7 Additional aspects of the second SLAM state.

[0016] Fig.10 Shows a specific implementation of the subject technology Figure 7 Additional aspects of the first SLAM state.

[0017] Fig.11 A flow chart illustrating exemplary operations that may be performed to operate an electronic device in accordance with an implementation of the subject technology.

[0018] Fig.12 Flowcharts illustrating additional operations that may be performed to operate an electronic device in accordance with implementations of the subject technology. DETAILED DESCRIPTION

[0019] The specific embodiments shown below are intended to be descriptions of various configurations of the subject technology and are not intended to represent the only configuration that the subject technology can be practiced. The accompanying drawings are incorporated herein and constitute a part of the specific embodiments. The specific embodiments include specific details intended to provide a thorough understanding of the subject technology. However, the subject technology is not limited to the specific details set forth herein, but can be practiced using one or more other specific implementations. In one or more specific implementations, structures and components are shown in block diagram form to avoid blurring the concept of the subject technology.

[0020] The physical environment refers to the physical world that people can sense and / or interact with without the help of electronic devices. The physical environment may include physical features, such as physical surfaces or physical objects. For example, the physical environment corresponds to a physical park including physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell. In contrast, the extended reality (XR) environment refers to a fully or partially simulated environment that people sense and / or interact with via electronic devices. For example, the XR environment may include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, and the like. In the case of an XR system, a subset of a person's physical movements or a representation thereof is tracked, and in response, one or more features of one or more virtual objects simulated in the XR system are adjusted in a manner that complies with at least one physical law. For example, the XR system can detect head movement, and in response, adjust the graphical content and sound field presented to the person in a manner similar to the way such views and sounds change in the physical environment. As another example, an XR system may detect movement of an electronic device (e.g., a mobile phone, tablet, laptop, etc.) presenting an XR environment, and in response, adjust the graphical content and sound fields presented to a person in a manner similar to how such views and sounds would change in a physical environment. In some cases (e.g., for accessibility reasons), an XR system may adjust features of graphical content in an XR environment in response to representations of physical movement (e.g., voice commands).

[0021] There are many different types of electronic systems that enable people to sense and / or interact with various XR environments. Examples include head-mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smartphones, tablets, and desktop / laptop computers. A head-mounted system may have an integrated opaque display and one or more speakers. Alternatively, a head-mounted system may be configured to accept an external opaque display (e.g., a smartphone). A head-mounted system may incorporate one or more imaging sensors for capturing images or video of a physical environment, and / or one or more microphones for capturing audio of a physical environment. Instead of an opaque display, a head-mounted system may have a transparent or translucent display. A transparent or translucent display may have a medium through which light representing an image is directed to a person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In some implementations, a transparent or translucent display can be configured to selectively become opaque. A projection-based system can employ retinal projection technology that projects a graphic image onto a person's retina. The projection system can also be configured to project a virtual object into a physical environment, such as a hologram or onto a physical surface.

[0022] Specific implementations of the subject technology described herein provide an XR system that uses an electronic device located on or near a mobile platform to display virtual content. Since the electronic device that displays the virtual content typically tracks its own motion in a physical scene in order to render the virtual content at a fixed location in a virtual or mixed reality scene, the motion of the electronic device caused by the motion of the mobile platform can cause undesirable errors in the display of the virtual content.

[0023] For example, an electronic device that may be carried or worn by a user (e.g., on the head) displays a virtual object to appear at a stationary position on the floor next to the user sitting on a train. When the user turns the device to look around an extended reality scenery including the train and the virtual object, the motion of the electronic device is detected and used to modify the displayed position of the virtual object on the display of the electronic device so that the virtual object appears to be stationary at that position on the floor. However, when the train begins to move, the electronic device also detects the train motion and may erroneously interpret the train motion as motion of the device relative to the position where the virtual object is displayed. In this scenario, the electronic device may erroneously move the position of the virtual object on the display of the electronic device to account for the motion of the train, causing the virtual object to appear to erroneously slide back down the aisle of the train.

[0024] In one or more specific implementations of the subject technology, systems, devices, and methods are provided that manage the use of inertial data from inertial sensors, such as one or more sensors of an inertial measurement unit (IMU), to control a device based on the orientation and / or motion of the device, whether the device is stationary relative to the ground, on a platform moving at a constant velocity relative to the ground, or on a platform having a varying velocity relative to the ground (e.g., accelerating or decelerating).

[0025] For example, an XR system may be provided that can detect and account for motion of a mobile platform. For example, an electronic device may detect that it is located on a mobile platform and control display of virtual content based on (i) motion of the mobile platform and / or (ii) motion of a device on the mobile platform. For example, the electronic device may control display of virtual content by using optical tracking data when the mobile platform is accelerating or decelerating (e.g., and reducing and / or otherwise managing use of other sensor data, such as some or all inertial data).

[0026] For example, the electronic device can manage the use of inertial data (motion data) from one or more inertial sensors (in some operating scenarios where the electronic device is located on a mobile platform) by continuing to use the inertial data but reducing the weight (e.g., treating the inertial data as higher uncertainty data compared to the processing of the inertial data when the electronic device is not located on the mobile platform). In this way, inertial data such as IMU measurements can be used in different ways according to the motion state of the mobile platform on which the device is set. In one or more specific implementations, the weights can also be changed based on the scene profile in which the physical environment of the electronic device is set. In various operating scenarios, the weights applied to the inertial data in the cost function optimization can be changed according to the platform motion from a set of initial weights applied when the electronic device is stationary or located on a non-mobile platform or a mobile platform in a constant speed motion state. For example, the weights can be reduced to zero based on the platform motion (e.g., during a time of high disturbance motion of the mobile platform) or reduced to any weight value between the initial value and zero for "milder" motion conditions of the mobile platform.

[0027] In one or more specific implementations, the electronic device may use a first SLAM system (which uses visual data from an image sensor and inertial data from an inertial sensor) to detect motion of a mobile platform (e.g., variable speed motion, such as accelerated motion or decelerated motion) (e.g., by detecting differences between the visual data and the inertial data of the first SLAM system), and use a second SLAM system (e.g., a pure visual SLAM system that does not incorporate inertial data from the inertial sensor) to control display of virtual content during the detected variable speed motion. During the variable speed motion and while using the pure visual SLAM system to control the display of virtual content, the electronic device may continue to use at least some inertial data (e.g., together with the pure visual SLAM system) to monitor whether the motion of the movable platform has changed from a variable speed motion state to a constant speed motion state (e.g., by comparing some or all of the inertial data with motion information based on the visual data), and may return to using the first SLAM system when constant speed platform motion or platform motion stops is detected based on this monitoring. In one or more specific implementations, in order to avoid high-frequency switching between the first SLAM system and the second SLAM system, the electronic device may modify the operation of the first SLAM system (e.g., by de-weighting the inertial data used by the first SLAM system) within a certain period of time (e.g., between one second and three seconds) before switching to the second SLAM system to confirm the detected change speed movement, and / or may operate the first SLAM system and the second SLAM system simultaneously within a certain period of time (e.g., between one second and three seconds) before switching back to the first SLAM system from the second SLAM system. In this way, the electronic device can process inertial data in various ways of operating the electronic device under various motion states of the movable platform on which the electronic device is set.

[0028] Figure 1A and Figure 1B An exemplary system 100 for use with various extended reality and / or other technologies is depicted.

[0029] In some examples, such as Figure 1A As shown, system 100 includes electronic device 100a. Electronic device 100a includes various components, such as processor 102, RF circuit 104, memory 106, image sensor 108, orientation sensor 110, microphone 112, position sensor 116, speaker 118, display 120 and touch-sensitive surface 122. These components are optionally communicated via communication bus 150 of electronic device 100a.

[0030] In some examples, elements of system 100 are implemented in a base station device (e.g., a computing device such as a remote server, a mobile device, or a laptop), and other elements of system 100 are implemented in a second device (e.g., a head mounted device). In some examples, electronic device 100a is implemented in the base station device or the second device.

[0031] like Figure 1B As shown, in some examples, system 100 includes two (or more) devices in communication, such as through a wired connection or a wireless connection. Electronic device 100b (e.g., base station device) includes processor 102, RF circuit 104 and memory 106. These components are optionally communicated through the communication bus 150 of electronic device 100b. Electronic device 100c (e.g., smart phone, tablet computer or wearable device such as smart watch or head-mounted device) includes various components, such as processor 102, RF circuit 104, memory 106, image sensor 108, orientation sensor 110, microphone 112, position sensor 116, speaker 118, display 120 and touch-sensitive surface 122. These components are optionally communicated through the communication bus 150 of electronic device 100c.

[0032] System 100 includes processor 102 and memory 106. Processor 102 includes one or more general purpose processors, one or more graphics processors, and / or one or more digital signal processors. In some examples, memory 106 is one or more non-transitory computer-readable storage media (e.g., flash memory, random access memory) storing computer-readable instructions configured to be executed by processor 102 to perform the techniques described below.

[0033] System 100 includes RF circuitry 104. RF circuitry 104 optionally includes circuitry for communicating with electronic devices, networks (such as the Internet, an intranet), and / or wireless networks (such as cellular networks and wireless local area networks (LANs)). RF circuitry 104 optionally includes circuitry for using near field communications and / or short range communications (such as ) circuit for communication.

[0034] System 100 includes display 120. Display 120 may have an opaque display. Display 120 may have a transparent or translucent display, which may be combined with a substrate through which light representing an image is directed to the individual's eyes. Display 120 may be combined with LEDs, OLEDs, digital light projectors, laser scanning light sources, liquid crystal on silicon, or any combination of these technologies. The substrate that transmits light may be an optical waveguide, an optical combiner, an optical reflector, a holographic substrate, or any combination of these substrates. In one example, a transparent or translucent display may be selectively switched between an opaque state and a transparent or translucent state. Other examples of display 120 include a head-up display, a car windshield capable of displaying graphics, a window capable of displaying graphics, a lens capable of displaying graphics, a tablet computer, a smart phone, and a desktop or laptop computer. Alternatively, system 100 may be designed to receive an external display (e.g., a smart phone). In some examples, system 100 is a projection-based system that uses retinal projection to project images onto an individual's retina or to project virtual objects into a physical set (e.g., onto a physical surface or as a hologram).

[0035] In some examples, system 100 includes a touch-sensitive surface 122 for receiving user inputs such as tap inputs and swipe inputs. In some examples, display 120 and touch-sensitive surface 122 form a touch-sensitive display.

[0036] System 100 includes an image sensor 108. Image sensor 108 optionally includes one or more visible light image sensors operable to acquire images of physical elements from a physical set, such as a charge coupled device (CCD) sensor and / or a complementary metal oxide semiconductor (CMOS) sensor. The image sensor also optionally includes one or more infrared (IR) sensors, such as a passive IR sensor or an active IR sensor, for detecting infrared light from the physical set. For example, an active IR sensor includes an IR emitter, such as an IR point emitter, for emitting infrared light into the physical set. Image sensor 108 also optionally includes one or more event cameras configured to capture movement of physical elements in the physical set. Image sensor 108 also optionally includes one or more depth sensors configured to detect the distance of the physical elements from system 100. In some examples, system 100 uses a combination of CCD sensors, event cameras, and depth sensors to detect physical sets around system 100. In some examples, image sensor 108 includes a first image sensor and a second image sensor. The first image sensor and the second image sensor are optionally configured to capture images of physical elements in the physical set from two different perspectives. In some examples, the system 100 uses the image sensor 108 to receive user input, such as gestures. In some examples, the system 100 uses the image sensor 108 to detect the position and orientation of the system 100 and / or the display 120 in the physical set. For example, the system 100 uses the image sensor 108 to track the position and orientation of the display 120 relative to one or more fixed elements in the physical set.

[0037] In some examples, system 100 includes microphone 112. System 100 uses microphone 112 to detect sounds from the user and / or the user's physical set. In some examples, microphone 112 includes a microphone array (including multiple microphones), which optionally operate in coordination to identify ambient noise or locate a sound source in the space of the physical set.

[0038] System 100 includes an orientation sensor 110 for detecting orientation and / or movement of system 100 and / or display 120. For example, system 100 uses orientation sensor 110 to track changes in position and / or orientation of system 100 and / or display 120, such as relative to physical elements in a physical scene. Orientation sensor 110 optionally includes one or more gyroscopes and / or one or more accelerometers.

[0039] Figure 2An exemplary architecture that may be implemented on an electronic device such as electronic device 100a, electronic device 100b, and / or electronic device 100c according to one or more specific implementations of the subject technology is shown, including hardware components 221 and logic processes 219. For purposes of explanation, Figure 2 The logical process 219 of the architecture is described as consisting of Figure 1A The electronic device 100a of the present invention is implemented, such as by a processor and / or memory of the electronic device; however, appropriate parts of the architecture may be implemented by any other electronic device, including the electronic device 100b and / or the electronic device 100c. However, not all of the components depicted may be used in all specific implementations, and one or more specific implementations may include additional or different components than those shown in the figures. Changes in the arrangement and type of these components may be made without departing from the spirit or scope of the claims listed herein. Additional components, different components, or fewer components may be provided.

[0040] Figure 2 Portions of the architecture's logic process 219 may be implemented in software or hardware, including by one or more processors and memory devices containing instructions that, when executed by the processor, cause the processor to perform the operations described herein. Figure 2 In the example of FIG. 1 , the electronic device 100 a includes a sensor 129 (eg, including Figure 1A and Figure 1B The camera 119 (e.g., implementing one or more of the image sensors 108, orientation sensors 110, and / or position sensors 116, and / or other sensors, such as an inertial measurement unit (IMU) including one or more accelerometers and / or gyroscopes and / or compasses and / or other magnetic sensors and motion sensors), for example, provides sensor data (e.g., depth sensor data from one or more depth sensors, location data such as global positioning system (GPS) data, Wi-Fi location data and / or near field communication location data, and / or device motion data from one or more motion sensors such as accelerometers, gyroscopes, compasses, inertial measurement units (IMUs) including one or more accelerometers and / or gyroscopes and / or compasses, and / or other magnetic sensors and motion sensors) to the motion detection engine 200. The camera 119 (e.g., implementing one or more of the image sensors 108) may also provide images, such as one or more video streams, to the motion detection engine 200. In one or more implementations, the camera 119 may also include one or more event-based sensors that report changes in pixel values ​​rather than the pixel values ​​themselves and may extend the camera sensitivity to a wider range of lighting conditions and provide a higher frame rate for the camera output pixel values.

[0041] The motion detection engine 200 may include one or more simultaneous localization and mapping (SLAM) systems that use sensor data (e.g., depth information, position data, motion data, magnetic data, and / or images) from the sensor 129 and the camera 119 to generate mapping, position, and / or posture information, which may include three-dimensional scene information, such as a three-dimensional map of some or all of the physical environment of the electronic device 100a and / or the device position, rotation, and / or motion (e.g., speed and / or acceleration) within the physical environment. For example, the motion detection engine 200 may include a visual-inertial (VI) SLAM system 287 (also referred to herein as a first SLAM system in some examples) and a pure visual (VO) SLAM system 289 (also referred to herein as a second SLAM system in some examples). The motion detection engine 200 may detect motion of the electronic device 100a (e.g., in one, two, three, four, five, or six dimensions). For example, the motion detection engine 200 may detect up to three degrees of translational motion and / or up to three degrees of rotational motion of the electronic device 100a (e.g., relative to a fixed reference frame, such as a reference frame fixed to the surface of the earth at or near the location of the electronic device, such as Figure 3 , and / or relative to a moving reference frame, such as a reference frame fixed to a movable platform, such as Figure 3 (x',y',z') reference frame).

[0042] Although the motion detection engine 200 Figure 2 200, but the motion detection engine 200 may be implemented as a plurality of separate processes that are executed serially and / or in parallel for detection of device motion and / or motion of a movable platform. Some or all of the operations described in conjunction with the motion detection engine 200 may be performed by the XR application 202, and / or by a rendering engine of computer-generated (CP) content such as the CP rendering engine 223. The motion detection engine 200 may include one or more SLAM systems (e.g., the VI SLAM system 287 and the VO SLAM system 289) for tracking the electronic device 100a relative to a reference frame (e.g., relative to a reference frame corresponding to a movable platform such as Figure 3 The (x', y', z') reference frame shown above or a fixed reference frame such as Figure 3 The motion of one of the (x, y, z) reference frames shown. Figure 2In the example of , the motion detection engine 200 includes a VI SLAM system 287 that receives visual (e.g., image) data from the camera 119 and inertial data (e.g., gyroscope data, accelerometer data, and / or magnetometer data) from the sensor 129, and a VO SLAM system 289 that receives the visual data from the camera 119 and generates an output independent of the inertial data. As described herein, the VI SLAM system 287 and the VO SLAM system 289 can operate together and / or separately to manage the use of inertial data to track the movement of the electronic device 100a relative to the mobile platform in various states of motion of the mobile platform and / or various states of motion of the electronic device 100a itself (e.g., as described below in conjunction with Figures 6 to 12 discussed in further detail).

[0043] like Figure 2 As shown, in one or more specific implementations, the motion detection engine 200 can receive sensor data from one or more external sensors 250. For example, the external sensor 250 can be a motion and / or position sensor implemented as part of a movable platform, such as a motion and / or position sensor implemented as part of a car, airplane, train, ship, or other movable platform. The motion detection engine 200 can receive sensor data from the external sensor 250 and / or motion and / or position information of the movable platform, as determined by processing circuitry at the movable platform.

[0044] like Figure 2 As shown, the XR application 202 may receive environmental information (e.g., including position information, motion information, scene information, etc.) from the motion detection engine 200. The XR application 202 may be a gaming application, a media player application, a content editor application, a training application, a simulator application, or generally any application that displays computer-generated (CP) content or virtual content in a virtual set and / or at a location dependent on a physical set, such as by anchoring the virtual content to an anchor position that is fixed relative to a fixed or moving reference frame in the physical set. In one or more specific implementations, one or more of the XR application 202, the motion detection engine 200, and / or the CP rendering engine may be part of an operating system-level process and / or framework that provides virtual content anchoring functionality.

[0045] The motion detection engine 200, the XR application 202, and / or the CP rendering engine 223 may determine, based on the detected motion of the electronic device, an anchor position of virtual content to be generated by the XR application 202. For example, the electronic device 100a (e.g., the motion detection engine 200) may use one or more of the sensors 129 (e.g., and / or the camera 119) to identify device motion of the electronic device 100a, and may determine that the device motion includes a first component associated with the motion of the mobile platform and a second component separate from the motion of the mobile platform.

[0046] One or more combinations of cameras and / or sensors on the electronic device itself and / or on the mobile platform may be used to detect and / or separate the first and second components of the motion of the device.

[0047] The electronic device 100a may determine an anchor position that is fixed relative to the mobile platform. The determined anchor position may be determined by the XR application 202 and / or the CP rendering engine 223 and / or used to display virtual content anchored to the anchor position that is fixed relative to the mobile platform using at least a second component of device motion that is separate from the motion of the mobile platform. For example, the second component of the device motion (e.g., the motion of the device relative to the mobile platform) may be used to track the position of the electronic device 100a relative to the determined anchor position. Virtual content anchored to the anchor position may be displayed by using the CP rendering engine 223 and using the display 225 (e.g., Figure 1A and Figure 1B The display 120 of the mobile platform may be configured to display rendered virtual content (eg, one or more virtual objects or an entire virtual scenery) anchored to an anchor position that is fixed relative to the mobile platform.

[0048] In any of the various implementations, the motion detection engine 200, the XR application 202, and / or the CP rendering engine 223 may generate an anchor position that is fixed relative to the movable platform.

[0049] For example, once CP content (e.g., a virtual cup, a virtual document, a virtual TV screen, a virtual cinema screen, a virtual keyboard, a virtual set, etc.) has been generated by the XR application 202, the CP content may be provided to the CP rendering engine 223, such as Figure 2As shown. Environmental information such as a depth map of the physical scenery may also be provided to the CP rendering engine 223. The CP rendering engine 223 may then render the CP content from the XR application 202 for display on the display 225 of the electronic device 100a. The CP content is rendered for display at an appropriate location on the display 225 so as to appear associated with an anchor position (e.g., provided by the motion detection engine 200). The display 225 may be, for example, an opaque display, and the camera 119 may be configured to provide a video pass-through feed to the opaque display. The CP content may be rendered for display at a location on the display corresponding to the displayed location of the anchor position in the video pass-through. As another example, the display 225 may be a transparent or translucent display. The CP content may be rendered for display at a location on the display corresponding to a direct view of the anchor position through the transparent or translucent display. Although Figure 2 The example shows a CP rendering engine 223 separate from the XR application 202, but it should be understood that in some specific implementations, the XR application 202 can render CP content for display on the display 225 without using a separate CP rendering engine 223.

[0050] Figures 3 to 5 An example of displaying virtual content by an electronic device at least partially coupled to a movable platform that is moving (which may be referred to as a mobile platform) according to aspects of the present disclosure is shown.

[0051] exist Figure 3 In the example of , the physical setting 300 of the electronic device such as the electronic device 100a includes a movable platform 304. The movable platform 304 can be implemented as, for example, a vehicle (e.g., a car, a bus, a truck, a golf cart, etc.), a train, a water vehicle (e.g., a boat, a ship, a submarine, etc.), an aircraft (e.g., an airplane, a helicopter), a skateboard, a bicycle, an elevator, an escalator, a moving walkway, or any other movable platform. Figure 3 In the example of FIG. 3 , movable platform 304 is moving with motion 322 (eg, speed and direction) relative to physical ground 302 in physical set 300. Physical ground 302 may represent, for example, an electronic device (eg, Figure 3 The physical ground 302 may form the basis of a fixed reference frame (e.g., an (x, y, z) reference frame) relative to which the movable platform 304, the electronic device 100a, and / or other physical objects may move. Figure 3 In the example of FIG. 3 , the physical scenery 300 also includes a physical object 308 that is stationary relative to the physical ground 302 and can be fixed to the physical ground.

[0052] exist Figure 3, electronic device 100a is moving with motion 322 equivalent to motion 322 of movable platform 304. For example, an electronic device such as electronic device 100a may move with movable platform 304 due to coupling 306 between the electronic device and movable platform 304. For example, coupling 306 may include electronic device 100a being coupled to movable platform 304 by being worn or held by a user sitting or standing on the movable platform, or may include other direct or indirect couplings with movable platform 304 (e.g., due to the electronic device resting on a table, chair, or other structure of the movable platform or being mounted or otherwise secured to the structure of the movable platform).

[0053] like Figure 3 As shown, a virtual object 320 may be displayed by an electronic device such as electronic device 100a. Figure 3 In the example of , a virtual object 320 is rendered and displayed by the electronic device 100a so that it appears to a user of the electronic device 100a to be moving with a motion 322 equivalent to the motion 322 of the movable platform (e.g., so as to appear stationary on the movable platform). An electronic device such as the electronic device 100a may, for example, determine that the electronic device is located on a varying speed platform (e.g., by detecting differences between visual data and inertial data from the VI SLAM system 287), and then display the virtual object 320 at a stationary position on (or relative to) the movable platform 304 using the VO SLAM system 289 during the varying speed movement. For example, the electronic device 100a may obtain but ignore some or all inertial data from an inertial sensor of the electronic device 100a when determining where to display the virtual object 320 during the varying speed movement of the movable platform 304. Figure 3 In the example of , the virtual object 320 is displayed as appearing as part of the physical scenery 300. However, this is merely illustrative, and it should be understood that the virtual object 320 may be displayed as appearing at a stationary location in the overall virtual scenery generated by the electronic device 100a, and move with the movable platform 304 (e.g., by managing the use of inertial data as described herein when determining where to display the virtual object 320).

[0054] An electronic device such as electronic device 100a may consider the motion 322 of the electronic device equivalent to the motion 322 of the movable platform by stopping, reducing and / or modifying the use of some or all sensor data and / or sensors affected by the motion of the movable platform. For example, after determining that the electronic device is moving with the movable platform 304 using the electronic device's IMU (e.g., by comparing visual data and inertial data from the VI SLAM system 287), the electronic device such as electronic device 100a may continue to use the electronic device's optical sensor and / or depth sensor to track the motion of the electronic device, while stopping the use of some or all IMU data and / or de-weighting some or all IMU data when platform-related variable speed motion is detected (e.g., in the case where the mobile platform causes vibratory motion of the electronic device).

[0055] The sensor data from the sensors 129 indicating the motion of the platform may include sensor data indicating acceleration and / or deceleration that is not detected in the visual data or optical data from the one or more cameras. Once the motion 322 of the mobile platform has been determined, the electronic device 100a may reduce and / or modify the use of inertial data to determine where and / or how to display virtual content, such as virtual objects 320, in the augmented reality setting.

[0056] In one or more specific implementations, the sensor 129 of the electronic device 100a includes an optical sensor (e.g., an imaging sensor and / or a camera), a depth sensor, and an IMU. The VI SLAM system 287 may be used initially to identify device motion. If the device motion determined using the VI SLAM system 287 is determined to indicate a variable speed motion caused by the coupling 306 of the electronic device 100a to the movable platform 304, the optical sensor and / or the depth sensor may be used and reduced data from the IMU may be used (e.g., some or all sensor data from the IMU data may be ignored, and / or some or all sensors of the IMU may be disabled to prevent the variable speed motion of the movable platform from affecting the display of the virtual content) to display virtual content such as a virtual object 320 anchored to a fixed anchor position relative to the movable platform. In some specific implementations, only a portion of the IMU data corresponding to the device motion may be ignored. For example, in some operating scenarios, only one or a subset of the sensors of the IMU may be used to continue tracking the motion of the electronic device. For example, only a magnetometer, only one or more gyroscopes (e.g., when the motion of the mobile platform is determined to be non-rotational motion), only an accelerometer (e.g., when the motion of the mobile platform is determined to be constant velocity motion), or a combination of these IMU sensors including less than all sensors of the IMU may be used in various operating scenarios. For example, in some operating scenarios, the VO SLAM system 289 may be used to control the device (e.g., to control the display of virtual content), and the inertial sensor data and / or the VI SLAM system 287 may be temporarily used only to determine when the variable velocity motion of the movable platform 304 has ended. The VI SLAM system 287 may then be used to track the position and / or orientation of the electronic device 100a relative to the movable platform 304 during the constant velocity motion of the movable platform 304.

[0057] exist Figure 3 In the example of FIG. 3 , the motion 322 of the electronic device 100a is the same as and entirely due to the motion 322 of the movable platform 304 (e.g., the electronic device 100a is fixed or stationary relative to the movable platform even though the system is moving relative to the physical ground 302). However, in other scenarios, the electronic device 100a may move relative to the mobile platform in addition to being moved by the mobile platform.

[0058] For example, Figure 4 The scene shows that the electronic device 100a is moving with a motion 400, which includes a first component (e.g., motion 322 caused by the motion 322 of the movable platform 304) and a second component such as additional motion 402. The additional motion 402 may be caused by, for example, a user or wearer of the electronic device 100a walking or otherwise moving around on the movable platform 304. Figure 4In the example of , the additional motion 402 is shown as a linear motion in the same direction as the motion 322. However, in various scenarios, the motion 400 of the electronic device 100a can include various components separate from the motion 322 of the movable platform, such as rotational motion of the electronic device 100a and / or other linear or non-linear translational motion of the electronic device 100a relative to the movable platform and relative to any anchor locations that are fixed relative to the movable platform.

[0059] In one or more embodiments, the VO SLAM system 289 can be used (e.g., using visual data from an optical sensor and / or a depth sensor of the sensor 129) to detect and / or track additional motion 402 such as rotational motion and / or translational motion of the electronic device 100a that is separate from the motion 322 of the mobile platform (such as when a user or wearer looks around the mobile platform and / or walks around on the mobile platform) while the movable platform 304 is in a state of varying speed, so that the virtual object 320 can be displayed at a fixed location on the mobile platform even as the electronic device 100a moves within the physical set 300 with the motion 322 and the additional motion 402.

[0060] In one or more implementations, an electronic device such as electronic device 100a on a mobile platform (such as movable platform 304 when movable platform 304 is in motion) may also use a SLAM system (e.g., VI SLAM system 287 and / or VO SLAM system 289) to track the motion of the electronic device separate from the motion of the mobile platform (e.g., a second component of the motion of the electronic device such as additional motion 402). The SLAM system may include, for example, one or more sensors of the electronic device such as sensor 129. In one or more implementations, the electronic device tracks the position and / or motion of the electronic device relative to movable platform 304 without tracking the motion of the movable platform (e.g., by using VO SLAM system 289 to effectively ignore the motion of the movable platform during the varying speed portion of the motion of the movable platform).

[0061] exist Figure 3 and Figure 4 In the example of , the virtual object 320 is displayed so as to appear to be stationary at a location on or within the movable platform 304.

[0062] Figure 5An example is shown in which a virtual object 320 is stationary relative to a physical object 500 on a movable platform 304. As shown, the physical object 500 moves with a motion 322 that is equivalent to and caused by the motion 322 of the movable platform 304. For example, the physical object 500 can be a structural part of the movable platform itself or can be an object that rests on or within the movable platform and / or is mechanically attached to the movable platform. In one or more specific implementations, as examples, the physical object 500 can be a seat on a train, a structural part of a vehicle, a table on a recreational vehicle (RV), or a door of an airplane (as examples).

[0063] In one or more specific implementations, the electronic device 100a can anchor the virtual object 320 to an anchor position that is fixed relative to the movable platform 304 and / or the physical object 500. The anchoring can also include anchoring the virtual content to a fixed position on the movable platform 304 as the electronic device 100a moves on the mobile platform by tracking the movement and / or orientation of the electronic device 100a using the VI SLAM system 287 during constant speed motion of the movable platform 304 and using the VO SLAM system 289 during phases of varying speed motion of the movable platform 304.

[0064] In one or more specific implementations, tracking the movement and / or orientation of the electronic device 100a may include using a visual-inertial SLAM system (e.g., VI SLAM system 287) of the device to identify the device movement of the electronic device 100a. In one or more specific implementations, the electronic device 100a may determine that the device movement includes a first component associated with the varying speed movement of the mobile platform and a second component separate from the varying speed movement of the mobile platform. For example, the electronic device 100a may identify the difference between the visual information (e.g., the device displacement estimate determined using the time-separated image frames) and the inertial information (e.g., the device displacement estimate determined using one or more inertial sensors in a time period corresponding to the time interval between the time-separated image frames) of the visual-inertial SLAM system 287. In one or more specific implementations, displaying virtual content anchored to a fixed anchor position relative to the movable platform 304 may include stopping the use of the visual-inertial SLAM system 287 and operating the pure visual SLAM system (e.g., VO SLAM system 289) of the device to track the orientation and / or movement of the electronic device 100a used for anchoring.

[0065] In one or more specific implementations, while operating the pure visual SLAM system 289, the electronic device 100a may determine that the motion of the movable platform is at or near a constant value based on a comparison of the gyroscope data (e.g., gyroscope-estimated device rotation) of the pure visual SLAM system 289 with the visual data (e.g., image-based rotation estimation). The electronic device may also temporarily operate both the pure visual SLAM system 289 and the visual inertial SLAM system 287 while comparing the outputs of the pure visual SLAM system 289 and the visual inertial SLAM system 287. The electronic device may also stop the operation of the pure visual SLAM system 289 based on the consistency between the outputs of the pure visual SLAM system 289 and the visual inertial SLAM system 287 (e.g., within a minimum time period that can correspond to at least a minimum number of frames (such as image frames), such as between one second and three seconds), while continuing to operate the visual inertial SLAM system 287.

[0066] In one or more specific implementations, the electronic device 100a may operate the VI SLAM system 287 and / or the VO SLAM system 289 in various motion states of the electronic device 100a. One or more of the various motion states may be caused by the movement of a movable platform (e.g., movable platform 304) on which the electronic device 100a is disposed. For example, Figure 6 An exemplary use case of an electronic device such as electronic device 100a operating during the course of various phases of aircraft flight 1001 is shown.

[0067] As shown, the electronic device 100a may be in a constant speed motion state 1000 (e.g., when the aircraft on which the electronic device is positioned is stationary or traveling on the ground at a constant speed or cruising in the air at a constant speed) or a variable speed motion state 1002 (e.g., when the aircraft on which the electronic device is positioned is accelerating during takeoff, experiencing turbulence, or decelerating for landing) according to different situations. It should also be understood that during any constant speed motion state 1000 and / or any variable speed motion state 1002 of the aircraft, the electronic device 100a may have its own motion state relative to the aircraft (e.g., relative to the aircraft, the electronic device may be stationary, moving at a constant translational or rotational speed, or experiencing accelerated translational and / or rotational motion). As shown in FIG. Figure 6As indicated, the electronic device 100a may operate the VI SLAM system 287 during a constant speed motion state 1000 of an aircraft on which the electronic device is set (e.g., and control device operations based on the output of the VI SLAM system, such as the display of virtual content anchored to a fixed location on the aircraft), and may operate the VO SLAM system 289 during a variable speed motion state 1002 of the aircraft (e.g., or another movable platform in other examples) (e.g., and control device operations based on the output of the VO SLAM system, such as the display of virtual content anchored to a fixed location on the aircraft) to track the position, orientation and / or motion of the electronic device relative to the aircraft and / or control other device operations during various motion states of the aircraft.

[0068] like Figure 6 As indicated, the aircraft on which the electronic device is disposed may also undergo one or more transition states 1014, in which the aircraft on which the electronic device is disposed changes from one motion state (e.g., one of constant speed motion or variable speed motion) to another motion state (e.g., the other of constant speed motion or variable speed motion). In one or more specific implementations, the electronic device 100a may temporarily operate both the VI SLAM system 287 and the VO SLAM system 289 during some or all of the transition states 1014. In one or more specific implementations, when operating both the VI SLAM system 287 and the VO SLAM system 289 (e.g., during a transition state 1014 of a mobile platform and / or when it is unclear whether the device is on a mobile platform during a changing speed state of the mobile platform or in any other state on a stationary or constant speed platform), the device may use the output of the VO SLAM system 289 (e.g., using only the output of the VI SLAM system to compare with the output of the VO SLAM system to confirm a switch in the motion state of the platform between a constant speed state and a changing speed motion state or vice versa) to control operation (e.g., displaying virtual content anchored to a fixed location on the aircraft).

[0069] Figure 7 It shows that different situations can be Figure 6 The three SLAM states (e.g., the first SLAM state 1100, the second SLAM state 1114, and the third SLAM state 1102) of an electronic device such as the electronic device 100a used during the constant speed motion state 1000, the variable speed motion state 1002, and the transition state 1014. Figure 7In the example of , a SLAM system whose output is used to control the device (e.g., control the output from the device) is indicated for each state (e.g., a VI SLAM system 287 for a first SLAM state 1100 corresponding to a constant speed motion state 1000 of a platform on which the device is set, and in some operating scenarios, a VO SLAM system 289 for both a third SLAM state 1102 corresponding to a variable speed motion state 1002 and a second SLAM state 1114 that may correspond to a transition state 1014). Figure 7 As shown, the electronic device 100a may also perform operations in each SLAM state (eg, using IMU data at blocks 1122, 1126, and / or 1130) to detect changes in the motion state of the platform on which the electronic device is disposed.

[0070] exist Figure 7 In the example of, in the first SLAM state 1100, the electronic device may operate (box 1128) only the VI SLAM system 287 (e.g., without operating the VO SLAM system 289) when using the VI SLAM system 287 to control device operation (e.g., predicting device posture and / or operating the device based on the predicted device posture), and may (box 1130) determine whether the device is in a bad tracking state (e.g., a state in which the uncertainty of the output of the VI SLAM system 287 is higher than a threshold) and / or whether there is a difference between the vision-based motion data generated by the VI SLAM system 287 and the inertial sensor-based motion data. For example, if the inertial data indicates a changing speed motion of the device, but a comparison of two or more adjacent or nearly adjacent image frames indicates different changing speeds (or no changing speed of the device), the difference may be detected. As shown, in response to the detection of a difference between the visual (image) data and the inertial data of the VI SLAM system 287, the device may switch to a second SLAM state 1114.

[0071] In the second SLAM state 1114, the device continues to operate the VI SLAM system 287 and also temporarily operates the VO SLAM system 289 (e.g., at box 1124), while using the VO SLAM system 289 to control device operations, such as gesture prediction and / or operations based on gesture prediction such as displaying virtual content. As shown in the figure, in the second SLAM state 1114, the electronic device can determine (box 1126) whether the component of the device motion is caused by the accelerated motion of the platform on which the device is set. For example, the electronic device can compare the output of the VI SLAM system 287 with the output of the VO SLAM system 289. In one or more specific implementations, the device can switch back to the first SLAM state 1100 when the output of the VI SLAM system 287 and the output of the VO SLAM system 289 are consistent (e.g., the same within a threshold difference), or can switch to the third SLAM state 1102 when the output of the VI SLAM system 287 and the output of the VO SLAM system 289 are inconsistent (e.g., the difference exceeds the threshold difference).

[0072] As shown, in the third SLAM state 1102 (e.g., when the device is on an accelerating platform), the electronic device may only operate (box 1120) the VO SLAM system 289 and may use the VO SLAM system 289 to control device operations, such as posture prediction and / or operations based on posture prediction such as displaying virtual content. In the third SLAM state 1102, the electronic device may also perform (box 1122) an inertial data verification operation. For example, the inertial data verification operation may include comparing a motion estimate (e.g., translation and / or rotational motion estimate) based on visual data (e.g., image frame differences) with a motion estimate from an inertial sensor (e.g., a rotation estimate from a gyroscope and / or a linear acceleration estimate from an accelerometer). In box 1122, if the motion estimate based on the visual data is consistent with the motion estimate from the inertial sensor (e.g., the same within different thresholds), the electronic device may switch to a second SLAM state 1114 and proceed in the second SLAM state 1114, as described above. In block 1122 , if the electronic device determines that the motion estimates based on the visual data are different (eg, differ by more than a difference threshold), the electronic device may remain in the third SLAM state 1102 .

[0073] On top Figure 7In the description of, for convenience, these three SLAM states are referred to as the first SLAM state 1100, the second SLAM state 1114, and the third SLAM state 1102, and it should be understood that the first SLAM state 1100, the second SLAM state 1114, and the third SLAM state 1102 may occur in any order in various orders according to the movement of the platform on which the device is set. In an exemplary use case, the third SLAM state 1102 may be used when the user first powers on the device or first picks up or uses the device and when an IMU verification operation occurs. In this example, the device may then switch to the second SLAM state 1114 to activate and initialize the VI SLAM system 287. In this example, the device may remain in the second SLAM state 1114 until the output of the VI SLAM system 287 and the output of the VO SLAM system 289 are relatively consistent, and then the device may switch to the initialized first SLAM state 1100 until accelerated motion and / or differential motion is detected, after which the device switches to the second SLAM state 1114 and / or the third SLAM state 1102.

[0074] Figures 8 to 12 It shows that Figure 7 Additional details of the operations performed during the SLAM state. Figures 8 to 12 As shown, the electronic device 100a may also perform operations in each SLAM state within a predetermined time period (e.g., corresponding to a predetermined number of frames), the operations utilizing various amounts of IMU data to help determine whether to switch to another state in the SLAM state. In this way, the electronic device may avoid erroneously switching between SLAM states when the motion state of the platform has not changed, and / or may avoid rapid switching between SLAM states (e.g., on a time scale of less than one second) caused by short-term and / or temporary / instantaneous platform motion changes. Figures 8 to 12 It is shown how strategic management and / or use of inertial data in various SLAM states can facilitate successful device operation even when the device is on a movable platform in various motion states, including constant velocity motion states and varying velocity motion states.

[0075] For example, Figure 8Operations that may be performed by the electronic device 100a when the electronic device is in the third SLAM state 1102 are shown. As shown, in the third SLAM state 1102, the electronic device 100a may perform an inertial validator operation 1200 (e.g., without operating the VI SLAM system 287). The inertial validator operation 1200 may include generating an image-based rotation estimate at block 1202 (e.g., by comparing and / or differentiating image frames from the camera 119, such as the kth frame and the k-1th frame, such as using a visual propagation operator operation such as a perspective n-point (PnP) or a 5-point image processing operation) and an inertial sensor (e.g., gyroscope)-based rotation estimate of the electronic device at block 1204. At block 1206, the electronic device determines whether the image-based rotation estimate of block 1202 and the inertial sensor (e.g., gyroscope)-based rotation estimate of block 1204 are consistent.

[0076] As shown, if the image-based rotation estimate of box 1202 and the inertial sensor (e.g., gyroscope)-based rotation estimate of box 1204 are inconsistent, the electronic device remains in the third SLAM state 1102 (box 1208). As shown, if the image-based rotation estimate of box 1202 and the inertial sensor (e.g., gyroscope)-based rotation estimate of box 1204 are consistent, the electronic device can determine (box 1210) whether the image-based rotation estimate of box 1202 and the inertial sensor (e.g., gyroscope)-based rotation estimate of box 1204 are consistent within at least a predetermined number of frames (e.g., number N) (e.g., corresponding to a predetermined minimum amount of time, such as at least one second, at least two seconds, or at least three seconds). As shown, if the image-based rotation estimate of box 1202 and the inertial sensor (e.g., gyroscope)-based rotation estimate of box 1204 are consistent, but inconsistent within at least a predetermined number of frames, the electronic device remains in the third SLAM state 1102 (box 1208). As shown, if the image-based rotation estimate of block 1202 and the inertial sensor (e.g., gyroscope)-based rotation estimate of block 1204 are consistent and consistent for at least a predetermined number of frames, the electronic device transitions (block 1212) to the second SLAM state 1114 (and activates the VI SLAM system 287, as described above in conjunction with Figure 7). In this way, the electronic device 100a can use a portion of the inertial data in a limited manner while using the VO SLAM system 289 (rather than using the inertial data) to control device operation to determine when the changing motion state of the movable platform on which the electronic device is disposed may have ended. In one or more specific implementations, determining (block 1210) whether the visual measurement values ​​and the inertial measurement values ​​are consistent for a predetermined number of frames (e.g., or a predetermined time period) before switching to the second SLAM state 1114 can help avoid erroneous switching when the device is on (e.g., still on) an accelerating platform and / or rapidly switching between SLAM states due to momentary motion changes of the movable platform.

[0077] like Fig. 9As shown, in the second SLAM state 1114, the electronic device 100a can operate in a dual SLAM mode 1300, in which the device operates both the VI SLAM system 287 and the VO SLAM system 289, and can perform a VI / VO comparison operation 1302. In the VI / VO comparison operation 1302, the electronic device determines whether the output of the VI SLAM system 287 (e.g., device pose estimation or prediction) and the output of the VO SLAM system 289 (e.g., device pose estimation or prediction) are consistent. As shown, if the output of the VI SLAM system 287 and the output of the VO SLAM system 289 are inconsistent, the electronic device switches (box 1304) back to the third SLAM state 1102 and stops the operation of the VI SLAM system 287. As shown, if the output of the VI SLAM system 287 and the output of the VO SLAM system 289 are consistent, the electronic device may determine (box 1306) whether the output of the VI SLAM system 287 and the output of the VO SLAM system 289 are consistent within at least a predetermined number of frames (e.g., number N) and / or a predetermined amount of time. As shown, if the output of the VI SLAM system 287 and the output of the VO SLAM system 289 are consistent, but inconsistent within at least a predetermined number of frames, the electronic device remains (box 1308) in the second SLAM state 1114. As shown, if the output of the VI SLAM system 287 and the output of the VO SLAM system 289 are consistent and consistent within at least a predetermined number of frames, the electronic device transitions (box 1310) to the first SLAM state 1100, and activates the VI SLAM system and stops the operation of the VO SLAM system. In this way, the electronic device 100a can use inertial data in a limited manner while using the VO SLAM system 289 (rather than using inertial data) to control device operation to determine when a changing motion state of a movable platform on which the electronic device is disposed has ended. In one or more specific implementations, determining (block 1306) whether the VI and VO outputs are consistent within a predetermined number of frames (e.g., or a predetermined time period) before switching to a first SLAM state can help avoid erroneous switching when the device is on a varying speed platform and / or rapid switching between SLAM states due to instantaneous motion changes of the movable platform.

[0078] like Fig.10As shown, in the first SLAM state 1100 (e.g., when the VISLAM system 287 is used to control the operation of the device while making full use of inertial data), the electronic device 100a can (e.g., without operating the VO SLAM system 289) generate a vision-based motion estimate 1400 (e.g., by comparing and / or differentiating image frames from the camera 119, such as the kth frame and the k-1th frame) and an inertial sensor (e.g., IMU)-based rotation estimate 1402 of the electronic device. At box 1404, the electronic device determines whether the vision-based motion estimate 1400 and the inertial sensor (e.g., gyroscope)-based motion estimate 1402 are consistent. As shown, if the vision-based motion estimate 1400 and the inertial sensor-based motion estimate 1402 are consistent, the electronic device remains (box 1410) in the first SLAM state 1100. As shown, if the vision-based motion estimation 1400 and the inertial sensor-based motion estimation 1402 are inconsistent, the electronic device may determine (box 1406) whether the vision-based motion estimation 1400 and the inertial sensor-based motion estimation 1402 are inconsistent within at least a predetermined number of frames (e.g., number N).

[0079] As shown, if the vision-based motion estimate 1400 and the inertial sensor-based motion estimate 1402 are inconsistent, but consistent within at least a predetermined number of frames, the electronic device continues (box 1412) to operate the VI SLAM system 287 (and control the device based on the VI SLAM system) in part by de-weighting the inertial sensor measurements (e.g., by assigning high uncertainty to the inertial sensor measurements within the VI SLAM system 287 calculations). As shown, if the vision-based motion estimate 1400 and the inertial sensor-based motion estimate 1402 are inconsistent and inconsistent within at least a predetermined number of frames, the electronic device transitions (box 1408) to the second SLAM state 1114 and activates the VO SLAM system 289. In one or more specific implementations, determining (block 1406) whether the visual measurements and the inertial motion measurements are inconsistent within a predetermined number of frames (e.g., and / or a predetermined time period) before switching to the second SLAM state 1114 can help avoid erroneous switching to the VO SLAM system 289 when the device is on a constant motion platform and / or rapid switching between SLAM states due to instantaneous motion changes of the movable platform. In one or more specific implementations, de-weighting the inertial measurement parameters at block 1412 while continuing to operate the device using the VISLAM system 287 and before switching to the second SLAM state 1114 can strategically reduce the use of inertial data to help reduce errors in device control (e.g., pose estimation and / or pose-based control) due to accelerated motion when the device verifies the presence of accelerated motion.

[0080] Fig.11 A flowchart of an exemplary process 1190 for operating an electronic device according to a specific implementation of the subject technology is shown. For the purpose of explanation, this article mainly refers to Figure 1A , Figure 1B and Figure 2 The process 1190 is described with reference to the electronic device 100a. However, the process 1190 is not limited to Figure 1A , Figure 1B and Figure 2 The electronic device 100a of the present invention, and one or more blocks (or operations) of process 1190 may be performed by one or more other components of other suitable devices (including electronic device 100b and / or electronic device 100c). Further for the purpose of explanation, some blocks of process 1190 are described herein as occurring sequentially or linearly. However, multiple blocks of process 1190 may occur in parallel. In addition, the blocks of process 1190 do not have to be performed in the order shown, and / or one or more blocks of process 1190 do not have to be performed and / or may be replaced by other operations.

[0081] like Fig.11As shown, at block 1192, an electronic device such as electronic device 100a may obtain inertial data from an inertial sensor of the electronic device.

[0082] At block 1194, the electronic device may be operated based on the inertial data while the electronic device is disposed on the mobile platform (e.g., the mobile platform 304) during various states of motion of the mobile platform (e.g., a stationary state, a constant speed motion state, a variable speed motion state, and / or a transition state) and in part by modifying the use of the inertial data according to the current stage of motion of the mobile platform. For example, the following description will be combined with Fig.12 Described are illustrative operations that may be performed to operate the electronic device based on inertial data while the electronic device is disposed on the mobile platform during various states of motion of the mobile platform and in part by modifying usage of the inertial data according to a current state of motion of the mobile platform.

[0083] Fig.12 1500 is a flowchart of an exemplary process 1500 for operating an electronic device according to a specific implementation of the subject technology. Figure 1A , Figure 1B and Figure 2 The process 1500 is described with reference to the electronic device 100a. However, the process 1500 is not limited to Figure 1A , Figure 1B and Figure 2 The electronic device 100a of the present invention may be used, and one or more blocks (or operations) of process 1500 may be performed by one or more other components of other suitable devices (including electronic device 100b and / or electronic device 100c). Further for the purpose of explanation, some blocks of process 1500 are described herein as occurring sequentially or linearly. However, multiple blocks of process 1500 may occur in parallel. In addition, the blocks of process 1500 do not have to be performed in the order shown, and / or one or more blocks of process 1500 do not have to be performed and / or may be replaced by other operations.

[0084] like Fig.12 As shown, at block 1502, an electronic device such as electronic device 100a may operate a first simultaneous localization and mapping (SLAM) system (e.g., a visual inertial SLAM system such as VI SLAM system 287) of the electronic device within a first time period. The electronic device may be disposed on a movable platform such as a car, train, airplane, elevator, escalator, moving walkway, or other movable or mobile platform as described herein.

[0085] At block 1504, the electronic device may use a first (e.g., visual-inertial) SLAM system to control an output of the electronic device (e.g., display of virtual content) during a first time period. For example, controlling the output of the electronic device may include displaying virtual content anchored to a mobile platform on which the electronic device is disposed.

[0086] At block 1506, the electronic device may detect a change in the motion state of the electronic device. For example, the change in the motion state of the electronic device may be caused by a change in the motion state of a platform on which the electronic device is disposed (e.g., a change from a constant motion state (which may include a constant zero motion state) to an accelerated motion state when the platform begins to move or change speed and / or direction).

[0087] For example, detecting a change in motion state may include detecting a difference between visual data and inertial data of a visual-inertial SLAM system, such as combining Figure 7 and Fig.10 The first SLAM state 1100 is described. In one or more specific implementations, the visual data may include an image-based rotation estimate of the electronic device, and the inertial data may include a gyroscope-based rotation estimate of the electronic device. The visual data and the inertial data may also or alternatively include other corresponding image-based motion estimates and inertial-based motion estimates such as linear motion estimates and / or acceleration estimates.

[0088] At box 1506, the electronic device may switch from a first (e.g., visual-inertial) SLAM system of the electronic device to a second simultaneous localization and mapping (SLAM) system (e.g., a pure visual SLAM system such as VO SLAM system 289) in response to detecting a change in motion state. Switching from the first (e.g., visual-inertial) SLAM system to the second (e.g., pure visual) SLAM system may include switching from a first SLAM state such as the first SLAM state 1100 described herein to another SLAM state such as the third SLAM state 1102 described herein (e.g., directly switching and / or switching via an additional SLAM state such as the second SLAM state 1114 described herein).

[0089] At block 1508, the electronic device may use a second (e.g., pure visual) SLAM system to control the output of the electronic device during the second time period. In one or more specific implementations, in response to detecting the difference and before the switch, the electronic device may also temporarily operate both the visual-inertial SLAM system and the pure visual SLAM system while comparing the outputs of the pure visual SLAM system and the visual-inertial SLAM system (e.g., as described above in conjunction with Figure 7 and Fig. 9For example, while temporarily operating both the visual-inertial SLAM system and the pure vision SLAM system, the electronic device may use the pure vision SLAM system to control the output of the electronic device during a third time period.

[0090] In one or more specific implementations, before temporarily operating both the pure visual SLAM system and the visual inertial SLAM system and after the detection, the electronic device may also temporarily continue to operate the visual inertial SLAM system while de-weighting the visual data of the visual inertial SLAM system (e.g., as described above in conjunction with Fig.10 The electronic device may also determine whether the difference occurs within a predetermined minimum amount of time while temporarily continuing to operate the visual-inertial SLAM system while de-weighting the visual data of the visual-inertial SLAM system (e.g., as described above in conjunction with Fig.10 1412 of FIG. 14B ).

[0091] The various processes defined herein contemplate the option of obtaining and utilizing a user's personal information. For example, such personal information may be utilized to provide extended reality for mobile platforms. However, to the extent such personal information is collected, such information should be obtained with the user's informed consent. Users should understand and control the use of their personal information as described herein.

[0092] Personal information will be used by appropriate parties only for lawful and reasonable purposes. Parties utilizing such information will adhere to privacy policies and practices that at a minimum comply with applicable laws and regulations. In addition, such policies should be comprehensive, user accessible, and recognized to meet or exceed government / industry standards. In addition, parties may not distribute, sell, or otherwise share such information except for any reasonable and lawful purpose.

[0093] However, users can limit the extent to which parties can access or otherwise obtain personal information. For example, settings or other preferences can be adjusted so that users can decide whether their personal information is accessible by various entities. In addition, while some features defined herein are described in the context of using personal information, aspects of these features can be implemented without the need to use such information. For example, if user preferences, account names, and / or location history are collected, this information can be obfuscated or otherwise generalized so that the information does not identify the corresponding user.

[0094] The above functions can be implemented in computer software, firmware or hardware. The technology can be implemented using one or more computer program products. The programmable processor and computer can be included in a mobile device or packaged as a mobile device. The process and logic flow can be performed by one or more programmable processors and one or more programmable logic circuits. General and special computing devices and storage devices can be interconnected through a communication network.

[0095] Some specific implementations include electronic components such as microprocessors, storage devices, and memories that store computer program instructions in machine-readable or computer-readable media (also referred to as computer-readable storage media, machine-readable media, or machine-readable storage media). Some examples of such computer-readable media include RAM, ROM, compact disk-read only (CD-ROM), compact disk-recordable (CD-R), compact disk-rewritable (CD-RW), digital versatile disk-read only (e.g., DVD-ROM, dual-layer DVD-ROM), various recordable / rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD card, mini-SD card, micro-SD card, etc.), magnetic and / or solid-state hard drives, read-only and recordable A computer readable medium may store a computer program that is executable by at least one processing unit and includes a set of instructions for performing various operations. Examples of computer programs or computer code include machine code, such as that produced by a compiler, and files including higher-level code that can be executed by a computer, electronic component, or microprocessor using an interpreter.

[0096] Although the above discussion mainly involves microprocessors or multi-core processors that execute software, some implementations are performed by one or more integrated circuits such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs). In some implementations, such integrated circuits execute instructions stored on the circuits themselves.

[0097] As used in this specification and any claims of this patent application, the terms "computer", "server", "processor" and "memory" refer to electronic or other technical devices. These terms exclude people or groups of people. For the purposes of this specification, the terms display or displaying means displaying on an electronic device. As used in this specification and any claims of this patent application, the terms "computer-readable medium" and "computer-readable media" are entirely limited to tangible, tangible objects that store information in a form that can be read by a computer. These terms do not include any wireless signals, wired download signals, and any other transient signals.

[0098] To provide interaction with a user, implementations of the subject matter described in this specification may be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, that the user can use to provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, voice, or tactile input. In addition, a computer may interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser in response to a request received from the web browser on the user's client device.

[0099] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component, such as a data server, or includes a middleware component, such as an application server, or includes a front-end component, such as a client computer with a graphical user interface or a web browser, through which a user can interact with a specific implementation of the subject matter described in this specification, or includes any combination of one or more such back-end components, middleware components, or front-end components. The components of the system may be interconnected by digital data communication of any form or medium, such as a communication network. Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), interconnected networks (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[0100] A computing system may include a client and a server. The client and the server are generally remote from each other and may interact via a communication network. The relationship between the client and the server is generated by computer programs running on respective computers and having a client-server relationship with each other. In some specific implementations, the server transmits data (e.g., an HTML page) to a client device (e.g., in order to display data to a user interacting with the client device and to receive user input from a user interacting with the client device). Data generated at the client device (e.g., a result of a user interaction) may be received at the server from the client device.

[0101] According to aspects of the subject disclosure, a method is provided that includes identifying device motion of a device using one or more sensors of a device; determining that the device motion includes a first component associated with motion of a mobile platform and a second component separate from the motion of the mobile platform; determining an anchor position fixed relative to the mobile platform; and displaying virtual content anchored to the anchor position fixed relative to the mobile platform using, using a display of the device, at least the second component of the device motion separate from the motion of the mobile platform.

[0102] According to aspects of the subject disclosure, a device is provided that includes a display; one or more sensors; and one or more processors configured to: use the one or more sensors to identify device motion of the device; determine that the device motion includes a first component associated with motion of a mobile platform and a second component separate from the motion of the mobile platform; determine an anchor position fixed relative to the mobile platform; and use at least the second component of the device motion separate from the motion of the mobile platform to display virtual content anchored to the anchor position fixed relative to the mobile platform.

[0103] According to aspects of the subject disclosure, there is provided a non-transitory computer-readable medium comprising instructions that, when executed by a computing device, cause the computing device to identify device motion of a device using one or more sensors of the device; determine that the device motion includes a first component associated with motion of a mobile platform and a second component separate from the motion of the mobile platform; determine an anchor position fixed relative to the mobile platform; and display virtual content anchored to the anchor position fixed relative to the mobile platform using at least the second component of the device motion separate from the motion of the mobile platform.

[0104] According to various aspects of the subject disclosure, a method is provided, the method comprising operating a first simultaneous localization and mapping (SLAM) system of an electronic device during a first time period; using the first SLAM system to control an output of the electronic device during the first time period; using the electronic device to detect a change in a motion state of the electronic device; switching from the first SLAM system of the electronic device to a second simultaneous localization and mapping (SLAM) system in response to detecting a change in the motion state; and using the second SLAM system to control the output of the electronic device during a second time period.

[0105] According to aspects of the subject disclosure, a method is provided that includes obtaining, by an electronic device, inertial data from an inertial sensor of the electronic device; and operating the electronic device based on the inertial data while the electronic device is disposed on a movable platform during various stages of motion of the movable platform and in part by modifying use of the inertial data according to a current stage of motion of the movable platform.

[0106] According to aspects of the subject disclosure, an electronic device is provided that includes a display; an inertial sensor; and one or more processors configured to: obtain inertial data from the inertial sensor; and operate the electronic device based on the inertial data when the electronic device is set on a movable platform during various motion stages of the movable platform and in part by modifying the use of the inertial data according to the current motion stage of the movable platform.

[0107] Those skilled in the art will recognize that various illustrative frames, modules, elements, parts, methods and algorithms described herein can be implemented as electronic hardware, computer software or a combination of the two. In order to illustrate this interchangeability of hardware and software, various illustrative frames, modules, elements, parts, methods and algorithms have been generally described above according to functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. The functions can be implemented in different ways for each specific application. Various parts and frames can be arranged differently (e.g., arranged in different orders, or divided in different ways), without departing from the scope of the subject technology.

[0108] It should be understood that the specific order or hierarchy of steps in the process disclosed by the present invention is an illustration of an exemplary method. Based on design preference requirements, it should be understood that the specific order or hierarchy of steps in the process can be rearranged. Some of the steps can be performed simultaneously. The attached method claims present elements of various steps in a sample order and are not meant to be limited to the specific order or hierarchy presented.

[0109] The previous description is provided to enable those skilled in the art to practice the various aspects described herein. The previous description provides various examples of the subject technology, and the subject technology is not limited to these examples. Various modifications of these aspects are obvious to those skilled in the art, and the general principles defined herein can be applied to other aspects. Therefore, the claims are not intended to be limited to the various aspects shown herein, but are intended to make the full scope consistent with the language claims, wherein the reference to the singular element is not intended to mean "only one", but to refer to "one or more", unless specifically indicated. Unless otherwise specifically stated, the term "some" refers to one or more. Male pronouns (e.g., his) include female and neutral (e.g., her and its), and vice versa. Titles and subtitles (if any) are only for convenience and do not limit the present invention described herein.

[0110] As used herein, the term website may include any aspect of a website, including one or more web pages, one or more servers for hosting or storing web-related content, and the like. Thus, the term website may be used interchangeably with the terms web page and server. The predicate words "configured to," "capable of operating to," and "programmed to" do not imply any specific tangible or intangible modification of a subject matter but are intended to be used interchangeably. For example, a component or a processor configured to monitor and control operations may also mean that the processor is programmed to monitor and control operations or that the processor is capable of operating to monitor and control operations. Similarly, a processor configured to execute code may be interpreted as a processor that is programmed to execute code or is capable of operating to execute code.

[0111] As used herein, the term automatically may include execution by a computer or machine without user intervention; for example, by instructions in response to a predicate action of a computer or machine or other initiating mechanism. The word "exemplary" is used herein to mean "serving as an example or illustration." Any aspect or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects or designs.

[0112] Phrases such as "aspects" do not mean that this aspect is essential to the subject technology or that this aspect applies to all configurations of the subject technology. The disclosure related to an aspect may apply to all configurations, or one or more configurations. An aspect may provide one or more examples. Phrases such as aspects may refer to one or more aspects, and vice versa. Phrases such as "implementation" do not mean that this implementation is essential to the subject technology or that this implementation applies to all configurations of the subject technology. The disclosure related to an implementation may apply to all implementations, or one or more implementations. An implementation may provide one or more examples. Phrases such as "implementation" may refer to one or more implementations, and vice versa. Phrases such as "configuration" do not mean that this configuration is essential to the subject technology or that this configuration applies to all configurations of the subject technology. The disclosure related to a configuration may apply to all configurations or one or more configurations. A configuration may provide one or more examples. Phrases such as "configuration" may refer to one or more configurations, and vice versa.

Claims

1. A device comprising: monitor; a plurality of sensors, the plurality of sensors comprising at least a plurality of inertial measurement unit sensors; as well as One or more processors configured to: identifying device motion of the device using the plurality of inertial measurement unit sensors; determining that the device motion includes a first component associated with motion of a mobile platform and a second component separate from the motion of the mobile platform; determining an anchor position fixed relative to the mobile platform; as well as The second component of the device motion separate from the motion of the mobile platform is tracked using less than all of the plurality of inertial measurement unit sensors to display virtual content anchored to the anchor position that is fixed relative to the mobile platform.

2. The device of claim 1 , wherein the one or more sensors include an optical sensor, a depth sensor, and an inertial measurement unit including the plurality of inertial measurement unit sensors, wherein the plurality of inertial measurement unit sensors include an accelerometer, a gyroscope, and a magnetometer, wherein the one or more processors are configured to identify motion of the device using the inertial measurement unit, and to display virtual content anchored to the anchor position fixed relative to the mobile platform using the optical sensor and the depth sensor without further input from the inertial measurement unit. 3 . The device of claim 1 , wherein the second component of the device motion separate from the motion of the mobile platform comprises rotational motion of the device.

4. The device of claim 3, wherein the second component of the device motion separate from the motion of the mobile platform comprises a translational motion of the device relative to the mobile platform and the anchor location.

5. The apparatus of claim 1 , wherein the one or more processors are further configured to: controlling display of the virtual content using a first simultaneous localization and mapping (SLAM) system of the device during a first time period; detecting a change in the first component associated with motion of the mobile platform; and In response to detecting a change in the first component, both the first simultaneous localization and mapping (SLAM) system and the second SLAM system are temporarily operated while comparing outputs of the first SLAM system and a different second SLAM system.

6. The device of claim 5, wherein the one or more processors are further configured to: in response to determining that the outputs of the first SLAM system and the second SLAM system have been consistent for at least a predetermined amount of time, continue to use the first SLAM system to control the display of the virtual content during a second time period.

7. The apparatus of claim 5, wherein the first SLAM system comprises a visual-inertial SLAM system.

8. The device of claim 7, wherein the one or more processors are configured to detect a change in the first component by detecting a difference between visual data of the visual-inertial SLAM system and inertial data of the visual-inertial SLAM system.

9. The device of claim 8, wherein the one or more processors are further configured to: in response to determining that the outputs of the first SLAM system and the second SLAM system have been inconsistent for at least a predetermined amount of time, switch to using the second SLAM system to control the display of the virtual content during a second time period.

10. The device of claim 8, wherein the visual data comprises an image-based rotation estimate of the device and the inertial data comprises a gyroscope-based rotation estimate of the device.

11. A method comprising: identifying device motion of the device using a plurality of inertial measurement unit sensors of the device; determining that the device motion includes a first component associated with motion of a mobile platform and a second component separate from the motion of the mobile platform; determining an anchor position fixed relative to the mobile platform; as well as The second component of the device motion separate from the motion of the mobile platform is tracked using less than all of the plurality of inertial measurement unit sensors to display virtual content anchored to the anchor position that is fixed relative to the mobile platform.

12. A method according to claim 11, wherein the device includes an optical sensor, a depth sensor and an inertial measurement unit including the multiple inertial measurement unit sensors, wherein the multiple inertial measurement unit sensors include an accelerometer, a gyroscope and a magnetometer, wherein the method includes using the inertial measurement unit to identify the device movement, and using the optical sensor and the depth sensor to display virtual content anchored to the anchor position fixed relative to the mobile platform without further input from the inertial measurement unit.

13. The method of claim 11, wherein the second component of the device motion separate from the motion of the mobile platform comprises rotational motion of the device.

14. The method of claim 11, wherein the second component of the device motion separate from the motion of the mobile platform comprises translational motion of the device relative to the mobile platform and the anchor location.

15. The method according to claim 11, further comprising: controlling display of the virtual content using a first simultaneous localization and mapping (SLAM) system of the device during a first time period; detecting a change in the first component associated with a motion of the mobile platform; In response to detecting a change in the first component, temporarily operating both the first simultaneous localization and mapping (SLAM) system and the second SLAM system while comparing outputs of the first SLAM system and a second, different SLAM system; and In response to determining that outputs of the first SLAM system and the second SLAM system have been inconsistent for at least a predetermined amount of time, switching to controlling display of the virtual content using the second SLAM system during a second time period. The method of claim 15 , wherein the first SLAM system comprises a visual-inertial SLAM system. 17 . The method of claim 16 , wherein detecting a change in the first component comprises detecting a change in the first component by detecting a difference between visual data of the visual-inertial SLAM system and inertial data of the visual-inertial SLAM system.

18. The method of claim 17, wherein the visual data comprises an image-based rotation estimate of the device and the inertial data comprises a gyroscope-based rotation estimate of the device.

19. A system configured to perform the method according to any one of claims 11 to 18.

20. A method for operating an electronic device, the method comprising: During a first time period, controlling the electronic device using a first simultaneous localization and mapping (SLAM) system of the electronic device; as well as During a second time period and in response to detecting a difference between the visual data from the first SLAM system and the inertial data from the first SLAM system, the electronic device is controlled using a second simultaneous localization and mapping SLAM system of the electronic device.

21. The method of claim 20, wherein the first simultaneous localization and mapping SLAM system is a visual-inertial SLAM system, and the second simultaneous localization and mapping SLAM system is a pure vision SLAM system.

22. The method of claim 21, wherein the difference between the visual data from the first SLAM system and the inertial data from the first SLAM system is caused by movement of a platform on which the electronic device is disposed.

23. The method of claim 20, wherein the visual data comprises an image-based rotation estimate of the electronic device and the inertial data comprises a gyroscope-based rotation estimate of the electronic device.

24. The method of claim 20, further comprising: In response to detecting the difference and before using the second SLAM system for control, both the first SLAM system and the second SLAM system are temporarily operated while comparing outputs of the first SLAM system and the second SLAM system.

25. The method according to claim 24, further comprising: While temporarily operating both the first SLAM system and the second SLAM system, the electronic device is controlled using the first SLAM system during a third time period.

26. The method according to claim 25, further comprising: Before temporarily operating both the first SLAM system and the second SLAM system and after the detecting: While de-weighting the visual data of the first SLAM system, temporarily continue to use the first SLAM system to control the electronic device; and While temporarily continuing to use the first SLAM system to control the electronic device while de-weighting the visual data of the first SLAM system, determining whether the discrepancy has occurred for a predetermined minimum amount of time.

27. The method of claim 26, wherein controlling the electronic device using the second SLAM system during the second time period comprises: In response to detecting the difference and determining that the difference has occurred for the predetermined minimum amount of time, the electronic device is controlled using the second SLAM system during the second time period.

28. An electronic device configured to perform the method according to any one of claims 20 to 27.

Images