Localization and mapping using images from multiple devices

Abstract

The present disclosure relates to positioning and mapping using images from multiple devices. The present disclosure as a whole relates to positioning and mapping. In some examples, an electronic device acquires first image data and acquires motion data using a motion sensor. The electronic device receives information corresponding to a second electronic device. The electronic device generates a representation of a first posture of the first electronic device using the first image data, the motion data, and the information corresponding to the second electronic device. The electronic device displays a virtual object on a display, wherein the display of the virtual object is based on the representation of the first posture of the first electronic device.

Description

[0001] Related Applications Citations

[0002] This application is a divisional application of the invention patent application with Chinese national application number 201910818909.0, application date August 30, 2019, and invention name “Positioning and mapping using images from multiple devices”. Technical Field

[0003] The present disclosure relates generally to computer vision, and more particularly to techniques for performing localization and mapping. Background Art

[0004] Some technologies for displaying computer-generated content on a device utilize computer-generated content to inaccurately enhance the physical environment. For example, a virtual object can be displayed at an inaccurate position relative to the physical environment. Summary of the invention

[0005] This article describes techniques for determining an accurate estimate of a device's pose to correctly augment the physical environment with computer-generated content. This results in a better user experience when computer-generated reality (CGR) techniques are implemented.

[0006] According to some examples, a method is described. The method is performed at a first electronic device having a display, a camera sensor, and a motion sensor. The method includes: acquiring first image data; acquiring motion data using the motion sensor; receiving information corresponding to a second electronic device; generating a representation of a first posture of the first electronic device using the first image data, the motion data, and the information corresponding to the second electronic device; and displaying a virtual object on the display, wherein the display of the virtual object is based on the representation of the first posture of the first electronic device.

[0007] According to some examples, a non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display, a camera sensor, and a motion sensor. The one or more programs include instructions for the following operations: acquiring first image data; acquiring motion data using a motion sensor; receiving information corresponding to a second electronic device; generating a representation of a first posture of the first electronic device using the first image data, the motion data, and the information corresponding to the second electronic device; and displaying a virtual object on the display, wherein the display of the virtual object is based on the representation of the first posture of the first electronic device.

[0008] According to some examples, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display, a camera sensor, and a motion sensor. The one or more programs include instructions for the following operations: acquiring first image data; acquiring motion data using a motion sensor; receiving information corresponding to a second electronic device; generating a representation of a first posture of the first electronic device using the first image data, the motion data, and the information corresponding to the second electronic device; and displaying a virtual object on the display, wherein the display of the virtual object is based on the representation of the first posture of the first electronic device.

[0009] According to some examples, an electronic device is described. The electronic device includes: a display; a camera sensor; a motion sensor; one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors. The one or more programs include instructions for the following operations: acquiring first image data; acquiring motion data using a motion sensor; receiving information corresponding to a second electronic device; generating a representation of a first posture of the first electronic device using the first image data, the motion data, and the information corresponding to the second electronic device; and displaying a virtual object on the display, wherein the display of the virtual object is based on the representation of the first posture of the first electronic device.

[0010] According to some examples, an electronic device is described. The electronic device includes: a display; a camera sensor; a motion sensor; a device for acquiring first image data; a device for acquiring motion data using the motion sensor; a device for receiving information corresponding to a second electronic device; a device for generating a representation of a first posture of the first electronic device using the first image data, the motion data, and the information corresponding to the second electronic device; and a device for displaying a virtual object on the display, wherein the display of the virtual object is based on the representation of the first posture of the first electronic device.

[0011] According to some examples, a method is described. The method is performed at a first electronic device. The method includes: receiving first image data and a representation of a first posture of the second electronic device in a three-dimensional space from a second electronic device, wherein the representation of the first posture of the second electronic device corresponds to the first image data; receiving second image data and a representation of a second posture of the third electronic device in a three-dimensional space from a third electronic device, wherein the representation of the second posture of the third electronic device corresponds to the second image data; generating a representation of one or more features in a three-dimensional space, wherein the representation of the one or more features is generated using the first image data, the representation of the first posture, the second image data, and the representation of the second posture; and transmitting the representation of the one or more features to the second electronic device and the third electronic device, wherein the representation of the one or more features is configured to be used by the second electronic device to generate a representation of a third posture of the second electronic device, and wherein the representation of the one or more features is configured to be used by the third electronic device to generate a representation of a fourth posture of the third electronic device.

[0012] According to some examples, a non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device. The one or more programs include instructions for the following operations: receiving first image data and a representation of a first posture of the second electronic device in a three-dimensional space from a second electronic device, wherein the representation of the first posture of the second electronic device corresponds to the first image data; receiving second image data and a representation of a second posture of a third electronic device in a three-dimensional space from a third electronic device, wherein the representation of the second posture of the third electronic device corresponds to the second image data; generating a representation of one or more features in a three-dimensional space, wherein the representation of the one or more features is generated using the first image data, the representation of the first posture, the second image data, and the representation of the second posture; and transmitting the representation of the one or more features to the second electronic device and the third electronic device, wherein the representation of the one or more features is configured to be used by the second electronic device to generate a representation of a third posture of the second electronic device, and wherein the representation of the one or more features is configured to be used by the third electronic device to generate a representation of a fourth posture of the third electronic device.

[0013] According to some examples, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device. The one or more programs include instructions for the following operations: receiving first image data and a representation of a first posture of the second electronic device in a three-dimensional space from a second electronic device, wherein the representation of the first posture of the second electronic device corresponds to the first image data; receiving second image data and a representation of a second posture of a third electronic device in a three-dimensional space from a third electronic device, wherein the representation of the second posture of the third electronic device corresponds to the second image data; generating a representation of one or more features in a three-dimensional space, wherein the representation of the one or more features is generated using the first image data, the representation of the first posture, the second image data, and the representation of the second posture; and transmitting the representation of the one or more features to the second electronic device and the third electronic device, wherein the representation of the one or more features is configured to be used by the second electronic device to generate a representation of a third posture of the second electronic device, and wherein the representation of the one or more features is configured to be used by the third electronic device to generate a representation of a fourth posture of the third electronic device.

[0014] According to some examples, an electronic device is described. The electronic device includes: one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors. The one or more programs include instructions for the following operations: receiving first image data and a representation of a first posture of the second electronic device in a three-dimensional space from a second electronic device, wherein the representation of the first posture of the second electronic device corresponds to the first image data; receiving second image data and a representation of a second posture of a third electronic device in a three-dimensional space from a third electronic device, wherein the representation of the second posture of the third electronic device corresponds to the second image data; generating a representation of one or more features in a three-dimensional space, wherein the representation of the one or more features is generated using the first image data, the representation of the first posture, the second image data, and the representation of the second posture; and transmitting the representation of the one or more features to the second electronic device and the third electronic device, wherein the representation of the one or more features is configured to be used by the second electronic device to generate a representation of a third posture of the second electronic device, and wherein the representation of the one or more features is configured to be used by the third electronic device to generate a representation of a fourth posture of the third electronic device.

[0015] According to some examples, an electronic device is described. The electronic device includes: a device for receiving first image data and a representation of a first posture of the second electronic device in a three-dimensional space from a second electronic device, wherein the representation of the first posture of the second electronic device corresponds to the first image data; a device for receiving second image data and a representation of a second posture of the third electronic device in a three-dimensional space from a third electronic device, wherein the representation of the second posture of the third electronic device corresponds to the second image data; a device for generating a representation of one or more features in a three-dimensional space, wherein the representation of the one or more features is generated using the first image data, the representation of the first posture, the second image data, and the representation of the second posture; and a device for transmitting the representation of the one or more features to the second electronic device and the third electronic device, wherein the representation of the one or more features is configured to be used by the second electronic device to generate a representation of a third posture of the second electronic device, and wherein the representation of the one or more features is configured to be used by the third electronic device to generate a representation of a fourth posture of the third electronic device. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1A to Figure 1B An exemplary system for use in various CGR techniques is shown.

[0017] FIG. 2A to FIG. 2D An electronic device using CGR technology is shown.

[0018] Figure 3 is a flow chart illustrating an exemplary method for determining a posture of an electronic device from the perspective of the electronic device.

[0019] Figure 4 is a flow chart illustrating an exemplary method for determining a posture of an electronic device from the perspective of a server. DETAILED DESCRIPTION

[0020] Figure 1A and Figure 1B An exemplary system 100 is shown for use in various CGR techniques.

[0021] In some examples, such as Figure 1A As shown, system 100 includes device 100a. Device 100a includes various components, such as processor 102, RF circuit 104, memory 106, image sensor 108, orientation sensor 110, microphone 112, position sensor 116, speaker 118, display 120, and touch-sensitive surface 122. These components are optionally in communication via a communication bus 150 of device 100a.

[0022] In some examples, elements of system 100 are implemented in a base station device (e.g., a computing device such as a remote server, a mobile device, or a laptop computer), and other elements of system 100 are implemented in a head mounted display (HMD) device designed to be worn by a user, wherein the HMD device communicates with the base station device. In some examples, device 100a is implemented in either the base station device or the HMD device.

[0023] like Figure 1B As shown, in some examples, system 100 includes two (or more) devices that communicate, such as communicating via a wired connection or a wireless connection. A first device 100b (e.g., a base station device) includes a processor 102, an RF circuit 104, and a memory 106. These components optionally communicate via a communication bus 150 of device 100b. A second device 100c (e.g., a head-mounted device) includes various components, such as a processor 102, an RF circuit 104, a memory 106, an image sensor 108, an orientation sensor 110, a microphone 112, a position sensor 116, a speaker 118, a display 120, and a touch-sensitive surface 122. These components optionally communicate via a communication bus 150 of device 100c.

[0024] In some examples, system 100 is a mobile device. In some examples, system 100 is a HDM device. In some examples, device 100 is a wearable HUD device.

[0025] System 100 includes processor 102 and memory 106. Processor 102 includes one or more general purpose processors, one or more graphics processors, and / or one or more digital signal processors. In some examples, memory 106 is one or more non-transitory computer-readable storage media (e.g., flash memory, random access memory) storing computer-readable instructions configured to be executed by processor 102 to perform the techniques described below.

[0026] System 100 includes RF circuitry 104. RF circuitry 104 optionally includes circuitry for communicating with electronic devices, networks (such as the Internet, an intranet), and / or wireless networks (such as cellular networks and wireless local area networks (LANs)). RF circuitry 104 optionally includes circuitry for communicating using near field communications and / or short range communications (such as ) circuit for communication.

[0027] System 100 includes a display 120. In some examples, display 120 includes a first display (e.g., a left eye display panel) and a second display (e.g., a right eye display panel), each display being used to display an image to a corresponding eye of a user. Corresponding images are displayed simultaneously on the first display and the second display. Optionally, the corresponding images include representations of the same virtual object and / or the same physical object from different viewpoints, thereby generating a parallax effect that provides the user with a stereoscopic effect of the object on the display. In some examples, display 120 includes a single display. For each eye of the user, the corresponding image is simultaneously displayed on a first area and a second area of ​​the single display. Optionally, the corresponding images include representations of the same virtual object and / or the same physical object from different viewpoints, thereby generating a parallax effect that provides the user with a stereoscopic effect of the object on the single display.

[0028] In some examples, system 100 includes a touch-sensitive surface 122 for receiving user inputs, such as tap inputs and swipe inputs. In some examples, display 120 and touch-sensitive surface 122 form a touch-sensitive display.

[0029] System 100 includes an image sensor 108. Image sensor 108 optionally includes one or more visible light image sensors operable to acquire images of physical objects from a physical environment, such as a charge coupled device (CCD) sensor and / or a complementary metal oxide semiconductor (CMOS) sensor. The image sensor also optionally includes one or more infrared (IR) sensors, such as a passive IR sensor or an active IR sensor, for detecting infrared light from the physical environment. For example, an active IR sensor includes an IR emitter, such as an IR point emitter, for emitting infrared light into the physical environment. Image sensor 108 also optionally includes one or more event cameras configured to capture the movement of physical objects in the physical environment. Image sensor 108 also optionally includes one or more depth sensors configured to detect the distance of physical objects from system 100. In some examples, system 100 uses a combination of CCD sensors, event cameras, and depth sensors to detect the physical environment around system 100. In some examples, image sensor 108 includes a first image sensor and a second image sensor. The first image sensor and the second image sensor are optionally configured to capture images of physical objects in the physical environment from two different perspectives. In some examples, the system 100 uses the image sensor 108 to receive user input, such as gestures. In some examples, the system 100 uses the image sensor 108 to detect the position and orientation of the system 100 and / or the display 120 in the physical environment. For example, the system 100 uses the image sensor 108 to track the position and orientation of the display 120 relative to one or more fixed objects in the physical environment. In some examples, the system 100 uses the image sensor 108 facing inward (e.g., facing the user) for gaze tracking and / or hand movement tracking, which can be used, for example, to control the user's avatar.

[0030] In some examples, system 100 includes microphone 112. System 100 uses microphone 112 to detect sounds from a user and / or the user's physical environment. In some examples, microphone 112 includes a microphone array (including multiple microphones), which optionally operate in coordination to identify ambient noise or locate a sound source in the space of a physical environment.

[0031] System 100 includes an orientation sensor 110 for detecting orientation and / or movement of system 100 and / or display 120. For example, system 100 uses orientation sensor 110 to track changes in position and / or orientation of system 100 and / or display 120, such as relative to physical objects in a physical environment. Orientation sensor 110 optionally includes one or more gyroscopes and / or one or more accelerometers.

[0032] The physical environment refers to the physical world that people can sense and / or interact with without the help of electronic systems. A physical environment, such as a physical park, includes physical objects, such as physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell.

[0033] In contrast, a CGR environment refers to a fully or partially simulated environment that people sense and / or interact with via an electronic system. In CGR, a subset of a person's physical movements, or representations thereof, are tracked, and in response, one or more features of one or more virtual objects simulated in the CGR environment are adjusted in a manner that complies with at least one law of physics. For example, a CGR system may detect a person's head turn, and in response, adjust the graphical content and sound field presented to the person in a manner similar to the way such views and sounds change in a physical environment. In some cases (e.g., for accessibility reasons), adjustments to features of virtual objects in a CGR environment may be made in response to representations of physical movement (e.g., voice commands).

[0034] A person may sense and / or interact with a CGR object using any of their senses, including vision, hearing, touch, taste, and smell. For example, a person may sense and / or interact with an audio object that creates a 3D or spatial audio environment that provides the perception of a point audio source in 3D space. As another example, an audio object may enable audio transparency that selectively introduces ambient sounds from the physical environment with or without computer-generated audio. In some CGR environments, a person may sense and / or interact only with an audio object.

[0035] Examples of CGR include virtual reality and mixed reality. A virtual reality (VR) environment refers to a simulated environment designed to be based entirely on computer-generated sensory input to one or more senses. A VR environment includes a plurality of virtual objects that a person can sense and / or interact with. For example, trees, buildings, and computer-generated images representing avatars of people are examples of virtual objects. A person can sense and / or interact with virtual objects in a VR environment through a simulation of the person's presence within the computer-generated environment, and / or through a simulation of a subset of the person's physical movement within the computer-generated environment.

[0036] In contrast to VR environments, which are designed to be based entirely on computer-generated sensory input, a mixed reality (MR) environment refers to a simulated environment designed to include sensory input from the physical environment, or representations thereof, in addition to computer-generated sensory input (e.g., virtual objects). On the virtuality continuum, a mixed reality environment is anything between a fully physical environment at one end and a virtual reality environment at the other end, but not including both ends.

[0037] In some MR environments, computer-generated sensory input can respond to changes in sensory input from the physical environment. In addition, some electronic systems used to render the MR environment can track position and / or orientation relative to the physical environment to enable virtual objects to interact with real objects (i.e., physical items from the physical environment or representations thereof). For example, the system can cause movement so that virtual trees appear stationary relative to the physical ground.

[0038] Examples of mixed reality include augmented reality and augmented virtuality. An augmented reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed on a physical environment or a representation thereof. For example, an electronic system for presenting an AR environment may have a transparent or translucent display through which a person can directly view the physical environment. The system may be configured to present virtual objects on a transparent or translucent display so that a person uses the system to perceive virtual objects superimposed on a physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of a physical environment that are representations of the physical environment. The system combines the image or video with the virtual object and presents the composition on an opaque display. People use the system to indirectly view the physical environment via an image or video of the physical environment and perceive virtual objects superimposed on the physical environment. As used herein, a video of a physical environment displayed on an opaque display is referred to as a "transparent video," meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images when presenting an AR environment on an opaque display. Further alternatively, the system may have a projection system that projects virtual objects into a physical environment, for example as a hologram or on a physical surface, so that a person using the system perceives the virtual objects superimposed on the physical environment.

[0039] An AR environment also refers to a simulated environment in which a representation of a physical environment is transformed by computer-generated sensory information. For example, in providing a pass-through video, the system may transform one or more sensor images to apply a selected perspective (e.g., a viewpoint) that is different from the perspective captured by the imaging sensor. For another example, a representation of a physical environment may be transformed by graphically modifying (e.g., enlarging) a portion thereof so that the modified portion may be a representative but not true version of the original captured image. For another example, a representation of a physical environment may be transformed by graphically eliminating or blurring a portion thereof.

[0040] An augmented virtual (AV) environment refers to a simulated environment in which a virtual or computer-generated environment combines one or more sensory inputs from a physical environment. The sensory input may be a representation of one or more features of the physical environment. For example, an AV park may have virtual trees and virtual buildings, but the faces of people are realistically reproduced from images taken of physical people. For another example, a virtual object may take the shape or color of a physical object imaged by one or more imaging sensors. For another example, a virtual object may take a shadow that conforms to the position of the sun in the physical environment.

[0041] There are many different types of electronic systems that enable people to sense and / or interact with various CGR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on people's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smart phones, tablets, and desktop / laptop computers. A head-mounted system can have one or more speakers and an integrated opaque display. Alternatively, a head-mounted system can be configured to receive an external opaque display (e.g., a smart phone). A head-mounted system can be combined with one or more imaging sensors for capturing images or videos of a physical environment, and / or one or more microphones for capturing audio of a physical environment. Instead of an opaque display, a head-mounted system can have a transparent or translucent display. A transparent or translucent display can have a medium through which light representing an image is directed to a person's eyes. The display can utilize digital light projection, OLED, LED, uLED, silicon-based liquid crystal, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In one embodiment, a transparent or translucent display can be configured to selectively become opaque. A projection-based system can employ retinal projection technology that projects a graphic image onto a person's retina. The projection system can also be configured to project a virtual object into a physical environment, such as a hologram or onto a physical surface.

[0042] Figure 2A An electronic device 200A and an electronic device 200B are shown. The electronic device 200A and / or the electronic device 200B can include some or all of the features of the system 100 described above.

[0043] exist Figure 2AAt this point, electronic device 200A and electronic device 200B are being used to enhance a user's view of a physical environment. Electronic device 200A and electronic device 200B display images of the physical environment captured by the camera sensor (e.g., image sensor 108) of the respective devices for viewing by the user. In addition to displaying images of physical objects such as cubes 202A and 202B, electronic device 200A and electronic device 200B also display virtual objects so that the virtual objects appear to exist in the physical environment, thereby enhancing the user's view of the physical environment. However, in order to display virtual objects (or to enhance the physical environment in some other way (e.g., changing the apparent color of physical objects)), it is advantageous that electronic device 200A and electronic device 200B determine their own posture (e.g., position and orientation) relative to the physical environment. The following description with respect to FIG. 2B to FIG. 2D Describes the process used to determine an accurate estimate of the device's pose.

[0044] exist Figure 2B At , the electronic device 200A initiates a process for mapping and positioning the device 200A relative to the physical environment using, for example, simultaneous localization and mapping (SLAM) technology such as visual inertial measurement (VIO). The electronic device 200A captures images of the cubes 202A and 202B via a camera sensor (e.g., image sensor 108) located on the back of the device. The electronic device 200A displays these captured images via the display 201. In order to determine its posture relative to the physical environment, the electronic device 200A uses the captured images in combination with data obtained via the orientation sensor 110 (e.g., inertial sensor). The electronic device 200A detects significant features (e.g., lines, segments, planes, points, and / or other 3D geometric elements and shapes) from the captured images (e.g., edges or corners of the cubes 202A and 202B in the field of view of the camera sensor), and estimates its position in 3D space by iteratively minimizing the error function of 3D position and posture estimation using the captured images and the data obtained via the orientation sensor 110, while also estimating its own posture. The electronic device is able to update the 3D position and pose estimates on additional captured images as the device moves.

[0045] For example, in Figure 2C As part of the positioning and mapping process, the electronic device 200A moves to another location in the physical environment. With the cubes 202A and 202B still in the field of view of the camera sensor, the electronic device 200A captures images of the cubes 202A and 202B from another perspective. The electronic device 200A displays these captured images via the display 201. The electronic device 200A detects Figure 2BBy comparing the positions of the features in the captured images and in combination with data from orientation sensor 110, the electronic device updates its estimate of the 3D position of the features (e.g., the position of a point in 3D space) and its own pose relative to the physical environment.

[0046] When successfully positioned and mapped, the electronic device 200A is able to display virtual content at the appropriate location on the display because the device is able to use the determined posture relative to the physical environment. Therefore, after (or when) successfully positioned and mapped, the electronic device 200A uses the posture estimation to position the virtual object 204 on the display 201 to display the virtual object 204. It is expected that the virtual object 204 will appear to be placed on the table, but the error associated with the posture estimation causes the electronic device 200A to display the virtual object 204 so that it appears to float above the table.

[0047] To improve the accuracy of the estimate of the posture of the electronic device 200A, more data is needed, especially data about the 3D position of salient features in the physical environment. One technique for improving the position estimate of salient features is to use data generated at a second device (e.g., electronic device 200B) in addition to the data generated at the electronic device 200A. By increasing the amount of data used to estimate the posture of the device (e.g., increasing the number of samples), the error associated with the posture estimate is reduced.

[0048] After (or when) positioning and mapping are successfully performed, the electronic device 200A transmits the captured image and the estimated posture of the device that captured the image to the server. In some examples, after (or when) positioning and mapping are successfully performed, the electronic device also sends inertial data about the movement of the electronic device. Using the captured image and its corresponding device posture (and optionally inertial data), the server performs reconstruction by performing similar mapping functions to estimate the position data of significant features (e.g., a set of 3D points, lines, segments, planes, and / or other 3D geometric elements and shapes) in the captured image. For example, the position data includes the Cartesian coordinates of the corners of the cubes 202A and 202B captured by the image.

[0049] In some examples, the electronic device 200A does not transmit the captured image and the estimated device posture to the server. In some examples, the electronic device 200A performs reconstruction using the captured image, the estimated posture, and / or the data generated at the electronic device 200B. In some examples, the electronic device 200A uses the image captured by the electronic device 200A and its corresponding posture to perform a similar mapping function to estimate the position data of the significant features in the captured image. In some examples, the electronic device 200A receives the image data captured by the electronic device 200B and / or the corresponding posture estimation of the electronic device 200B. In some examples, the electronic device 200B uses the motion data and image data captured by the electronic device 200B to generate the posture estimation of the electronic device 200B, as described above with respect to the generation of the posture estimation of the electronic device 200A. In some examples, the electronic device 200A uses the received image data and / or the corresponding posture estimation of the electronic device 200B to generate the posture estimation of the electronic device 200A (or improve the existing posture estimation). In some examples, electronic device 200B uses images captured by electronic device 200B and their corresponding postures to perform similar mapping functions to estimate position data of significant features in the captured images. In some examples, electronic device 200A receives this position data. In some examples, electronic device 200A uses this position data to generate a posture estimate of electronic device 200A (or improve an existing posture estimate). In some examples, the image data captured by electronic device 200A and electronic device 200B corresponds to one or more parts of the same physical object in the physical environment (e.g., including data about it).

[0050] When (or after) generating the position data for the set of salient features, the server transmits the position data defining the salient features of the 3D map having the defined origin to the electronic device 200B. When (or after) receiving the position data, the electronic device 200B registers to the 3D map by performing positioning using the received position data. Specifically, the electronic device 200B attempts to match the received position data with the salient features detected by the electronic device 200B in the image captured via its camera sensor. Once the electronic device 200B matches the received position data with the salient features detected by the electronic device 200B, the electronic device 200B determines its own posture relative to the 3D map (e.g., relative to the defined origin).

[0051] After (or when) positioning is successfully performed, the electronic device 200B transmits the captured image and the posture of the device that captured the image to the server. The electronic device 200A also transmits the captured image and the posture of the device that captured the image to the server. Reconstruction is performed on the captured images and their corresponding postures from the electronic device 200A and the electronic device 200B, and the server generates an updated estimate of the position data of the salient features in the captured image. In order to generate an updated estimate, the server uses a minimization function (e.g., least squares) on the posture data and the position data of the salient features from the two devices. In some examples, the process described above for generating an updated estimate of the position data can be performed using more than two devices, thereby obtaining more data (e.g., samples) for accurate posture estimation. In some examples, in order to generate an updated estimate of a dynamic scene (e.g., a scene where a physical object is moving), the server uses a timestamp corresponding to the time when the image was captured, where the timestamp is generated at the electronic device 200A and / or 200B. In some examples, the timestamp is used to determine which captured images to compare (e.g., images are compared between images captured at the same time or within a certain threshold (e.g., 30ms)). In some examples, when the scene is static (eg, physical objects are not moving), the server does not use timestamps to generate updated estimates.

[0052] When (or after) generating an updated estimate of the location data of the salient features, the server transmits the updated estimate to the electronic device 200A and the electronic device 200B. The updated estimate corresponds to (e.g., includes updates to) the salient features used by the electronic device 200A and / or the electronic device 200B to perform positioning. In some examples, the updated estimate corresponds to some or all of the salient features. In some examples, the updated estimate includes new 3D features in addition to the 3D features used by a particular device to perform positioning. For example, the updated estimate can include location data of salient features observed by one electronic device but not observed by other electronic devices. In some examples, the updated estimate includes only new 3D features.

[0053] exist Figure 2D At (or after) receiving the updated estimate from the server, the electronic device 200A performs positioning using the updated estimate. Specifically, the electronic device 200A generates a new posture of the device based on the updated estimate received from the server. Similarly, the electronic device 200B performs positioning and generates a new posture of the device based on the updated estimate received from the server.

[0054] After electronic device 200A and electronic device 200B update the estimate of their respective postures using the updated estimate, some of the processes discussed above are repeated to continue to generate further updated estimates of the postures of each device. Electronic devices 200A and 200B continue to transmit the captured images of the physical environment and the postures of the captured images to the server. The server continues to perform reconstruction by generating updated estimates of the position data of the salient features in the captured images using the images and their corresponding postures. The server transmits the updated estimate to electronic devices 200A and 200B, and electronic devices 200A and 200B update the estimates of their respective postures again. This cycle repeats, thereby continuously improving the accuracy of the position estimate of the 3D feature. A more accurate position estimate of the 3D feature leads to a more accurate estimate of the posture of the electronic device (e.g., an estimate with reduced error).

[0055] Figure 3 is a flow chart illustrating an exemplary method for determining (eg, estimating) a posture of an electronic device (eg, 200A, 200B). In some examples, the method is performed by electronic device 200A and / or electronic device 200B.

[0056] At box 302, an electronic device (e.g., 200A, 200B) acquires first image data. In some examples, the first image data uses a raw image format. In some examples, the first image data uses a compressed file format (e.g., jpeg). In some examples, the first image data (e.g., a captured image) is acquired using a camera sensor (e.g., image sensor 108). In some examples, the first image data corresponds to a physical object (e.g., 202A, 202B) in a physical environment (e.g., including data thereof or data about it).

[0057] At block 304, the electronic device acquires motion data using a motion sensor (e.g., orientation sensor 110). In some examples, the motion sensor is one or more inertial measurement units. In some examples, the motion data includes acceleration, rotation, and / or altitude data. In some examples, the motion data is measured relative to a known reference frame.

[0058] At box 306, the electronic device receives information corresponding to the second electronic device. In some examples, the information corresponding to the second electronic device includes a second image data and a representation of the posture of the second electronic device. In some examples, the second image data uses a raw image format. In some examples, the second image data uses a compressed file format (e.g., jpeg). In some examples, the second image data (e.g., the captured image) is acquired using a camera sensor of the second electronic device. In some examples, the second image data corresponds to a physical object (e.g., 202A, 202B) in a physical environment (e.g., including its data or data about it). In some examples, the first image data and the second image data correspond to the same physical object (e.g., 202A, 202B) in a physical environment (e.g., including its data or data about it). In some examples, the representation of the posture of the second electronic device is generated at the second electronic device using (e.g., based on) image data captured by the second electronic device and motion data from a motion sensor of the second electronic device. In some examples, the representation of the posture of the second electronic device is generated at a third electronic device (e.g., a server communicating with the first electronic device and the second electronic device).

[0059] In some examples, the information corresponding to the second electronic device includes a representation of one or more features in a three-dimensional space (e.g., a physical environment) obtained by the second electronic device or using information obtained by the second electronic device. In some examples, the representation of the one or more features is the Cartesian coordinates of one or more features in the physical environment (e.g., the corners of cubes 202A and 202B in the field of view of the camera sensor). In some examples, features include points, lines, segments, planes, and / or other 3D geometric elements and shapes. In some examples, the representation of one or more features corresponds to a physical object (e.g., 202A, 202B) in the physical environment (e.g., the representation of one or more features includes the position in space of certain features of the physical object). In some examples, the representation of the one or more features in the three-dimensional space is generated at a third electronic device (e.g., a host server communicating with 202A and / or 202B; 202A; 202B) using the initial image data transmitted by the first electronic device and the representation of the initial posture transmitted by the first electronic device. In some examples, the representation of the one or more features in the three-dimensional space is generated using image data transmitted by a second electronic device (e.g., an electronic device having a display, a camera sensor, and a motion sensor) and a representation of a gesture transmitted by the second electronic device, and the second electronic device is different from the first electronic device and the third electronic device. Therefore, in some examples, the received representation of the one or more features in the three-dimensional space is based on image data captured by multiple electronic devices and based on the representation of the gestures of the multiple electronic devices. As described above, the representation of one or more features in the three-dimensional space can include the updated positions of some or all of the features detected by the first electronic device in the image data, or the positions of the features that have not been detected or observed by the first electronic device. For example, the representation of one or more features can include the positions of only those features whose positions have been updated, the positions of all features determined by the second (or third) electronic device, and / or the positions of those features that have not been detected or observed by the first electronic device. In some examples, the representation of the first gesture and the received representation of the one or more features are located on the same coordinate system (e.g., Cartesian coordinate system, spherical coordinate system). In some examples, the representation of the first gesture and the received representation of the gesture of the second electronic device are on the same coordinate system. In some examples, the first posture of the first device is based on a rotation and a translation of the first electronic device, where the rotation and the translation are on the same coordinate system.

[0060] At box 308, the electronic device generates a representation of a first posture of the first electronic device using the first image data, the motion data, and the information corresponding to the second electronic device. In some examples, the representation of the first posture of the first electronic device includes, for example, a position and orientation defined by rotation and translation relative to a reference 3D coordinate system. In some examples, the first posture is a posture of at least a portion of the captured image data. In some examples, generating a representation of the first posture of the first electronic device includes generating or using a representation (e.g., position data) of one or more features corresponding to one or more features extracted from the image data. In some examples, features include points, lines, segments, planes, and / or other 3D geometric elements and shapes.

[0061] In some examples, before receiving information corresponding to the second electronic device, the electronic device acquires initial image data (e.g., image data different from the first image data). In some examples, before receiving information corresponding to the second electronic device, the electronic device acquires initial motion data using a motion sensor. In some examples, before receiving information corresponding to the second electronic device, the electronic device generates a representation of an initial posture of the first electronic device using the initial image data and the initial motion data. In some examples, before receiving information corresponding to the second electronic device, the electronic device transmits the initial image data and the representation of the initial posture of the first electronic device. In some examples, the first electronic device transmits the initial image data and the representation of the initial posture to the second electronic device and / or the third electronic device.

[0062] In some examples, instead of transmitting the initial image data and the representation of the initial posture, the electronic device transmits a representation of one or more features corresponding to (e.g., corresponding to) one or more features extracted from (e.g., corresponding to) the image data and / or motion data (e.g., position data). For example, the first electronic device transmits the representation of the one or more features to the second electronic device.

[0063] At box 310, the electronic device displays a virtual object (e.g., 204) on a display (e.g., 201), wherein the display of the virtual object is based on a representation of a first posture of the first electronic device. In some examples, the electronic device displays a representation of a physical object (e.g., 202A, 202B), wherein the representation of the physical object is displayed simultaneously with the virtual object (e.g., 204). For example, the physical object is displayed so that it partially (e.g., but not completely) overlaps with the representation of the virtual object, thereby causing a portion of the virtual object to not be displayed. In some examples, the electronic device displays only virtual content, wherein the display of the virtual content is based on the determined posture of the electronic device.

[0064] In some examples, after (or in response to) displaying the virtual object at block 310, the electronic device acquires new motion data, new image data, a new representation of the posture of the second electronic device, and / or a new representation of one or more features in the physical environment (e.g., position data). Using some or all of the newly acquired data, the electronic device generates a new representation of the posture of the first electronic device, as described with respect to block 308.

[0065] Some of the examples described above with respect to method 300 involve interaction between two electronic devices and a host server. For example, a first electronic device and a second electronic device can provide data to a server, wherein the server can generate a representation of one or more features based on the data provided. In some examples, the first electronic device (or the second electronic device) generates a new (e.g., updated) representation of the posture of the first electronic device (or the second electronic device) using the representation of one or more features generated by the server. In addition, some of the examples described above with respect to method 300 can be performed by two electronic devices, one of which acts as a host server. For example, the first electronic device can generate a new representation of the posture of the first electronic device using data generated at the second electronic device. In some examples, the second electronic device transmits the corresponding representation of the image data captured by the second electronic device and / or the posture of the second electronic device to the first electronic device. In some examples, the second electronic device can generate a representation of one or more features using motion (e.g., orientation) data and image data acquired by the second electronic device. In some examples, the second electronic device transmits the representation of the one or more features to the first electronic device.

[0066] Figure 4 1 is a flow chart illustrating an exemplary method 400 for reconstruction using image and position data received from one or more electronic devices (e.g., 200A, 200B). In some examples, the method is performed by a server in communication with electronic device 200A and / or electronic device 200B. The server optionally includes some or all of the features of system 100. In some examples, the method is performed by electronic device 200A and / or electronic device 200B (e.g., one of the electronic devices acts as a server for the other electronic device).

[0067] At box 402, an electronic device (e.g., 100, 200A, 200B) receives first image data and a representation of a first posture of the second electronic device in three-dimensional space (e.g., a position and orientation defined by rotation and translation relative to a reference 3D coordinate system) from a second electronic device (e.g., 200A, 200B), wherein the representation of the first posture of the second electronic device corresponds to the first image data. In some examples, the first image data and the representation of the first posture are received from the second electronic device performing box 308 of method 300. In some examples, the image data corresponds to a physical object (e.g., 202A, 202B) in a physical environment (e.g., including its data). In some examples, the posture for which the representation is received is a posture of capturing at least a portion of the image data. In some examples, the image data uses a raw image format. In some examples, the image data uses a compressed file format (e.g., jpeg). In some examples, the first image data is acquired using a first camera sensor (e.g., image sensor 108) of the second electronic device. In some examples, the electronic device does not receive image data, but instead receives from a second electronic device a representation of one or more features corresponding to one or more features extracted from the image data (e.g., position data). In some examples, the features include points, lines, segments, planes, and / or other 3D geometric elements and shapes.

[0068] At box 404, the electronic device receives second image data and a representation of a second posture of the third electronic device in three-dimensional space from a third electronic device (e.g., 200A, 200B), wherein the representation of the second posture of the third electronic device corresponds to the second image data. In some examples, the image data corresponds to a physical object (e.g., 202A, 202B) in a physical environment (e.g., including its data). In some examples, the posture for which the representation is received is a posture of capturing at least a portion of the image data. In some examples, the image data uses a raw image format. In some examples, the image data uses a compressed file format (e.g., jpeg). In some examples, the second image data is acquired using a second camera sensor (e.g., image sensor 108) of the third electronic device. In some examples, the representation of the first posture of the second electronic device and the representation of the second posture of the third electronic device are on the same coordinate system (e.g., Cartesian coordinate system, spherical coordinate system). In some examples, the electronic device does not receive image data, but receives a representation (e.g., position data) of one or more features corresponding to one or more features extracted from the image data from the third electronic device.

[0069] At box 406, the electronic device generates a representation of one or more features in a three-dimensional space, wherein the representation of the one or more features is generated using the first image data, the representation of the first posture, the second image data, and the representation of the second posture. In some examples, the representation of one or more features corresponds to a physical object (e.g., 202A, 202B) in a physical environment (e.g., the representation of one or more features includes the position in space of some features of the physical object). In some examples, the features include points, lines, segments, planes, and / or other 3D geometric elements and shapes. In some examples, the representation of the one or more features in the three-dimensional space is generated in response to receiving the first image data from the second electronic device and receiving the second image data from the third electronic device (or thereafter). In some examples, the representation of the one or more features is generated in response to receiving the representation of the first posture and the representation of the second posture (or thereafter). In some examples, the representation of the one or more features is generated by performing a minimization function on at least a portion of the first image data, the representation of the first posture of the second electronic device, the second image data, and the representation of the second posture of the third electronic device (in some examples, performing a minimization function on all of these data). In some examples, the representation of the one or more features is generated by performing a minimization function on at least a portion of a representation of the one or more features received from a second electronic device, a representation of a first posture of the second electronic device, a representation of one or more features received from a third electronic device, and a representation of a second posture of the third electronic device (in some examples, performing a minimization function on all of these data).

[0070] At box 408, the electronic device transmits the representation of the one or more features to the second electronic device and the third electronic device, wherein the representation of the one or more features is configured to be used by the second electronic device to generate a representation of a third posture of the second electronic device (e.g., rotation, translation), and wherein the representation of the one or more features is configured to be used by the third electronic device to generate a representation of a fourth posture of the third electronic device (e.g., rotation, translation).

[0071] In some examples, after transmitting the representation of the one or more features to the second electronic device and the third electronic device, the electronic device receives third image data and a representation of a fifth posture of the second electronic device in three-dimensional space (e.g., such as a position and orientation defined by rotation and translation relative to a reference 3D coordinate system) from the second electronic device, wherein the representation of the fifth posture of the second electronic device (e.g., such as a position and orientation defined by rotation and translation relative to the reference 3D coordinate system) corresponds to the third image data (e.g., the posture is a posture of at least a portion of the captured image data); receives fourth image data and a representation of a sixth posture of the third electronic device in three-dimensional space (e.g., such as a position and orientation defined by rotation and translation relative to the reference 3D coordinate system) from the third electronic device. , wherein the representation of the sixth posture of the third electronic device corresponds to the fourth image data (e.g., the posture is a posture of at least a portion of the captured image data); generating a representation of one or more features in a three-dimensional space, wherein the representation of the one or more features is generated using the third image data, the representation of the fifth posture, the fourth image data, and the representation of the sixth posture; and transmitting the representation of the one or more features to the second electronic device and the third electronic device, wherein the representation of the one or more features is configured to be used by the second electronic device to generate a representation of the seventh posture of the second electronic device (e.g., rotation, translation), and wherein the representation of the one or more features is configured to be used by the third electronic device to generate a representation of the eighth posture of the third electronic device (e.g., rotation, translation). In some examples, features include points, lines, segments, planes, and / or other 3D geometric elements and shapes.

[0072] In some examples, the electronic device can transmit representations of some or all of the features generated at block 406. For example, the electronic device can transmit representations of all of the features generated at block 406, only those features whose locations have been updated from a previous iteration of method 400, only those features that have not been previously sent to a second electronic device or a third electronic device, or any other desired subset of features.

[0073] Advantageously, methods 300 and 400 improve the accuracy and speed of positioning and mapping by providing more data to operate on. In addition, methods 300 and 400 provide the electronic device with mapping data (e.g., location data of features) for areas of the environment that have not yet been observed by the electronic device. This can result in faster positioning of the electronic device and faster mapping of the environment when encountering those areas of the environment.

Claims

1. A first electronic device, comprising: one or more processors; as well as A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for: receiving, from a second electronic device, first image data and a representation of a first posture of the second electronic device in three-dimensional space; receiving, from a third electronic device, second image data and a representation of a second posture of the third electronic device in the three-dimensional space; generating a representation of one or more features in the three-dimensional space using the first image data, the representation of the first pose, the second image data, and the representation of the second pose; as well as The representation of the one or more features is transmitted to the second electronic device and the third electronic device for use in generating a representation of a gesture of the second electronic device and the third electronic device.

2. The first electronic device according to claim 1, wherein: the representation of the first gesture of the second electronic device corresponds to the first image data; and The representation of the second gesture of the third electronic device corresponds to the second image data.

3. The first electronic device according to claim 1, wherein: The representation of the one or more features is configured to be used by the second electronic device to generate a representation of a third gesture of the second electronic device, and The representation of the one or more features is configured to be used by the third electronic device to generate a representation of a fourth gesture of the third electronic device. 4 . The first electronic device of claim 1 , wherein the first image data is acquired using a first camera sensor of the second electronic device, and wherein the second image data is acquired using a second camera sensor of the third electronic device. 5 . The first electronic device of claim 1 , wherein the representation of the first gesture of the second electronic device and the representation of the second gesture of the third electronic device are on the same coordinate system.

6. A first electronic device according to claim 1, wherein the representation of the one or more features is generated by performing a minimization function on at least a portion of the first image data, the representation of the first posture of the second electronic device, the second image data, and the representation of the second posture of the third electronic device.

7. The first electronic device according to claim 1, wherein the one or more programs further comprise instructions for: After transmitting the representation of the one or more characteristics to the second electronic device and the third electronic device: receiving, from the second electronic device, third image data and a representation of a fifth posture of the second electronic device in the three-dimensional space, wherein said representation of said fifth posture of said second electronic device corresponds to said third image data; receiving, from the third electronic device, fourth image data and a representation of a sixth posture of the third electronic device in the three-dimensional space, wherein said representation of said sixth posture of said third electronic device corresponds to said fourth image data; generating a second representation of one or more features in the three-dimensional space, wherein the second representation of the one or more features is generated using the third image data, the representation of the fifth pose, the fourth image data, and the representation of the sixth pose; and transmitting the second representation of the one or more characteristics to the second electronic device and the third electronic device, wherein the second representation of the one or more features is configured to be used by the second electronic device to generate a representation of a seventh gesture of the second electronic device, and Wherein the second representation of the one or more features is configured to be used by the third electronic device to generate a representation of an eighth gesture of the third electronic device.

8. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a first electronic device, the one or more programs comprising instructions for: receiving, from a second electronic device, first image data and a representation of a first posture of the second electronic device in three-dimensional space; receiving, from a third electronic device, second image data and a representation of a second posture of the third electronic device in the three-dimensional space; generating a representation of one or more features in the three-dimensional space using the first image data, the representation of the first pose, the second image data, and the representation of the second pose; as well as The representation of the one or more features is transmitted to the second electronic device and the third electronic device for use in generating a representation of a gesture of the second electronic device and the third electronic device.

9. The computer-readable storage medium of claim 8, wherein: the representation of the first gesture of the second electronic device corresponds to the first image data; and The representation of the second gesture of the third electronic device corresponds to the second image data.

10. The computer-readable storage medium of claim 8, wherein: The representation of the one or more features is configured to be used by the second electronic device to generate a representation of a third gesture of the second electronic device, and The representation of the one or more features is configured to be used by the third electronic device to generate a representation of a fourth gesture of the third electronic device.

11. The computer-readable storage medium of claim 8, wherein the first image data is acquired using a first camera sensor of the second electronic device, and wherein the second image data is acquired using a second camera sensor of the third electronic device.

12. The computer-readable storage medium of claim 8, wherein the representation of the first gesture of the second electronic device and the representation of the second gesture of the third electronic device are on a same coordinate system.

13. A computer-readable storage medium according to claim 8, wherein the representation of the one or more features is generated by performing a minimization function on at least a portion of the first image data, the representation of the first posture of the second electronic device, the second image data, and the representation of the second posture of the third electronic device.

14. The computer-readable storage medium of claim 8, wherein the one or more programs further comprise instructions for: After transmitting the representation of the one or more characteristics to the second electronic device and the third electronic device: receiving, from the second electronic device, third image data and a representation of a fifth posture of the second electronic device in the three-dimensional space, wherein said representation of said fifth posture of said second electronic device corresponds to said third image data; receiving, from the third electronic device, fourth image data and a representation of a sixth posture of the third electronic device in the three-dimensional space, wherein said representation of said sixth posture of said third electronic device corresponds to said fourth image data; generating a second representation of one or more features in the three-dimensional space, wherein the second representation of the one or more features is generated using the third image data, the representation of the fifth pose, the fourth image data, and the representation of the sixth pose; and transmitting the second representation of the one or more characteristics to the second electronic device and the third electronic device, wherein the second representation of the one or more features is configured to be used by the second electronic device to generate a representation of a seventh gesture of the second electronic device, and Wherein the second representation of the one or more features is configured to be used by the third electronic device to generate a representation of an eighth gesture of the third electronic device.

15. A method comprising: At the first electronic device: receiving, from a second electronic device, first image data and a representation of a first posture of the second electronic device in three-dimensional space; receiving, from a third electronic device, second image data and a representation of a second posture of the third electronic device in the three-dimensional space; generating a representation of one or more features in the three-dimensional space using the first image data, the representation of the first pose, the second image data, and the representation of the second pose; as well as The representation of the one or more features is transmitted to the second electronic device and the third electronic device for use in generating a representation of a gesture of the second electronic device and the third electronic device.

16. The method of claim 15, wherein: the representation of the first posture of the second electronic device corresponds to the first image data; the representation of the second posture of the third electronic device corresponds to the second image data; The representation of the one or more features is configured to be used by the second electronic device to generate a representation of a third gesture of the second electronic device, and The representation of the one or more features is configured to be used by the third electronic device to generate a representation of a fourth gesture of the third electronic device.

17. The method of claim 15, wherein the first image data is acquired using a first camera sensor of the second electronic device, and wherein the second image data is acquired using a second camera sensor of the third electronic device.

18. The method of claim 15, wherein the representation of the first gesture of the second electronic device and the representation of the second gesture of the third electronic device are on a same coordinate system.

19. A method according to claim 15, wherein the representation of the one or more features is generated by performing a minimization function on at least a portion of the first image data, the representation of the first posture of the second electronic device, the second image data, and the representation of the second posture of the third electronic device.

20. The method of claim 15, further comprising: After transmitting the representation of the one or more characteristics to the second electronic device and the third electronic device: receiving, from the second electronic device, third image data and a representation of a fifth posture of the second electronic device in the three-dimensional space, wherein said representation of said fifth posture of said second electronic device corresponds to said third image data; receiving, from the third electronic device, fourth image data and a representation of a sixth posture of the third electronic device in the three-dimensional space, wherein said representation of said sixth posture of said third electronic device corresponds to said fourth image data; generating a second representation of one or more features in the three-dimensional space, wherein the second representation of the one or more features is generated using the third image data, the representation of the fifth pose, the fourth image data, and the representation of the sixth pose; and transmitting the second representation of the one or more characteristics to the second electronic device and the third electronic device, wherein the second representation of the one or more features is configured to be used by the second electronic device to generate a representation of a seventh gesture of the second electronic device, and Wherein the second representation of the one or more features is configured to be used by the third electronic device to generate a representation of an eighth gesture of the third electronic device.

Images