Object scanning for subsequent object detection

Abstract

Various embodiments disclosed herein include devices, systems, and methods that facilitate the creation of a 3D model for object detection based on a scan of an object. Some embodiments provide a user interface with which a user interacts to facilitate the scanning of an object, thereby creating a 3D model of the object for later object detection. The user interface may include an indicator that provides visual or auditory feedback to the user, indicating the direction in which the capture device is facing relative to the scanned object. Sensors on the device (e.g., an inertial measurement unit (IMU), a gyroscope, etc.) or other technologies (e.g., visual inertial odometry (VIO)) may be used and the direction of the capture device may be identified based on the user positioning the device so that the object is in view.

Description

Technical Field

[0001] The present disclosure generally relates to systems, devices, and methods for scanning physical objects to facilitate detection and tracking of those objects in images and videos. Background Art

[0002] Various techniques can be used to detect objects in images or videos. Some object detection techniques are based on previous scans of an object that generate a model or other information about the object. Once a user has generated such a model or other information for an object, the model or other information can be compared with images and videos obtained later to identify the object in those images and videos. For example, a user can perform a scan of a statue to generate a 3D model of the statue. Other users can then walk around the museum where the statue is located, using mobile phones or other devices to capture images or videos, and the 3D model can be used to detect the presence and location of the statue in those images and videos.

[0003] Existing techniques for capturing scans that can be used for object detection later may have certain disadvantages. For example, such techniques may require users to scan objects from multiple directions to capture videos or images of the object, but may not be sufficient to guide users to effectively and easily obtain such scans. It may be difficult for a user to know whether he or she has scanned the object from enough directions or how to move the camera to obtain additional required images. Existing techniques may also require manual identification of the object in an image or 3D representation of the object. For example, a user may be required to locate a 3D bounding box that roughly estimates the position and range of the object. The system may use the 3D bounding box to limit the detected key points that will be attributed to the object, for example, allowing the system to ignore key points detected on a nearby desktop or from a nearby object to provide an accurate 3D model of the object. Positioning such a 3D bounding box may be difficult for users, particularly for users who try to use a 2D display and interface to do so. For these and other reasons, existing techniques for capturing scans that can be used for object detection later may be burdensome to users and may not be sufficient to facilitate distinguishing objects from other elements of a physical setting. Summary of the invention

[0004] Various embodiments disclosed herein include devices, systems, and methods that facilitate the creation of a 3D model for object detection based on a scan of an object. Some embodiments provide a user interface with which a user interacts to facilitate the scanning of an object, thereby creating a 3D model of the object for later object detection. The user interface may include an indicator that provides visual or auditory feedback to the user, indicating the direction in which the capture device is facing relative to the scanned object. The orientation of the capture device may be detected using sensors on the device (e.g., an inertial measurement unit (IMU), a gyroscope, etc.) or other techniques (e.g., visual inertial odometry (VIO)). The orientation of the device relative to an object in a physical scene may be detected based on detecting a change in the orientation of the capture device in a coordinate system of the physical scene.

[0005] As the user walks around the object or otherwise moves the device with the device's camera facing the object, the indicator shows or informs the user of the current camera direction. In some implementations, the image from the device's camera is presented on a portion of the display, such as as a real-time video, and the indicator is presented on multiple sides of the displayed camera content, such as a ring around the real-time video image from the device's camera. The elements of such indicators may change in appearance to indicate the current direction of the camera relative to the object. For example, the user may hold the device to view the image of the object within the oval indicator on the device display. A portion of the indicator (e.g., an element on one side of the indicator and therefore on one side of the object) may have an appearance indicating the current camera direction (e.g., magnified, colored red, etc.). When the user moves the device to the opposite side of the object and holds the device again to view another image of the object within the oval indicator on the device display, a different portion of the indicator (e.g., an element on the opposite side of the indicator and therefore on the opposite side of the object) may be given an appearance indicating the current camera direction (e.g., magnified, colored red, etc.).

[0006] Such user interfaces with indicators indicating camera directions provide useful feedback to users during the object scanning process. During such scanning processes, users may move devices around objects to view objects from different directions, and may choose to create image captures from certain directions in certain cases. When a user recognizes such a "capture", the appearance of the indicator may change to indicate that the user has created a capture from the corresponding camera direction. For example, the indicator element may change in appearance, such as to a permanent state, to reflect that the capture of the object has been initiated by the user. For example, a user may perform a first image capture from the front side of an object and the element on the bottom of the indicator may permanently turn yellow, then perform a second image capture from the left side of the object and the element on the left side of the indicator may permanently turn yellow, then perform a third image capture from the back side of the object and the element on the top of the indicator may permanently turn yellow, then perform a fourth image capture from the right side of the object and the element on the right side of the indicator may permanently turn yellow. The user can easily visualize which direction the image of the object has been captured, which direction has not been captured, and can also see the current direction of the device. Once the user has captured enough image captures from enough directions, a 3D model of the object can be automatically generated for subsequent object detection. The user interface may include additional facilitating features as described herein and as otherwise appropriate given a particular implementation (eg, directing the user to also obtain top-down or bottom-up images).

[0007] Some specific implementations of the present disclosure involve performing various operations on a computing device having a processor, a memory, and an image sensor to facilitate creating a 3D model for object detection based on a scan of an object. An exemplary method involves providing an indicator in a user interface that indicates possible directions from which an image of an object in a physical setting can be captured. The method modifies the indicator based on the direction of a device (e.g., a camera of the device) relative to the object. Thus, for example, a user interface may provide a graphical indicator that is a dashed circle displayed around content captured by the camera of the device. As a user moves the device around the object and based on the current orientation of the device and the resulting direction of the device relative to the object, the shape, size, or color of the different dashes around the circle may change to indicate the current direction of the device relative to the object. The method captures images of an object from multiple directions, for example, automatically and / or based on user input to create such a capture, and generates a model of the object (e.g., a feature point cloud) for detecting the object in subsequent images.

[0008] Some embodiments of the present disclosure involve performing various operations on a computing device having a processor, a memory, and an image sensor to facilitate identification of an object during creation of a 3D model for object detection based on scanning of the object. In some embodiments, a model of an object is generated based on user identification of a user-captured image, which is used to distinguish the object from other items in the image that would otherwise be part of the 3D model. An exemplary method records images of an object from multiple directions. Such images may be recorded continuously, for example, as a video. The method identifies a subset of images as user-captured images during recording of the images. For example, as the user moves around the object, the user may select a user interface command (e.g., a capture button) at some point. For example, 10 images out of 200 recorded images may be identified as user-captured images.

[0009] The method generates a model (e.g., a feature point cloud) based on images of the object (e.g., using all 200 images). Generating the model may involve distinguishing the object from nearby items. Therefore, the method may identify a portion of the model corresponding to the object based on the image (e.g., using a subset of the user-captured images). In some implementations, each of the user-captured images created based on user input (e.g., when the user selects a capture button) is associated with a spatial volume (e.g., a spatial cone associated with the images captured during that time). In some implementations, the user captures these user-captured images, where the object is completely displayed within a boundary on the display (e.g., within a circular graphical indicator). The method may identify the spatial volume (e.g., cone) associated with each capture, and distinguish the object from other items based on the understanding that the object is within those spatial volumes. As the user captures the user-captured images from different directions (e.g., selecting a capture button), multiple volumes are identified. The location of the object may be determined based on the intersection of those volumes (e.g., within the intersection of multiple cones) and distinguished from other items in the physical scene. Furthermore, the model may be generated on an ongoing basis, and each capture may be used to crop out portions of the model that are outside the volume of space associated with that capture. During scanning from multiple directions, the model of the object may be appropriately constrained to include only the object and exclude other items that might otherwise have been included in the model.

[0010] According to some specific implementations, a device includes one or more processors, non-volatile memory, and one or more programs; the one or more programs are stored in the non-volatile memory and are configured to be executed by one or more processors, and the one or more programs include instructions for performing or causing the execution of any of the methods described herein. According to some specific implementations, a non-volatile computer-readable storage medium stores instructions that, when executed by one or more processors of the device, cause the device to perform or cause the execution of any of the methods described herein. According to some specific implementations, a device includes one or more processors, non-volatile memory, an image sensor, and a device for performing or causing the execution of any of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] So that the present disclosure may be understood by those of ordinary skill in the art, a more detailed description may be obtained with reference to aspects of some exemplary implementations, some of which are illustrated in the accompanying drawings.

[0012] Figure 1 Exemplary physics objects in a physics scene are shown according to some implementations.

[0013] Figure 2 According to some specific implementations, a display having Figure 1 A mobile device with an augmented reality (ER) scene of an exemplary physical object.

[0014] Figure 3 According to some specific implementations Figure 1 A scan of an exemplary physical object.

[0015] Figure 4 According to some specific implementations, Figure 3 Example point cloud generated from the first scan of .

[0016] Figure 5 Exemplary components of an apparatus for generating a 3D model of a physical object and detecting the physical object are shown according to some implementations.

[0017] Figure 6 is a flowchart representation of a method of creating a model of an object according to some specific implementations.

[0018] Figure 7 Shown in Figure 3 The orientation of the device relative to the object at a first instance during a scan of the object.

[0019] Figure 8 According to some embodiments, a display is shown to indicate that the device is Figure 7 An indicator of direction during the first instance depicted in .

[0020] Fig. 9 Shown in Figure 3 The orientation of the device relative to the object at a later instance during a scan of the object.

[0021] Fig.10 According to some embodiments, a display is shown to indicate that the device is Fig. 9 An indicator of direction during the later instance depicted in .

[0022] Fig.11 An indicator is shown that is displayed to indicate to a user that an image captured is captured from a certain direction, according to some implementations.

[0023] Fig.12 An indicator is shown that permanently modifies a display to indicate to a user that an image captured was previously captured from a certain direction, according to some implementations.

[0024] Fig.13 An indicator is shown displayed to indicate that images captured by multiple users are captured from multiple directions, according to some implementations.

[0025] Fig.14 is a flowchart representation of a method for facilitating identification of an object during creation of a 3D model for object detection based on a scan of the object, according to some implementations.

[0026] Fig.15 A model for clipping an object based on a user-captured image of the object during scanning is shown according to some implementations.

[0027] Fig.16 User interface features that facilitate an exemplary scanning workflow are shown.

[0028] Fig.17 User interface features that facilitate an exemplary scanning workflow are shown.

[0029] Fig.18 User interface features that facilitate an exemplary scanning workflow are shown.

[0030] As is common practice, the various features shown in the drawings may not be drawn to scale. Therefore, the sizes of the various features may be arbitrarily expanded or reduced for clarity. In addition, some drawings may not depict all components of a given system, method, or device. Finally, throughout the specification and drawings, similar reference numerals may be used to represent similar features. DETAILED DESCRIPTION

[0031] Many details are described in order to provide a thorough understanding of the example implementations shown in the accompanying drawings. However, the accompanying drawings only illustrate some example aspects of the present disclosure and should not be considered limiting. One of ordinary skill in the art will appreciate that other effective aspects or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail in order to avoid obscuring more relevant aspects of the exemplary implementations described herein.

[0032] Figure 1 is a block diagram illustrating an example of a physics object 105 in a physics scene 100. Although this example and other examples discussed herein illustrate a 3D model of a single object 105, the techniques disclosed herein are also applicable to multiple objects.

[0033] The phrase "physical scenery" refers to the world that individuals can sense and / or interact with without the use of electronic systems. A physical scenery, such as a physical park, includes physical elements, such as physical wildlife, physical trees, and physical plants. People can directly sense and / or otherwise interact with the physical scenery, for example, using one or more senses, including vision, smell, touch, taste, and hearing.

[0034] As used herein, the phrase "physical object" refers to any type of item or combination of items in a physical set, including but not limited to building blocks, toys, figurines, furniture, doors, buildings, pictures, paintings, sculptures, lamps, signs, tables, floors, walls, desks, bodies of water, human faces, human hands, human hair, other body parts, entire human bodies, animals or other organisms, clothing, paper, magazines, books, vehicles, machines or other man-made objects, and any other natural or man-made item or group of items that are present in a physical set and can be identified and modeled.

[0035] Figure 2 Yes Display Figure 1 1 is a block diagram of a mobile device 120 with an augmented reality (ER) scene 200 of an exemplary physical object 105. In contrast to a physical scene, an augmented reality (ER) scene refers to a fully (or partially) computer-generated scene that various people can sense and / or otherwise interact with through the use of an electronic system. In ER, the movement of a person is monitored in part, and in response thereto, at least one property corresponding to at least one virtual object in the ER scene is changed in a manner consistent with one or more laws of physics. For example, in response to the ER system detecting that the person is looking up, the ER system can adjust various audio and graphics presented to the person in a manner consistent with the way such sounds and appearances would change in a physical scene. Adjustments to properties of virtual objects in the ER scene can also be made, for example, in response to representations of movement (e.g., voice commands).

[0036] A person may sense and / or interact with ER objects using one or more senses, such as vision, smell, taste, touch, and hearing. For example, a person may sense and / or interact with objects that create a multi-dimensional or spatial acoustic scenery. A multi-dimensional or spatial acoustic scenery provides an individual with the perception of discrete sound sources in a multi-dimensional space. Such objects may also implement acoustic transparency, which may selectively incorporate audio from the physical scenery with or without computer-generated audio. In some ER scenery, a person may sense and / or interact with only audio objects.

[0037] Virtual reality (VR) is an example of ER. A VR set refers to an augmented set that is configured to include only computer-generated sensory input for one or more senses. The VR set includes a plurality of virtual objects that a person can sense and / or interact with. A person can sense and / or interact with virtual objects in a VR set by simulating at least some of the person's actions within the computer-generated set and / or by simulating a person or his / her presence within the computer-generated set.

[0038] Mixed reality (MR) is another example of ER. An MR set refers to an augmented set that is configured to integrate computer-generated sensory input (e.g., virtual objects) with sensory input from a physical set or a representation of sensory input from a physical set. On the reality spectrum, MR sets are between and do not include fully physical sets at one end and VR sets at the other end.

[0039] In some MR sets, computer-generated sensory inputs may be adjusted based on changes in sensory inputs from the physical set. Additionally, some electronic systems used to render the MR set may detect position and / or orientation relative to the physical set to enable interaction between real objects (i.e., physical elements from the physical set or representations thereof) and virtual objects. For example, the system may detect movement and adjust computer-generated sensory inputs accordingly so that, for example, a virtual tree appears stationary relative to a physical structure.

[0040] Augmented reality (AR) is an example of MR. AR scenery refers to an enhanced scenery in which one or more virtual objects are superimposed on a physical scenery (or a representation thereof). For example, an electronic system may include an opaque display and one or more imaging sensors for capturing a video and / or image of a physical scenery. For example, such a video and / or image may be a representation of a physical scenery. The video and / or image is combined with a virtual object, where the combination is then displayed on an opaque display. The physical scenery can be viewed indirectly by a person via an image and / or video of the physical scenery. Therefore, a person can observe a virtual object superimposed on the physical scenery. When the system captures an image of the physical scenery and uses the captured image to display the AR scenery on an opaque display, the displayed image is referred to as video transmission. Alternatively, a transparent or translucent display may be included in an electronic system for displaying an AR scenery, so that an individual can directly view the physical scenery through a transparent or translucent display. Virtual objects may be displayed on a translucent or transparent display, so that an individual observes virtual objects superimposed on the physical scenery. In another example, a projection system may be utilized to project virtual objects onto the physical scenery. For example, virtual objects may be projected onto a physical surface, or as a hologram, such that an individual observes the virtual object superimposed on a physical setting.

[0041] AR scenery may also refer to an augmented scenery in which a representation of a physical scenery is modified by computer-generated sensory data. For example, at least a portion of the representation of the physical scenery can be graphically modified (e.g., enlarged) so that the modified portion can still represent the originally captured image (but not an exact copy). Alternatively, when providing video pass-through, one or more sensor images may be modified to apply a specific viewpoint that is different from the viewpoint captured by the image sensor. For another example, a portion of the representation of the physical scenery may be changed by graphically blurring or eliminating the portion.

[0042] Augmented Virtual (AV) is another example of MR. An AV set refers to an augmented set that is a virtual or computer-generated set combined with one or more sensory inputs from a physical set. Such sensory inputs may include representations of one or more features of the physical set. Virtual objects may, for example, incorporate colors associated with physical elements captured by an imaging sensor. Alternatively, virtual objects may adopt features consistent with, for example, current weather conditions corresponding to the physical set, such as weather conditions identified via imaging, online weather information, and / or weather-related sensors. For another example, an AR park may include virtual structures, plants, and trees, although animals within the AR park set may include features accurately replicated from images of physical animals.

[0043] Various systems allow people to sense and / or interact with the ER scenery. For example, a head-mounted system may include one or more speakers and an opaque display. As another example, an external display (e.g., a smart phone) may be incorporated into the head-mounted system. The head-mounted system may include a microphone for capturing audio of the physical scenery and / or an image sensor for capturing images / video of the physical scenery. A transparent or translucent display may also be included in the head-mounted system. A translucent or transparent display may, for example, include a substrate through which light (representing an image) is directed to a person's eyes. The display may also include an LED, an OLED, a silicon-based liquid crystal, a laser scanning light source, a digital light projector, or any combination thereof. The substrate through which the light is transmitted may be an optical reflector, a holographic substrate, an optical waveguide, an optical combiner, or any combination thereof. A transparent or translucent display may, for example, selectively transition between a transparent / translucent state and an opaque state. As another example, the electronic system may be a projection-based system. In a projection-based system, retinal projection may be used to project an image onto a person's retina. Alternatively, the projection-based system may also project the virtual object into a physical setting, such as, for example, projecting the virtual object as a hologram or onto a physical surface. Other examples of ER systems include a window configured to display graphics, headphones, earphones, a speaker arrangement, a lens configured to display graphics, a head-up display, a car windshield configured to display graphics, an input mechanism (e.g., a controller with or without haptic capabilities), a desktop or laptop computer, a tablet computer, or a smart phone.

[0044] exist Figure 2 In the example of , device 120 captures one or more images of physical object 105 and compares those images to the 3D model (e.g., previously captured keyframes, point cloud values, etc.) to detect and track the presence of physical object 105. Device 120 may, for example, use RGB-D information, infrared camera-based depth detection, and other such techniques to determine the pose (e.g., position and orientation) of physical object 105. Thus, after detecting the physical object and determining its pose, device 120 may align the 3D model of the object with the physical object in a coordinate system corresponding to the physical space of the physical scene.

[0045] In view of this alignment, the device is able to provide an ER scene 200 that combines aspects of the physical scene with augmented content. In this example, the ER scene 200 includes a depiction 210 of the physical object 105 and augmented content 215 that includes a text bubble and the text phrase "It's magnetic." The augmented content 215 is positioned relative to the depiction 210 of the physical object 105 based on the alignment of the physical object 105 with the 3D model. For example, the creator of the ER scene 200 may have specified that the augmented content 215 is to be displayed at a location determined based on a fixed point at the center of the surface of the 3D model of the physical object 105. Once the 3D model is aligned with the physical object 105, the device 120 determines the appropriate location for the augmented content 215 and generates the ER scene 200 for display.

[0046] exist Figure 3 In the example, scanning 300 includes moving the device (e.g., Figure 2 The scan is performed while capturing an image of the physical object 105 while the device 120 is being scanned. Paths 315a-315i include the device in multiple image sensor poses 310a, 310b, 310c, 310d, 310e, 310f, 310g, 310i, 310j from an initial pose 310a at the beginning of image recording to a final pose 310j at the end of recording. During scan 300, the device may track its own pose (e.g., via tracking position and orientation changes of one or more motion sensors on the device, inertial data, etc.). Therefore, the relative position of the data for the images captured at each of the poses 310a, 310b, 310c, 310d, 310e, 310f, 310g, 310i, 310j in the first 3D coordinate system is known. The image data from these images may be combined to generate a 3D model of the physical object based on these known poses (e.g., Figure 4 ).

[0047] Figure 4 Shown by Figure 3 A 3D model (eg, an exemplary point cloud) 400 generated by scanning of a 3D image sensor. The 3D model 400 may include points having descriptors, the descriptors including color values ​​of the points.

[0048] The device used by content creators (e.g., for image capture and 3D model creation) and end users (e.g., for object detection using 3D models) can be any of a variety of devices including a processor, a non-transitory computer-readable medium, and an image sensor. In some implementations, the device is a head mounted device (HMD) worn by a content creator or an end user. In some implementations, the device is a handheld electronic device (e.g., a smartphone or tablet), a laptop, or a desktop computer that is configured to create 3D models of physical objects and facilitate the creation of ER scenes (e.g., for content creators), or to present ER scenes to end users. In some implementations, the device is an ER cabin, housing, or room configured to present ER scenes, wherein the end user does not wear or hold the device.

[0049] Figure 5 is a block diagram illustrating exemplary components of a device for generating a 3D model of a physical object and detecting the physical object according to some implementations. In various implementations, these functions may be separated into one or more independent devices. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, as a non-limiting example, in some specific implementations, the device 120 includes one or more processing units 502 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 506, one or more communication interfaces 508 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, SPI, I2C, or similar types of interfaces), one or more programming (e.g., I / O) interfaces 510, one or more displays 512, one or more internal or external-facing image sensor systems 514, a memory 520, and one or more communication buses 504 for interconnecting these components and various other components.

[0050] In some implementations, the one or more communication buses 504 include circuits that interconnect and control communications between system components. In some implementations, the one or more I / O devices and sensors 506 include at least one of a touch screen, soft keys, keyboards, virtual keyboards, buttons, knobs, joysticks, switches, dials, inertial measurement units (IMUs), accelerometers, magnetometers, gyroscopes, thermometers, one or more physiological sensors (e.g., blood pressure monitors, heart rate monitors, blood oxygen sensors, blood glucose sensors, etc.), one or more microphones, one or more speakers, a haptic engine, one or more depth sensors (e.g., structured light, time of flight, etc.), etc. In some implementations, movement, rotation, or position of the device 120 detected by the one or more I / O devices and sensors 506 provides input to the device 120.

[0051] In some implementations, the one or more displays 512 are configured to present ER content. In some implementations, the one or more displays 512 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light emitting field effect transistor (OLET), organic light emitting diode (OLED), surface conduction electron emitter display (SED), field emission display (FED), quantum dot light emitting diode (QD-LED), microelectromechanical system (MEMS), or similar display types. In some implementations, the one or more displays 512 correspond to diffractive, reflective, polarized, holographic, etc. waveguide displays. In one example, the device 120 includes a single display. In another example, the device 120 includes a display for each eye. In some implementations, the one or more displays 512 are capable of presenting ER content.

[0052] In some implementations, the one or more image sensor systems 514 are configured to obtain image data corresponding to at least a portion of a local scene of the device 120. The one or more image sensor systems 514 can include one or more RGB cameras (e.g., with complementary metal oxide semiconductor (CMOS) image sensors or charge coupled device (CCD) image sensors), RGB-D cameras, monochrome cameras, IR cameras, or event-based cameras, etc. In various implementations, the one or more image sensor systems 514 also include an illumination source that emits light, such as a flash.

[0053] The memory 520 includes a high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid-state memory devices. In some specific implementations, the memory 520 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices or other non-volatile solid-state storage devices. The memory 520 optionally includes one or more storage devices remotely located from the one or more processing units 502. The memory 520 includes a non-transitory computer-readable storage medium. In some specific implementations, the memory 520 or the non-transitory computer-readable storage medium of the memory 520 stores the following programs, modules and data structures, or a subset thereof, including an optional operating system 530 and one or more application programs 540.

[0054] The operating system 530 includes processes for handling various basic system services and for performing hardware-related tasks. In some implementations, the operating system 530 includes built-in ER functionality, such as an ER scene creation feature or an ER scene viewer configured to be called from one or more application programs 540. The application programs 540 include a scanner 542 configured to create scans of physical objects and create 3D models of the physical objects based on the scans. The application programs 540 also include detectors 544 configured to detect physical objects in images obtained later, such as during ER scene rendering, using these 3D models.

[0055] Figure 5 It is more of a functional description of various features present in a particular implementation, as opposed to a schematic diagram of the implementations described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Fig. 9 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various specific implementations. The actual number of modules and the division of specific functions and how the features are allocated among them will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software, or firmware selected for a specific specific implementation.

[0056] Figure 6 is a flowchart representation of a method for creating a model of an object according to some specific implementations. In some specific implementations, method 600 is performed by a device (e.g., Figure 2 and Figure 5The method 600 may be performed on a mobile device, an HMD, a desktop computer, a laptop computer, or a server device. In some implementations, the method 600 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the method 600 is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory).

[0057] At block 610, method 600 provides an indicator in a user interface that indicates a possible direction from which an image of an object in a physical scene can be captured. In some implementations, the indicator is a ring of elements (e.g., dashes) displayed around a real-time image captured by a sensor on a user's device. In some implementations, the indicator is a ring of dashes oriented radially outward from a center of the ring. In some implementations, the indicator indicates a possible direction perpendicular to a direction of gravity identified using a sensor on the device.

[0058] In some implementations, method 600 captures and / or records real-time images of the physical set and displays a portion of those images (e.g., some or all pixels) on the user's device. In some implementations, only a subset of the pixels of each image associated with the area within the indicator (e.g., within the element ring of the indicator) is displayed. Other portions of the images may be cropped. In some implementations, the indicator is displayed on top of the images from the device, for example, a dashed line ring may be displayed on top of each image, indicating that the relevant portion of each image is the portion within the indicator.

[0059] At block 620, method 600 modifies the indicator based on the orientation of the device (e.g., the device's camera) relative to the object. Thus, for example, a user interface may provide a graphical indicator that is a dashed circle displayed around content captured by the device's camera. As the user moves the device around the object and based on the device's current orientation and the resulting orientation of the device relative to the object, the shape, size, or color of the different dashes around the circle may change to indicate the current orientation of the device relative to the object. In some implementations, the indicator is modified by changing the size, shape, or color of a subset of elements to indicate the orientation of the device. For example, the elements of the indicator may have a normal size, shape, and color except for the elements of the indicator on the relevant side associated with the current orientation of the device. In some implementations, the indicator is modified by changing the size, shape, or color of a first subset of elements from a first state (e.g., normal) to a second state (e.g., zoomed in) to indicate the current orientation of the device, and by returning the size, shape, or color of a second subset of elements from the second state (e.g., zoomed in) to the second state (e.g., normal) to indicate that the device is no longer oriented in the previous orientation. In some implementations, the indicator includes radially oriented elements, and modifying the indicator involves changing the length, thickness, or color of a subset of those elements to indicate the orientation of the device.

[0060] Note that method 600 does not require (e.g., via computer vision) to actually determine that the device is on a particular side of an object. Instead, method 600 may rely on the user placing the object in the current view of the camera and the orientation of the device determined, for example, based on sensors on the device.

[0061] In some implementations, the orientation of the device relative to an object in the physical scene is detected based on detecting a change in the orientation of the device in a coordinate system of the physical scene. In some implementations, the orientation of the device relative to the object in the physical scene is based on motion data from a motion sensor on the device that indicates a change in a position or orientation of the device in the physical scene.

[0062] At block 630, method 600 captures images of the object from multiple directions. Such images may be recorded, for example, automatically and / or based on user input to create such captures. In some implementations, all recorded images will be used to generate a model of the object. In other implementations, only a subset of the images (e.g., only user-captured images) will be used to generate the model. In some implementations, images are recorded continuously, and a subset of those images are identified as user-captured images based on user input.

[0063] Identifying some images as user-captured images can be used to change the appearance of the indicator. For example, a permanent (e.g., for the remainder of the current scan) change is created to the color, shape, or size of the indicator to indicate that the user has initiated a user-captured image from one or more specific directions. In some implementations, the method receives a captured image as input of a user-captured image, and modifies the indicator based on the orientation of the device relative to the object when the input is received. In some implementations, the modification of the indicator based on the input of the captured image is persistent during the scan of the object. For example, the indicator element can be changed to be thicker and yellow in color, and remain thick and yellow, even when the user subsequently moves to the other side of the object and changes the orientation of the device.

[0064] In some implementations, the method 600 guides the user to provide additional information for the model to be generated. For example, the method 600 may request the user to obtain a top-down image of the object, obtain a bottom-up image of the object, etc.

[0065] At block 640, method 600 generates a model of the object for use in detecting the object in subsequent images. The 3D model may be a point cloud of points associated with descriptors. The point cloud may include points with descriptors based on the scan data. All images or a subset of images (e.g., only user-captured images) may be used to distinguish the object from other items in the physical scene that should not be included in the model.

[0066] In some embodiments, at the beginning of the scanning operation, the device or method presents a user interface in which the camera feed is displayed together with the capture button. The device or method visually or audibly instructs or asks the user to take a photo from the front of the object / take a photo of the front of the object. The user points the camera in front of the object and presses the capture button. This, together with the gravity from the IMU, is used to orient the coordinate system of the scan. The device or method presents an indicator with dashes to indicate the device direction. For example, the indicator (e.g., its shape, perimeter, etc.) can identify the coverage of the object that the current image has. The size, shape, color, or other attributes of the indicator can indicate the current direction of the device. As the user moves the device around the object and based on the orientation of the capture device and the resulting direction of the device relative to the object, different dashes around the circle can be modified (e.g., enlarged or reduced in size) to indicate the current direction of the device relative to the object.

[0067] When the user clicks the capture button again, the modification to the indicator element (e.g., a thicker dash associated with the current device orientation) may be to change color and be permanently in a modified (e.g., thicker) state. This lets the user know that a particular side of the object has been scanned, and the user can focus on scanning other parts or sides of the object. The user continues to move around the object and click the capture button until the dashes on the ring are all permanently modified (e.g., thick and a different color). Finally, the user may be asked to take a photo of the object from a top-down perspective. The images of the object from the scan and top-down perspective (as well as any other relevant information obtained during the process) can then be used to generate a model of the object. Note that 100% coverage (e.g., all thicker colored dashes) may not be required for the process to work, but may be used as a threshold to provide a model of at least a threshold quality. A lower quality model may be obtained without all the required images.

[0068] Figure 7 Shown in Figure 3 105. In this example, the device has an image sensor posture 310a in front of the object 105. In posture 310a, the device is oriented toward the front of the object 105. Alternatively, the device can also be described as pointing in front of the object 105.

[0069] Figure 8 Shown in Figure 7 An indicator is displayed during the first instance to indicate the orientation of the device 120 relative to the object 105. The indicator includes elements displayed around the depiction of the object, such as elements 820a-820e. In this example, the indicator elements 820a-820e are enlarged to indicate that the device has an orientation toward the front of the object 105. In some implementations, the initial orientation of the device at the beginning of the scan is associated with a particular indicator element and an associated orientation (e.g., the initial orientation can be assumed to be the front and indicated by the bottom element of the indicator).

[0070] In addition, multiple indicator elements can be used to indicate a particular direction. Figure 8 As shown, one indicator (820c) may indicate the orientation of the device, while additional nearby indicators indicate that the current orientation of the device is also sufficient to obtain information associated with images from other directions (e.g., 820a, 820b, 820d, 820e). In some implementations, multiple indicator element modifications are provided to make the indicator modifications easier to understand. In some implementations, multiple indicator element modifications indicate that the current image will provide image capture data associated with multiple camera directions, and therefore it is not necessary to record images from those other (nearby) directions to generate a model of the object.

[0071] Fig. 9 Shown in Figure 3 105. The device has an image sensor pose 310f behind object 105 at a later instance during a scan of the object.

[0072] Fig.10 is displayed to indicate that the device is Fig. 9 105. The indicator still includes elements displayed around the depiction of the object, such as elements 820a-820e and elements 820f-820j). In this example, indicator elements 820f-820j are enlarged to indicate that the device has an orientation facing the rear side of object 105, and indicator elements 820a-820e are reduced in size to indicate that the device no longer has an orientation facing the rear side of object 105. Figures 7 and 8 direction.

[0073] Fig.11 104. An indicator is shown displayed to indicate to the user that the image captured is captured from a certain direction. In this example, when the device has an image sensor posture 310a located in front of the object 105 (e.g., as Figure 7 105 ), the user has provided user input to capture the user-captured image. In this example, indicator elements 820a-820e are enlarged to indicate that the device has a direction toward the front of object 105, and the indicator elements have a modified color (e.g., from red to yellow) to indicate that the user has captured the user-captured image from that direction. These modifications can become permanent (e.g., for the remainder of the scan) based on the user input that captured the user-captured image.

[0074] Fig.12 An indicator is shown to modify the display to indicate to the user that the image captured was previously captured from a certain direction. Fig.11 After the user captures the image, the user moves the device to an image sensor gesture 310j behind the object 105 (e.g., as Fig. 9 ). In this example, indicator elements 820f-820j are enlarged to indicate that the device has a direction toward the back side of object 105. In this example, indicator elements 820a-820e continue to have the new color (e.g., yellow), indicating that the user previously captured the image from the direction associated with indicator elements 820a-820e.

[0075] Fig.13An indicator is shown that is displayed to indicate that images captured by multiple users are captured from multiple directions. In this example, the user captures images from the direction associated with indicator elements 820a-820j, as indicated by the permanent color of those indicator elements. In addition, the remaining indicator elements 820k-820x remain in a normal state, indicating that the user should continue to scan the object from additional directions to obtain images of the object required for creating a 3D model (or otherwise determined to be appropriate). The user can use this visual feedback from indicator elements 820a-820x to continue scanning the object 105 to visualize the directions that have been captured, the directions that still need to be captured, and the current direction of the device. In some examples, the user can continue to scan the object until all indicator elements are changed to indicate that the object has been recorded with images from sufficient surrounding directions. In some examples, the user can continue to identify the image capture initiated by the user until all indicator elements are changed to indicate that the object has been recorded with images from sufficient surrounding directions.

[0076] Fig.14 is a flowchart representation of a method for facilitating identification of an object during creation of a 3D model for object detection based on a scan of the object according to some implementations. In some implementations, method 1400 is performed by a device (e.g., Figure 2 and Figure 5 The method 1400 may be performed on a mobile device, an HMD, a desktop computer, a laptop computer, or a server device. In some implementations, the method 1400 is performed by a processing logic component, including hardware, firmware, software, or a combination thereof. In some implementations, the method 1400 is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory).

[0077] At block 1410, method 1400 records images of the object from multiple directions. Such images may be recorded continuously, for example, as a video. The images include a depiction of the physical object and a physical setting for the physical object. For example, the images may include a depiction of the physical object and a table or floor on which the physical object is placed, a wall against which the physical object is resting, an adjacent physical object, or a nearby physical object.

[0078] At block 1420, the method 1400 generates a model (eg, a feature point cloud) based on the image. The model may include a first portion corresponding to the physical object and a second portion corresponding to the physical scenery (eg, a nearby wall, table, or other physical object).

[0079] At block 1430, method 1400 identifies a subset of images during the recording of the images. The subset of images may be identified based on user input during the recording of the images. For example, as a user moves a mobile device to capture a stream of images (e.g., a video), the user may also periodically press an image capture button to designate the images captured at that point in time as user-captured images. The user interface may include a user interface command (e.g., a capture button) that is initiated at certain moments when the user moves the captured images around the physical object. For example, 10 images out of 200 recorded images may be identified as user-captured images.

[0080] The user interface may instruct the user to provide such user input in certain circumstances. For example, the user interface may include an indicator that identifies the area being captured on the real-time view of the image (e.g., an oval or rectangular figure identifying the center portion of the display), and when the object is within the identified area, the user may be instructed to provide user input. This may help distinguish between images that include the object within the area and images that do not include the object within the area. For example, as the user moves around and captures many images of the physical object, some of these images may not be focused on the physical object, for example, the physical object may not be fully depicted within the specified area of ​​the image. The user input may identify images in the image that have a depiction of the object within the specified area, so that these images can be used to identify the location of the physical object in three-dimensional space.

[0081] At box 1440, method 1400 identifies a spatial volume corresponding to the physical object based on the identified subset of images. In some specific implementations, this involves identifying the intersection of the spatial volumes identified based on the subset of images. For example, a spatial volume can be identified based on each image in the subset of images, for example, identifying a spatial volume of the physical scenery corresponding to a specific area (e.g., within a central rectangle) of each image. The intersection of these spatial volumes may be more likely to include physical objects and exclude other physical aspects of the physical scenery than other images captured during recording. The spatial volume may be based on images from multiple directions and may therefore include boundaries on many or all sides of the physical object that limit the spatial volume to a volume that is relatively closely adjacent to the space occupied by the physical object. In some specific implementations, when a new image is added to the subset of images, the spatial volume corresponding to the physical object is updated based on the added image.

[0082] At block 1450, based on identifying the spatial volume, method 1400 identifies a first portion of the model corresponding to the physical object. Method 1400 may identify (or clip) a portion of the model corresponding to the object based on the image, for example, using a spatial volume identified based on a subset of images captured by the user.

[0083] In some implementations, each of the subsets of images is associated with a spatial volume (e.g., a frustum / cone associated with the images captured during that time). In some implementations, the user captures these user-captured images in which the physical object is displayed entirely within the region boundaries or other indicators on the display (e.g., within a circular graphical indicator). The method 1400 may identify the spatial volume (e.g., frustum / cone) associated with each capture, and distinguish the physical object from other items based on the understanding that the physical object is within those spatial volumes. As the user captures the user-captured images from different directions (e.g., selecting a capture button), multiple volumes are identified. The location of the physical object may be determined based on the intersection of those volumes (e.g., within the intersection of multiple frustums / cones). In addition, the model may be generated on an ongoing basis, and each user-captured image may be used to cut out portions of the model that are outside the spatial volume associated with the capture. In the process of scanning from multiple directions, it may be appropriate to only include physical objects and exclude other items that may otherwise have been included in the model.

[0084] In some implementations, during scanning, the device or method continuously detects feature points (even when the user does not click the capture button). When the user clicks the capture button, the device or method evaluates the camera's view to cut out a volume of space (e.g., a cone) that contains the physical object. This technique is based on the assumption that the physical object is in view when the user presses the capture button. The system or method can obtain the volume defined by each press of the capture button (around the indicator and from top to bottom) and determine based on the intersection of those volumes to distinguish the physical object from other aspects of the physical scenery.

[0085] Fig.15 1 is a block diagram illustrating clipping a model of an object based on user-captured images of the object during scanning. In this example, cones or wedges 1510a-1510b associated with two user-captured images are used to determine the inclusion of object 105 in the model while excluding items 1500a-1500f. The intersection of the volumes can be used as a bounding volume for the scanned object and used by the model generation unit. The user does not need to locate the 3D bounding box. Instead, the bounding region can be automatically created based on the user's selection of the user-captured images during the scanning process of the object.

[0086] The apparatus or method may also use plane detection to detect any type of plane (eg, horizontal, vertical, angled, etc.) to define the boundaries of the scanned object. Feature points on the plane and outside the bounding volume may not be used to represent the object.

[0087] The user interface may be configured to provide instructions, prompts, and other guidance to help facilitate the capture of an image of an object and good anchoring results. The user interface may provide prompts to guide the user to avoid scanning shiny, moving, or tiny objects. The user interface may guide the user to place the object on a flat surface with uneven texture. The user interface may guide the user to take about a threshold number (e.g., about 5) of photos from different angles.

[0088] like Fig.16 As shown, the user interface may provide an indicator 1610 that shows the user what will be captured in the scan based on the camera's current image content (e.g., overlaid on the current real-time image). The user interface may instruct the user to frame the object using indicator 1610. The user interface may instruct the user to capture the front of the object.

[0089] The user interface may include an indicator indicating a particular side (e.g., the front of the object), such as indicator 1620. As the user moves around the object capturing images from different positions, an object side indicator such as indicator 1620 may be displayed to help orient the user relative to the position of the different sides of the object and / or which sides the user has captured images of.

[0090] The user interface may include a capture button 1630 or other user interface element that receives input to record an image of the currently displayed object. The device may continuously display video capture from the device's camera, and may designate certain frames of the video as captured images based on user selection of the capture button 1630.

[0091] After the user has captured an image of an object from a particular viewpoint, the user interface may change to indicate that an image of the object has been captured. Fig.17 As shown, the capture button 1630 is changed so that a portion 1730 of the perimeter 1740 of the capture button 1630 includes an indication (e.g., a line, a fill color, etc.) indicating that the image has been captured and the text message "Done" is displayed. In this example, the portion 1730 corresponds to a percentage (e.g., 25%) of the perimeter 1740 of the capture button 1630 to provide a visual indication corresponding to the percentage (e.g., 25%) of the desired image that has been captured.

[0092] The user interface may provide an indication of the desired movement (e.g., an arrow indicator or text) indicating the direction in which the user should move the image capture device to capture additional images of the object, such as Fig.18 shown.

[0093] The user interface may provide guidance or instructions based on detecting the motion or motion change of the device. For example, based on the user stopping moving for at least a time threshold (e.g., 5-10 seconds), the user interface may provide the user with a message with additional instructions or helpful content.

[0094] In another example, once the device's orientation is within the range of the desired position on a particular side (e.g., within 60-120 degrees on the right side), the user interface may prompt the user to capture an image of the object, for example, with a message "Capture right side." If the device's orientation passes through the range, the user interface may stop displaying the message and prompt the user to return.

[0095] After a predetermined number (e.g., three) of photos (e.g., front, right, and back) have been taken, the user may be provided with an option to skip obtaining an image from the other side (e.g., left), e.g., to accommodate situations where the user does not have access to or provide a portion or view of the object. The user interface may provide the user with a message to skip the step, e.g., a message indicating: "Skip step—If you do not have access to your object, then skip this step. Capture everything you can for best results."

[0096] The user interface may guide the user to position and orient the device to capture a 2D perspective view of the object through instructions and / or visual indicators. For example, such a 2D perspective view may provide images of the front, back, left, and right sides of the object. The user interface may determine that sufficient 2D perspective views have been obtained, and based on this criterion and / or other criteria, determine that one or more orthogonal views (e.g., top-down) are also desired or required. The user interface may guide the user to obtain one or more orthogonal views of the object through instructions and / or visual indicators, such as through arrows and / or text messages. The user interface may fade in and out a portion of the visual display (e.g., 50%) when in orthogonal view capture mode. The user interface may provide confirmation after an orthogonal view capture is performed by undoing the change to that portion of the visual display and / or provide an indication to the user that all necessary images of the object for scanning have been captured (e.g., a message "Done!").

[0097] After confirmation is complete, a user interface may be used to instruct the user to test the scan by attempting to detect an object based on the scan. If the object cannot be found during such a test, a message may be provided, such as a message indicating "we could not find this object" and an option to retry the test or rescan. The user interface may continue to attempt to detect the object as the user moves around, and may provide a message "trying to detect your object from different angles." The user interface may enable the user to review the capture and decide whether to discard or start over.

[0098] Numerous specific details are set forth herein to provide a comprehensive understanding of the claimed subject matter. However, those skilled in the art will appreciate that the claimed subject matter may be practiced without these specific details. In other instances, methods, devices, or systems known to those of ordinary skill are not described in detail so as not to obscure the claimed subject matter.

[0099] Unless otherwise specifically noted, it should be understood that throughout the specification, discussions utilizing terms such as "process," "compute," "calculate," "determine," and "identify" refer to the actions or processes of a computing device, such as one or more computers or similar electronic computing devices, that manipulate or transform data represented as physical electronic or magnetic quantities within a memory, register, or other information storage device, transmission device, or display device of a computing platform.

[0100] The one or more systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device may include any suitable arrangement of components that provide results conditioned on one or more inputs. Suitable computing devices include multi-purpose microprocessor-based computer systems that access stored software that programs or configures the computing system from a general-purpose computing device to a dedicated computing device that implements one or more specific implementations of the subject matter of the present invention. Any suitable programming, scripting, or other type of language or combination of languages ​​may be used to implement the teachings contained herein in software for programming or configuring a computing device.

[0101] The specific implementation of the method disclosed herein can be performed in the operation of such a computing device. The order of the boxes presented in the above examples can be changed, for example, the boxes can be reordered, combined or divided into sub-boxes. Some boxes or processes can be executed in parallel.

[0102] The use of "suitable for" or "configured to" herein is meant to be open and inclusive language that does not exclude devices that are suitable for or configured to perform additional tasks or steps. In addition, the use of "based on" is meant to be open and inclusive, as a process, step, calculation, or other action "based on" one or more of the stated conditions or values ​​may in practice be based on additional conditions or values ​​beyond those stated. The headings, lists, and numbers included herein are for ease of explanation only and are not intended to be limiting.

[0103] It will also be understood that, although the terms "first", "second", etc. may be used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first node may be referred to as a second node, and similarly, a second node may be referred to as a first node, which changes the meaning of the description, as long as all occurrences of the "first node" are consistently renamed and all occurrences of the "second node" are consistently renamed. Both the first node and the second node are nodes, but they are not the same node.

[0104] The terms used herein are only for describing specific implementations and are not intended to limit the claims. As used in the description of this specific implementation and the appended claims, the singular forms of "a" and "the" are intended to also cover the plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and covers any and all possible combinations of one or more items in the associated listed items. It will also be understood that the term "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts or their groupings.

[0105] As used herein, the term “if” may be interpreted to mean “when the antecedent is true” or “when the antecedent is true” or “in response to determining” or “upon determining” or “in response to detecting” that the antecedent is true, depending on the context. Similarly, the phrase “if it is determined that [the antecedent is true]” or “if [the antecedent is true]” or “when [the antecedent is true]” is interpreted to mean “upon determining that the antecedent is true” or “in response to determining” or “upon determining” that the antecedent is true or “when detecting that the antecedent is true” or “in response to detecting” that the antecedent is true, depending on the context.

[0106] The foregoing description and summary should be understood to be illustrative and exemplary in every aspect, rather than restrictive, and the scope of the invention disclosed herein is determined not only by the detailed description of the exemplary specific implementations, but by the full breadth allowed by the patent law. It should be understood that the specific implementations shown and described herein are only illustrative of the principles of the invention, and that various modifications can be implemented by those skilled in the art without departing from the scope and essence of the invention.

Claims

1. A non-transitory computer-readable storage medium storing program instructions executable by a computer on a device to perform operations comprising: providing an indicator in the user interface, the indicator comprising a plurality of elements, each element indicating a different direction of a plurality of possible directions from which an image of the physical object can be captured; modifying the indicator based on a detected orientation of the device relative to the physical object, wherein different ones of the plurality of elements are changed to indicate different current orientations of the device relative to the physical object; capturing a first set of images of the physical object from a subset of the plurality of possible directions; modifying a visual characteristic of one or more of the elements of the indicator corresponding to the subset to graphically distinguish directions for which the captured image has been captured from directions for which additional images need to be captured; capturing a second set of images of the physical object from the directions from which additional images need to be captured; as well as A model of the physical object is generated using the first set of images and the second set of images, the model including features for detecting the physical object in a sensor image.

2. The non-transitory computer-readable storage medium of claim 1, wherein the operations further comprise: The orientation of the device relative to the physical object is detected based on detecting a change in the orientation of the device in a coordinate system of a physical scene in which the physical object is located.

3. The non-transitory computer-readable storage medium of claim 1 , wherein the operations further comprise detecting the direction of the device relative to the object in a physical scene based on motion data from a motion sensor on the device, the motion data indicating a change in a position or orientation of the device. 4 . The non-transitory computer-readable storage medium of claim 1 , wherein the indicator comprises a plurality of elements displayed around an image of the physical object captured by a sensor of the device. 5 . The non-transitory computer-readable storage medium of claim 4 , wherein the plurality of elements are displayed in an arc-shaped arrangement around the image of the physical object. 6 . The non-transitory computer-readable storage medium of claim 4 , wherein modifying the indicator comprises changing a visual characteristic of one or more of the displayed elements to indicate the orientation of the device relative to the physical object.

7. The non-transitory computer-readable storage medium of claim 6, wherein a first element of the plurality of elements represents a first device orientation, and wherein modifying the indicator comprises: When the orientation of the device matches the first device orientation, changing a visual characteristic of the first element from a first state to a second, different state; as well as When the orientation of the device does not match the first device orientation, returning the visual characteristic of the first element to the first state.

8. The non-transitory computer-readable storage medium of claim 6, wherein a first element of the plurality of elements represents a first device orientation, and wherein modifying the indicator comprises: After capturing an image of the object when the direction of the device matches the first device orientation, changing a visual characteristic of the first element from a first state to a different second state when the direction of the device matches the first device orientation.

9. The non-transitory computer-readable storage medium of claim 8, wherein modifying the indicator comprises: When the orientation of the device does not match the first device, the display of the visual characteristic of the first element is maintained in the second state.

10. The non-transitory computer-readable storage medium of claim 1, wherein modifying an indicator based on the detected direction comprises: expand the first element by the first amount; identifying one or more additional elements based on proximity to the first element; as well as The one or more additional elements are expanded by a second amount, wherein the second amount is less than the first amount. 11 . The non-transitory computer-readable storage medium of claim 10 , wherein expanding the second amount of the one or more additional elements indicates that the current image will provide image capture data associated with multiple directions.

12. A system for generating a model of a physical object, the system comprising: a non-transitory computer-readable storage medium; camera; and one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium includes program instructions that, when executed on the one or more processors, cause the system to perform operations including: providing an indicator in the user interface, the indicator comprising a plurality of elements, each element indicating a different direction of a plurality of possible directions from which an image of the physical object can be captured; modifying the indicator based on a detected orientation of the device relative to the physical object, wherein different ones of the plurality of elements are changed to indicate different current orientations of the device relative to the physical object; capturing a first set of images of the physical object from a subset of the plurality of possible directions; modifying a visual characteristic of one or more of the elements of the indicator corresponding to the subset to graphically distinguish directions for which the captured image has been captured from directions for which additional images need to be captured; capturing a second set of images of the physical object from the directions from which additional images need to be captured; and A model of the physical object is generated using the first set of images and the second set of images, the model including features for detecting the physical object in a sensor image.

13. The system of claim 12, wherein the indicator comprises a plurality of elements displayed around an image of the physical object captured by a sensor of the device.

14. The system of claim 13, wherein the plurality of elements are arranged in an arc arrangement.

15. The system of claim 13, wherein modifying the indicator comprises changing a visual characteristic of one or more of the displayed elements to indicate the orientation of the device relative to the physical object.

16. The system of claim 13, wherein a first element of the plurality of elements represents a first device orientation, and wherein modifying the indicator comprises: When the orientation of the device matches the first device orientation, changing a visual characteristic of the first element from a first state to a second, different state; as well as When the orientation of the device does not match the first device orientation, returning the visual characteristic of the first element to the first state.

17. The system of claim 12, wherein modifying the indicator comprises: changing a visual characteristic of a first subset of the elements from a first state to a second state to indicate the current orientation of the device; as well as The visual characteristic of the second subset of the elements is returned from the second state to a second state to indicate that the device is no longer oriented in a previous orientation.

18. The system of claim 12, wherein the operations further comprise: receiving an input of a captured image; as well as When the input is received, the indicator is modified based on the orientation of the device relative to the physical object.

19. The system of claim 12, wherein generating the model comprises generating the model based on identifying a portion of the model corresponding to the physical object, the identifying being based on the image based input recorded as a subset of a captured image.

20. The system of claim 19, wherein the portion of the model corresponding to the physical object is identified based on identifying a spatial volume from the captured image.

21. A method for identifying a model, the method comprising: At a device that includes a processor: recording images of a physical object from a plurality of directions, wherein the images include depictions of the physical object and a physical setting for the physical object; generating a model based on the image, wherein the model includes a first portion corresponding to the physical object and a second portion corresponding to the physical set; identifying a subset of said images during said recording of said images; identifying a volume of space corresponding to the physical object based on the identified subset of the images; as well as Based on identifying the spatial volume, the first portion of the model corresponding to the physical object is identified.

22. The method of claim 21, wherein the subset of images is identified based on user input during the recording of the images.

23. A method for generating a model of a physical object, the method comprising: At a device that includes a processor: providing an indicator in the user interface, the indicator comprising a plurality of elements, each element indicating a different direction of a plurality of possible directions from which an image of the physical object can be captured; modifying the indicator based on a detected orientation of the device relative to the physical object, wherein at least one of the elements is changed to indicate a current orientation of the device relative to the physical object; capturing a first set of images of the physical object from a subset of the plurality of possible directions; modifying a visual characteristic of one or more of the elements of the indicator corresponding to the subset to graphically distinguish directions for which the captured image has been captured from directions for which additional images need to be captured; capturing a second set of images of the physical object from the directions from which additional images need to be captured; as well as A model of the physical object is generated using the first set of images and the second set of images, the model including features for detecting the physical object in a sensor image.

24. The method according to claim 23, further comprising: The orientation of the device relative to the physical object is detected based on detecting a change in the orientation of the device in a coordinate system of a physical scene in which the physical object is located.

25. The method of claim 23, further comprising detecting the direction of the device relative to the object in a physical scene based on motion data from a motion sensor on the device, the motion data indicating a change in position or orientation of the device.

26. The method of claim 23, wherein the indicator comprises a plurality of elements displayed around an image of the physical object captured by a sensor of the device.

27. The method of claim 26, wherein the plurality of elements are displayed in an arcuate arrangement around the image of the physical object.

28. The method of claim 26, wherein modifying the indicator comprises changing a visual characteristic of one or more of the displayed elements to indicate the orientation of the device relative to the physical object.

29. The method of claim 28, wherein a first element of the plurality of elements represents a first device orientation, and wherein modifying the indicator comprises: When the orientation of the device matches the first device orientation, changing a visual characteristic of the first element from a first state to a second, different state; as well as When the orientation of the device does not match the first device orientation, returning the visual characteristic of the first element to the first state.

30. The method of claim 28, wherein a first element of the plurality of elements represents a first device orientation, and wherein modifying the indicator comprises: After capturing an image of the object when the direction of the device matches the first device orientation, changing a visual characteristic of the first element from a first state to a different second state when the direction of the device matches the first device orientation.

31. The method of claim 30, wherein modifying the indicator comprises: When the orientation of the device does not match the first device, the display of the visual characteristic of the first element is maintained in the second state.

Images