Identifying an object for object recognition based on a user gaze determined by a head-mounted device

Abstract

To register objects for subsequent recognition by a human-centric head-mounted device, the head-mounted device receives input from the user to enter registration mode. The head-mounted device includes multiple imaging devices that capture images of local areas surrounding the device. The eye-tracking unit of the head-mounted device determines the user's gaze direction, and the device identifies objects within the local area the user's gaze is pointing to, visually distinguishing the identified objects for the user. In response to confirmation from the user that an object has been registered, the imaging devices capture one or more images of the identified object, which are used to train an instance classifier to recognize the object. When an object has been registered, the head-mounted device notifies the user.

Description

Technical Field

[0001] This disclosure relates generally to artificial reality systems, and more particularly to object recognition within local areas of artificial reality systems. Background Technology

[0002] Various devices, such as augmented reality (AR) headsets, implement one or more computer vision methods to detect or recognize objects included in images. For example, an AR headset includes an imaging device that captures images of a local area surrounding the AR headset and detects one or more objects within that local area from those images. Object recognition can be performed at the category level or at the instance level; at the category level, objects from a specific set of categories are detected; at the instance level, specific objects are detected based on training on a set of example images of a specific object. Devices such as AR headsets are trained to recognize specific objects using one or more object registration methods.

[0003] In traditional object registration, users capture different images of an object from different angles and against different backgrounds. These captured images are used to train a model to recognize or detect objects in subsequent images. Capturing multiple images of an object from different angles or multiple images of an object with other distinct features is time-consuming and carries a high risk of errors when the user captures the images. For example, during object registration, there are specific requirements for the object's image, namely, cropping around the object to segment it from its background. This requires the user to precisely position the imaging device relative to the object to be registered, thus creating resistance during the registration process. Furthermore, users often have to perform specific gestures to identify the object to be registered, such as specifically pointing at the object or providing a series of other inputs to identify a particular object, further increasing the complexity of the user performing or initiating object registration. Summary of the Invention

[0004] Instance-level object detection allows the detection of specific objects from an image of a region. Head-mounted devices (e.g., augmented reality (AR) headsets) can use instance-level detection to detect specific objects in images of a local area surrounding the headset captured by one or more imaging devices on the headset. Detecting specific objects in a local area allows for the tracking of different objects within that area, while also allowing the headset to provide information about those different objects to the user. To detect specific objects, the headset utilizes the user's gaze detection to train an instance classifier for that specific object.

[0005] In various embodiments, the head-mounted device determines the user's gaze direction within a local area from an eye-tracking unit that captures data describing the user's single or binocular eyes. For example, the eye-tracking unit determines the user's gaze direction based on images of the user's eyes captured when illuminated with a structured light pattern. The head-mounted device also captures images of the local area from one or more imaging devices and determines the location of the user's gaze direction within a 3D model of the local area. Based on the location of the user's gaze direction within the 3D model of the local area, the head-mounted device identifies objects and subsequently captures images of the identified objects via one or more imaging devices. An instance classifier is trained to detect identified objects based on the captured images of the identified objects.

[0006] According to a first aspect, a method is provided, the method comprising: receiving input from a user at a head-mounted device worn by a user to register one or more objects in a local region surrounding the head-mounted device for subsequent detection; determining, by the head-mounted device, the user's gaze direction in the local region based on captured information describing the user's monocular or binocular eyes; identifying objects in the local region based on the determined gaze direction of the user; capturing one or more images of the identified objects from one or more imaging devices included on the head-mounted device and configured to capture images of the local region; and storing the captured one or more images in association with tags corresponding to the identified objects.

[0007] In some embodiments, identifying objects in a local region based on the determined gaze direction of the user includes: identifying one or more regions of one or more images of the local region captured by one or more imaging devices as candidate objects; identifying regions in a three-dimensional model of the local region corresponding to each candidate object; and identifying objects in the three-dimensional model of the local region corresponding to candidate objects in the region pointed to by the user's gaze direction.

[0008] In some embodiments, identifying objects in a 3D model of a local region corresponding to candidate objects within a region pointed to by the user's gaze direction includes: determining, by the head-mounted device, a bounding box corresponding to each candidate object in the 3D model of the local region without input from the user; and identifying candidate objects included in the bounding box at a location including the user's gaze direction.

[0009] In some embodiments, identifying candidate objects included in a bounding box at a location including the user's gaze direction includes: identifying candidate objects included in a bounding box at a location including the user's gaze direction for at least a threshold amount of time.

[0010] In some embodiments, identifying one or more regions of one or more images of a local region captured by one or more imaging devices as candidate objects includes: applying a category classifier to one or more images of the local region, the category classifier identifying regions of the images that include at least one object.

[0011] In some embodiments, a tag corresponding to an identified object is received from the user after one or more images of the identified object are captured.

[0012] In some embodiments, the tag corresponding to the identified object is received from the user after the object is identified and before one or more images of the identified object are captured.

[0013] In some embodiments, capturing one or more images of an identified object from one or more imaging devices included in a head-mounted device and configured to capture images of a local area includes capturing a plurality of images of the identified object, each of the plurality of images corresponding to a different position of the identified object relative to one or more imaging devices.

[0014] In some embodiments, the method further includes: training an instance classifier to detect identified objects within a local region of an image based on one or more images of the captured identified objects.

[0015] In some embodiments, the method further includes displaying to a user an indication that the instance classifier has been trained via a display element of a head-mounted device.

[0016] According to another aspect, a head-mounted device is provided, comprising: a frame; one or more display elements coupled to the frame, each display element configured to generate image light for presentation to a user; one or more imaging devices coupled to the frame, the one or more imaging devices configured to capture images of a local area surrounding the frame; an eye-tracking unit configured to determine the user's gaze direction based on captured information describing the user's monocular or binocular eyes; and an object registration module including a processor and a non-transitory computer-readable storage medium having a plurality of instructions encoded thereon, the plurality of instructions, when executed by the processor, causing the processor to: receive input from the user to register one or more objects in a local area surrounding the head-mounted device for subsequent detection; identify objects in the local area based on the determined gaze direction of the user; capture one or more images of the identified objects from the one or more imaging devices; and store the captured one or more images associated with tags corresponding to the identified objects.

[0017] In some embodiments, identifying objects in a local region based on the determined gaze direction of the user includes: identifying one or more regions of one or more images of the local region captured by one or more imaging devices as candidate objects; identifying regions in a three-dimensional model of the local region corresponding to each candidate object; and identifying objects in the three-dimensional model of the local region corresponding to candidate objects in the region pointed to by the user's gaze direction.

[0018] In some embodiments, identifying objects in a 3D model of a local region corresponding to candidate objects within a region pointed to by the user's gaze direction includes: determining, by the head-mounted device, a bounding box corresponding to each candidate object in the 3D model of the local region without input from the user; and identifying candidate objects included in the bounding box at a location including the user's gaze direction.

[0019] In some embodiments, identifying candidate objects included in a bounding box at a location including the user's gaze direction includes: identifying candidate objects included in a bounding box at a location including the user's gaze direction for at least a threshold amount of time.

[0020] In some embodiments, identifying one or more regions of one or more images of a local region captured by one or more imaging devices as candidate objects includes: applying a category classifier to one or more images of the local region, the category classifier identifying regions of the images that include at least one object.

[0021] In some embodiments, a tag corresponding to an identified object is received from the user after one or more images of the identified object are captured.

[0022] In some embodiments, the tag corresponding to the identified object is received from the user after the object is identified and before one or more images of the identified object are captured.

[0023] In some embodiments, capturing one or more images of an identified object from one or more imaging devices included in a head-mounted device and configured to capture images of a local area includes capturing a plurality of images of the identified object, each of the plurality of images corresponding to a different position of the identified object relative to one or more imaging devices.

[0024] In some embodiments, a plurality of instructions encoded on a non-transitory computer-readable storage medium further enable the processor to: train an instance classifier to detect identified objects within a local region of an image based on one or more images of captured identified objects.

[0025] In some embodiments, a plurality of instructions encoded on a non-transitory computer-readable storage medium further enable the processor to: display to the user via a display element of a head-mounted device an indication that the instance classifier has been trained. Attached Figure Description

[0026] Figure 1A This is a perspective view of a head-mounted device implemented as an eyeglass device according to one or more embodiments.

[0027] Figure 1B This is a perspective view of a head-mounted device implemented as a head-mounted display according to one or more embodiments.

[0028] Figure 2 This is a block diagram of an eye-tracking unit included in a head-mounted device according to one or more embodiments.

[0029] Figure 3 This is a block diagram of an object registration module according to one or more embodiments.

[0030] Figure 4 This is a flowchart illustrating a method for registering objects with a head-mounted device for identification, according to one or more embodiments.

[0031] Figure 5 This is an example of a head-mounted device, according to one or more embodiments, registering objects from an image of a local area for identification.

[0032] Figure 6 It is a system including a head-mounted device according to one or more embodiments.

[0033] These accompanying drawings depict various embodiments for illustrative purposes only. Those skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods shown herein can be employed without departing from the principles described herein. Detailed Implementation

[0034] Various devices, such as augmented reality (AR) headsets, implement one or more computer vision methods to detect or recognize objects included in images. Object recognition can include class-level recognition of objects from a specific set of categories or instance-level detection of a specific object, or some combination thereof. Object instance-level detection identifies a specific object from an image of a region. Headsets, such as augmented reality (AR) headsets, can utilize object instance-level detection to detect specific objects from captured images of a local area surrounding the headset. Detecting specific objects in a local area allows the headset to track different objects in that local area and provide the user with information about the different objects in the local area.

[0035] To identify specific objects through instance-level detection, an instance classifier is trained through a training process that registers specific objects. Utilizing the user's gaze direction allows the head-mounted device to simplify the registration of objects in a local region surrounding the device for use in training the instance classifier. The head-mounted device determines the user's gaze direction within the local region from an eye-tracking unit that captures data describing the user's monocular or binocular vision. For example, the eye-tracking unit determines the user's gaze direction based on images of the user's eyes captured while the user's eyes are illuminated with a structured light pattern. Based on the location of the user's gaze direction within the local region, the head-mounted device identifies objects corresponding to the location of the user's gaze direction in a 3D model of the local region. For example, the head-mounted device identifies objects corresponding to locations in the 3D model of the local region where the user's gaze direction has remained for at least a threshold amount of time. The head-mounted device uses one or more imaging devices that capture images of the local region to capture images of the identified objects. In various embodiments, different images of the identified objects correspond to different locations of the identified objects relative to one or more of a plurality of imaging devices. The captured images of the identified objects are stored and used to train the instance classifier, which then detects the identified objects from the images of the local region.

[0036] Using a trained instance classifier, a head-mounted device can identify specific objects in subsequently captured image data. A head-mounted device with one or more imaging devices captures subsequent image data of a local region. The head-mounted device can apply an instance classifier to the subsequent image data to identify registered objects. In one or more embodiments, the head-mounted device first performs category-level object detection on the subsequent image data to identify objects in the subsequent image data. The head-mounted device can then apply the instance classifier to the identified objects to determine whether one of a plurality of identified objects in the local region is a registered object. After identifying a registered object, the head-mounted device can perform one or more actions on the registered object (e.g., providing one or more notifications associated with the identification of the registered object).

[0037] Embodiments of the present invention may include, or may combine with, an artificial reality system. An artificial reality is a form of reality that has been adjusted in some way before being presented to a user. This artificial reality may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content or generated content combined with captured (e.g., real-world) content. Artificial reality content may include video, audio, haptic feedback, or some combination thereof, any of which may be presented in a single-channel or multi-channel manner (e.g., stereoscopic video that produces a three-dimensional effect for the viewer). Furthermore, in some embodiments, the artificial reality may also be associated with applications, products, accessories, services, or some combination thereof for creating content in the artificial reality and / or otherwise using it in the artificial reality. Artificial reality systems that deliver artificial reality content can be implemented on a variety of platforms, including wearable devices (e.g., head-mounted devices) connected to a host computer system, standalone wearable devices (e.g., head-mounted devices), mobile devices or computing systems, or any other hardware platform capable of delivering artificial reality content to one or more viewers.

[0038] Figure 1A This is a perspective view of a head-mounted device 100 implemented as an eyewear device according to one or more embodiments. In some embodiments, the eyewear device is a near-eye display (NED). Typically, the head-mounted device 100 can be worn on a user's face to present content (e.g., media content) using a display component and / or an audio system. However, the head-mounted device 100 can also be used to present media content to a user in different ways. Examples of media content presented by the head-mounted device 100 include one or more images, videos, audio, or some combination thereof. The head-mounted device 100 includes a frame and may include a display component, a depth camera assembly (DCA), an audio system, and other components such as a position sensor 190, the display component including one or more display elements 120. Although Figure 1A The illustration shows example locations of components of the head-mounted device 100 on the head-mounted device 100, but these components may be located at other locations on the head-mounted device 100; on a peripheral device paired with the head-mounted device 100; or some combination thereof. Similarly, the head-mounted device 100 may have more than Figure 1A The components shown may have more or fewer components.

[0039] Frame 110 holds other components of the head-mounted device 100. Frame 110 includes a front component that holds one or more display elements 120, and an end component (e.g., a temple) that attaches to the user's head. The front component of frame 110 extends across the top of the user's nose. The length of the end component may be adjustable (e.g., an adjustable temple length) to fit different users. The end component may also include a curved portion behind the user's ears (e.g., a temple tip, an ear piece).

[0040] One or more display elements 120 provide light to a user wearing a head-mounted device 100. As shown, the head-mounted device includes a display element 120 for each of the user's eyes. In some embodiments, the display elements 120 generate image light that is provided to the eyebox of the head-mounted device 100. The eyebox is the location of the user's eyes in the space occupied by the eyes when wearing the head-mounted device 100. For example, the display element 120 may be a waveguide display. A waveguide display includes a light source (e.g., a two-dimensional source, one or more line sources, one or more point sources, etc.) and one or more waveguides. Light from the light source is coupled into one or more waveguides, which output light in a manner that creates a pupil replication in the eyebox of the head-mounted device 100. The coupling of light in and / or the coupling of light out of one or more waveguides may be accomplished using one or more diffraction gratings. In some embodiments, the waveguide display includes a scanning element (e.g., a waveguide, a mirror, etc.) that scans the light as it is coupled into the one or more waveguides. Note that in some embodiments, one or both of the display elements 120 are opaque and do not transmit light from a local area surrounding the head-mounted device 100. This local area is the area surrounding the head-mounted device 100. For example, this local area could be a room where a user wearing the head-mounted device 100 is inside, or the user wearing the head-mounted device 100 might be outdoors and the local area is an outdoor area. In this context, the head-mounted device 100 generates VR content. Alternatively, in some embodiments, one or both of the display elements 120 are at least partially transparent, such that light from the local area can be combined with light from the one or more display elements to generate AR and / or MR content.

[0041] In some embodiments, the display element 120 does not generate image light, but rather acts as a lens that transmits light from a localized area to the eye-friendly area. For example, one or both of the display elements 120 may be an uncorrected (over-the-counter) lens or a prescription lens (e.g., a single-vision lens, bifocal lens, trifocal lens, or progressive lens) that helps correct a user's visual impairment. In some embodiments, the display element 120 may be polarized and / or tinted to protect the user's eyes from the sun's rays.

[0042] In some embodiments, display element 120 may include another optical block (not shown). The optical block may include one or more optical elements (e.g., lenses, Fresnel lenses, etc.) that guide light from display element 120 to the eye-friendly area. The optical block may, for example, correct aberrations in some or all of the image content, magnify some or all of the image, or some combination thereof.

[0043] DCA determines depth information for a portion of a local area surrounding the head-mounted device 100. DCA includes one or more imaging devices 130 and a DCA controller. Figure 1A (Not shown in the figure), and may also include an illuminator 140. In some embodiments, the illuminator 140 uses light to illuminate a portion of a local area. This light may be, for example, structured light in the infrared (IR) region (e.g., dot-patterned structured light, strip structured light, etc.), an IR flash for time-of-flight (ToF), etc. In some embodiments, one or more imaging devices 130 capture an image of a portion of the local area including light from the illuminator 140. As shown in the figure, Figure 1A A single illuminator 140 and two imaging devices 130 are shown. In an alternative embodiment, there is no illuminator 140 and at least two imaging devices 130.

[0044] The DCA controller uses the captured image and one or more depth determination techniques to calculate the depth information of that portion of a local area. The depth determination techniques may be, for example, direct time-of-flight (ToF) depth sensing, indirect ToF depth sensing, structured light, passive stereo analysis, active stereo analysis (using textures added to the scene by light from illuminator 140), some other techniques for determining the depth of the scene, or some combination thereof.

[0045] The DCA may include an eye-tracking unit that determines eye-tracking information. The eye-tracking information may include information about the position and orientation of a single or binocular eye (within its respective eye-fitting zone). The eye-tracking unit may include one or more cameras. The eye-tracking unit estimates the angular orientation of a single or binocular eye based on image captures of the single or binocular eye by the one or more cameras. In some embodiments, the eye-tracking unit may also include one or more illuminators that illuminate the single or binocular eye with an illumination pattern (e.g., structured light, flash, etc.). The eye-tracking unit can use the illumination pattern in the captured images to determine the eye-tracking information. The head-mounted device 100 may prompt a user to opt in to allow operation of the eye-tracking unit. For example, by opting in, the head-mounted device 100 may detect and store any images of the user or the user's eye-tracking information.

[0046] Based on information about the position and orientation of the user's single or binocular eyes, the eye-tracking unit determines the direction of the user's gaze. For example, the eye-tracking unit determines a vector or ray representing the fixation of the user's gaze relative to the user's head position. In various embodiments, the eye-tracking unit determines the user's gaze in each eye based on the position and orientation of each eye in the user's eyes. In various embodiments, the eye-tracking unit may employ various models or combinations of models to determine the direction of the user's gaze based on information about the position and orientation of the user's single or binocular eyes. As described below... Figures 3 to 5 As further described, the object registration module identifies objects in local areas based on the direction of the user's gaze and trains an instance classifier to detect the identified objects.

[0047] Figure 2 A block diagram of one embodiment of an eye-tracking unit 200 included in a head-mounted device 100 is shown. For example, the eye-tracking unit 200 is included in a depth camera assembly (DCA), as described above. Figure 1A Further description. However, in other embodiments, the eye-tracking unit 200 is a separate component from the DCA. Figure 2 In one example, the eye-tracking unit 200 includes an illumination source 205, one or more imaging devices 210, and a controller 215. However, in other embodiments, it is combined with... Figure 2 Compared to the components described, the eye-tracking unit 200 includes different or additional components. Furthermore, in some embodiments, [the components are...]. Figure 2 The functions provided by the multiple components shown can be combined into a single component.

[0048] When a user wears the head-mounted device 100, an illumination source 205 emits light toward one or both of the user's eyes. In some embodiments, the eye-tracking unit 200 includes an illumination source 205 for each of the user's eyes, so that different illumination sources illuminate each of the user's eyes. Alternatively, a single illumination source 205 emits light toward both of the user's eyes. At least a portion of the light from the illumination source 205 illuminates the user's eyes. The light emitted by the illumination source 205 may be structured light with an infrared (IR) wavelength (e.g., dot-patterned structured light, strip structured light, etc.), an IR flash for time-of-flight, or have other characteristics. In various embodiments, the light emitted by the illumination source 205 has a wavelength invisible to the user wearing the head-mounted device, such as an infrared (IR) wavelength.

[0049] One or more imaging devices 210 capture images of a user's single or binocular eyes. For example, a single imaging device 210 captures images of both of the user's eyes. As another example, the eye-tracking unit 200 includes two imaging devices 210, each capturing images of a different eye of the user. Images of the user's eyes are captured by the imaging devices 210 when each of the user's eyes is illuminated by light from the illumination source 205. The captured images include light reflected from the illumination source 205 by the user's eyes.

[0050] When a user wears head-mounted device 100, controller 215 determines the position and orientation of the user's monocular or binocular eyes from images captured by one or more imaging devices 210. Based on the position and orientation of the user's monocular or binocular eyes, controller 215 determines the user's gaze direction towards a local area. The gaze direction indicates the location of the local area the user is viewing. For example, the controller determines the distance between the center of the pupil of each eye and the reflection from the cornea of ​​that eye, based on reflected light from illumination source 205 included in the image of the user's eyes captured by imaging device 210. Based on the distance determined for each of the user's eyes, controller 215 determines the angle of each of the user's eyes. Based on the angle of the user's eyes, controller 215 determines the user's gaze direction. For example, controller 215 determines the user's gaze direction as the point where light rays from the centers of the user's eyes intersect in the local area, and identifies the location in the local area that the user is gazing towards as a point in the local area. In other embodiments, the controller 215 determines the user's gaze direction by one or more other methods based on images of the user's monocular or binocular eyes captured by one or more imaging devices 210 (including reflections of light emitted by the illumination source 205).

[0051] In various embodiments, controller 215 determines the user's gaze direction and the point of fixation. In such embodiments, controller 215 determines eye vergence, depth of fixation, and gaze direction. Depth of fixation indicates the depth of the user's gaze relative to the position pointed to by the head-mounted device 100 worn by the user.

[0052] Return to reference Figure 1A The audio system provides audio content. The audio system includes a transducer array, a sensor array, and an audio controller 150. However, in other embodiments, the audio system may include different components and / or additional components. Similarly, in some cases, the functions described with reference to the components of the audio system may be distributed among the multiple components in a manner different from that described herein. For example, some or all of the functions of the controller may be performed by a remote server.

[0053] A transducer array presents sound to the user. The transducer array includes multiple transducers. The transducers may be speaker 160 or tissue transducers 170 (e.g., bone conduction transducers or cartilage conduction transducers). Although speaker 160 is shown as located outside frame 110, speaker 160 may also be enclosed within frame 110. In some embodiments, head-mounted device 100 includes a speaker array (instead of separate speakers for each ear) comprising multiple speakers integrated into frame 110 to improve the directionality of the presented audio content. Tissue transducers 170 are coupled to the user's head and directly vibrate the user's tissue (e.g., bone or cartilage) to produce sound. The number and / or location of the transducers may vary depending on the specific requirements of the device. Figure 1A The quantities and / or positions shown are different.

[0054] A sensor array detects sound within a localized area of ​​a head-mounted device 100. The sensor array includes multiple acoustic sensors 180. Each acoustic sensor 180 captures sound emitted from one or more sound sources within the localized area (e.g., a room). Each acoustic sensor is configured to detect sound and convert the detected sound into an electronic format (analog or digital). The acoustic sensor 180 may be a sound wave sensor, a microphone, a sound transducer, or a similar sensor suitable for detecting sound.

[0055] In some embodiments, one or more acoustic sensors 180 may be placed in the ear canal of each ear (e.g., acting as a binocular microphone). In some embodiments, these acoustic sensors 180 may be placed on the outer surface of the head-mounted device 100, on the inner surface of the head-mounted device 100, separate from the head-mounted device 100 (e.g., as part of some other device), or some combination of the above locations. The number and / or location of the acoustic sensors 180 may vary with... Figure 1A The number and / or locations shown may vary. For example, the number of acoustic detection locations can be increased to increase the amount of audio information collected and improve the sensitivity and / or accuracy of that information. The acoustic detection locations can be oriented such that the microphone can detect sound in a wide range of directions around the user wearing the head-mounted device 100.

[0056] Audio controller 150 processes information from a sensor array describing the sound detected by the sensor array. Audio controller 150 may include a processor and a computer-readable storage medium. Audio controller 150 may be configured to generate direction of arrival (DOA) estimates, generate acoustic transfer functions (e.g., array transfer function and / or head-related transfer function), track the location of sound sources, form beams in the direction of sound sources, classify sound sources, generate sound filters for loudspeaker 160, or some combination thereof.

[0057] Position sensor 190 generates one or more measurement signals in response to motion of head-mounted device 100. Position sensor 190 may be located on a portion of the frame 110 of head-mounted viewer 100. Position sensor 190 may include an inertial measurement unit (IMU). Examples of position sensor 190 include one or more accelerometers, one or more gyroscopes, one or more magnetometers, other suitable types of sensors for detecting motion, a class of sensors for error correction of the IMU, or some combination thereof. Position sensor 190 may be located external to the IMU, internal to the IMU, or some combination thereof.

[0058] In some embodiments, the head-mounted device 100 can provide simultaneous localization and mapping (SLAM) for updating the position of the head-mounted device 100 and the model of a local region. For example, the head-mounted device 100 may include a passive camera assembly (PCA) that generates color image data. The PCA may include one or more RGB (red, green, blue) cameras that capture images of some or all of a local region. In some embodiments, some or all of the imaging devices 130 of the DCA may also be used as the PCA. The images captured by the PCA and the depth information determined by the DCA can be used to determine parameters of the local region, generate a model of the local region, update the model of the local region, or some combination thereof. In addition, the position sensor 190 tracks the position (e.g., location and orientation) of the head-mounted device 100 within a room. The following is combined with Figure 6Additional details regarding the components of the head-mounted device 100 are discussed.

[0059] Figure 1B This is a perspective view of a head-mounted device 105 implemented as an HMD according to one or more embodiments. In embodiments describing AR and / or MR systems, a portion of the front of the HMD is at least partially transparent in the visible band (approximately 380 nm to 750 nm), and a portion of the HMD between the front of the HMD and the user's eyes is at least partially transparent (e.g., a partially transparent electronic display). The HMD includes a front rigid body 115 and a band 175. The head-mounted device 105 includes a plurality of components similar to those referenced above. Figure 1A The components described are the same, but modified for integration with HMD shape elements. For example, the HMD includes a display assembly, DCA, audio system, and position sensor 190. Figure 1B An illuminator 140, multiple speakers 160, multiple imaging devices 130, multiple acoustic sensors 180, and a position sensor 190 are shown. These speakers 160 can be located in various positions, such as being coupled to a band 175 (as shown), coupled to a front rigid body 115, or being configured to be inserted into a user's ear canal.

[0060] Figure 3 This is a block diagram of one embodiment of the object registration module 300. In various embodiments, the object registration module 300 is included in the head-mounted device 100. For example, the object registration module 300 is included in or coupled to the frame 110 of the head-mounted device 100. In other embodiments, the object registration module 300 is physically separated from the frame 110 of the head-mounted device 100 and communicatively coupled to one or more components of the frame 110. Figure 3 In the example, the object registration module 300 includes an object selector 305, a category classifier 310, an instance classifier 315, an object map 320, and a communication module 325. In other embodiments, it is combined with... Figure 3 Compared to the components described, the object registration module 300 includes more components, different components, or fewer components.

[0061] Furthermore, the object registration module 300 includes a processor and one or more non-transitory computer-readable storage media. The one or more non-transitory computer-readable storage media have a plurality of instructions encoded thereon, which, when executed by the processor, cause the processor to provide the following combination... Figure 3 Further description of the features.

[0062] Object selector 305 receives the gaze direction of a user from eye-tracking unit 200 of head-mounted device 100 and images of a local region surrounding head-mounted device 100 from one or more imaging devices 130. Based on the user's gaze direction, object selector 305 identifies the location of objects generated from detected objects within a 3D model of the local region. In various embodiments, object selector 305 generates a 3D model based on depth information obtained from a depth camera assembly (DCA) or other depth sensor and objects detected by category classifier 310 or instance classifier 315. Furthermore, each object in the 3D model has a corresponding bounding box specifying the boundaries of that object. The bounding boxes of the objects are aligned with one or more imaging devices 130 of head-mounted device 100 to provide the boundaries of the objects from the viewpoint of the imaging devices 130. For example, the 3D model includes bounding boxes corresponding to each candidate object detected by category classifier 310 from one or more images in the local region.

[0063] Based on candidate objects identified in the image of the local region by the category classifier 310, the object selector 305 identifies regions of the 3D model of the local region as objects that include both the candidate objects and the location pointed to by the user's gaze. In various embodiments, the object selector 305 identifies bounding boxes within the 3D model of the local region that intersect with light rays corresponding to the user's gaze direction, and identifies objects corresponding to the identified bounding boxes. In various embodiments, the object selector 305 receives identifiers of candidate objects within the local region from the category classifier 310 and regions of the image of the local region corresponding to the candidate objects, and maps the identifiers of the candidate objects to regions within the 3D model of the local region. Based on the location of the light rays within the 3D model of the local region corresponding to the location pointed to by the user's gaze, the object selector 305 identifies candidate objects corresponding to bounding boxes within the 3D model pointed to by the user's gaze. For example, the object selector 305 identifies candidate objects corresponding to bounding boxes included in the 3D model of the local region where light rays corresponding to the user's gaze point towards the region for at least a threshold amount of time. In some embodiments, if a ray in a 3D model of a local region intersects with multiple bounding boxes (each bounding box corresponding to a candidate object), the object selector 305 selects the candidate object corresponding to the bounding box closest to the user.

[0064] In some embodiments, eye-tracking unit 200 determines the user's gaze direction and the user's gaze point. In such embodiments, eye-tracking unit 200 determines eye convergence and gaze depth, as well as the gaze direction. Gazing depth identifies the depth of the user's gaze relative to the position pointed to by the head-mounted device 100 worn by the user. Object selector 305 receives the user's gaze point and selects objects based on the user's gaze point and direction. For example, object selector 305 selects candidate objects corresponding to a bounding box that corresponds to the region in the 3D model of the local area that is closest to the user's gaze point. Alternatively, object selector 305 selects candidate objects corresponding to a bounding box in the 3D model of the local area that intersects the light ray corresponding to the user's gaze direction and is closest to the head-mounted device.

[0065] Alternatively, object selector 305 determines a ray from eye-tracking unit 200 that corresponds to the user's gaze direction and projects that ray onto a two-dimensional position within an image of a local region captured by imaging device 130. Object selector 305 selects objects from the image of the local region detected by classifier 310 that intersect with the ray corresponding to the user's gaze direction (e.g., objects corresponding to bounding boxes in the image of the local region that intersect with the ray corresponding to the user's gaze direction). For example, object selector 305 identifies candidate objects corresponding to bounding boxes included in the image of the local region that intersect with the ray corresponding to the user's gaze for at least a threshold amount of time.

[0066] In various embodiments, object selector 305 identifies candidate objects based on the user's gaze direction in response to receiving input from the user. The received input indicates that the user is registering objects for subsequent detection by instance classifier 315. For example, object selector 305 receives input in response to acoustic sensors 180 of head-mounted device 100 capturing audio from the user (e.g., capturing specific words or phrases). As another example, object selector 305 receives input from a controller or other device coupled to the head-mounted device, with which the user interacts. In yet another example, object selector 305 receives input in response to imaging device 130 of the head-mounted device capturing specific poses or movements from the user. One or more display elements 120 of head-mounted device 100 may display one or more interface elements with which the user interacts, thus object selector 305 receives input in response to user interaction with one or more interface elements.

[0067] Additionally, the object selector 305 displays one or more cues to the user in response to recognizing an object based on the user's gaze direction. For example, the object selector 305 transmits the cues to one or more display elements 120 of the head-mounted device 100, which display the cues to the user. In some embodiments, the cues visually distinguish the object recognized based on the user's gaze direction from other objects in a local area. For example, the cues are icons or text displayed by the display elements 120 of the head-mounted device 100 near the object selected based on the user's gaze direction. As another example, the cues are the boundaries surrounding the recognized object displayed by the display elements 120. Alternatively or additionally, the object selector 305 transmits the cues to one or more speakers 160 of the head-mounted device 100, which play the cues to the user. The cues may be messages for repositioning the recognized object relative to one or more imaging devices 130, so different images of the recognized object reflect different positions of the object relative to one or more of the imaging devices 130. For example, the prompt indicates to the user the direction to move the imaging device 130 relative to the identified object in order to capture images from different relative positions of the imaging device 130 relative to the identified object. When an image of the identified object is captured from a specific orientation of the imaging device 130 relative to the identified object, the prompt may display an indication or other signal. Furthermore, the object selector 305 may send a message to the user via one or more display elements 120 or to one or more speakers 160 in response to storing one or more images of the identified object or in response to training an instance classifier 315 for the identified object.

[0068] In various embodiments, object selector 305 modifies the captured image of the identified object to remove portions of the image outside the region including the identified object (e.g., removing portions of the image outside the bounding box of the identified object), and stores the modified image in association with a label corresponding to the identified object provided by the user. For example, object selector 305 modifies the captured image of the identified object after the user has identified the object. As further described below, images associated with the objects identified by object selector 305 are then retrieved to train instance classifier 315.

[0069] Category classifier 310 is a trained model that, when applied to one or more images of a local region received from one or more imaging devices 130, identifies any objects in those images. In various embodiments, category classifier 310 is a trained region-based convolutional neural network (R-CNN). Based on features of different regions of the local region image, category classifier 310 identifies regions in the image that include objects of a category or type. Category classifier 310 does not identify specific objects from the image, but rather identifies regions in the image that include objects of one or more categories or types, for which it is trained. For example, category classifier 310 identifies regions in the image that include objects of the category "cup" as candidate objects, but does not distinguish between different objects of the category "cup" in the image. Therefore, category classifier 310 identifies regions within the image of a local region that may contain objects based on one or more object categories.

[0070] For each region within the image identified as a candidate object, the category classifier 310 also assigns the size of the candidate object to the region. In various embodiments, the category classifier 310 determines the bounding box of the candidate object such that the region of the image corresponding to the candidate object is enclosed within the bounding box. The category classifier 310 determines the size of the bounding box of each candidate object based on the features of the candidate object, so different candidate objects can be enclosed by bounding boxes of different sizes. Furthermore, in various embodiments, the category classifier 310 determines the size of the bounding box without user input, thereby simplifying the identification of the region of the image corresponding to the candidate object. In various embodiments, the category classifier 310 identifies the coordinates within the bounding box image of each candidate object detected in the image and associates a candidate object identifier with each bounding box to identify different candidate objects.

[0071] Instance classifier 315 is a trained model that detects specific objects in images from an imaging device. To detect a specific object, the object first registers with instance classifier 315 in response to being identified by object selector 305. Instance classifier 315 detects specific identified objects, rather than object categories, thus distinguishing different objects with a common category. In various embodiments, instance classifier 315 is a machine learning model including a set of weights. These weights are parameters used by the machine learning model to transform input data received by the model into output data. For instance classifiers, input data includes one or more images of objects, and output is a label applied to the object, where the label identifies the object to the user. Weights can be generated through a training process, thereby training the machine learning model based on a set of training examples and labels associated with the training examples. In various embodiments, the training process includes: applying the machine learning model to training examples, comparing the output of the machine learning model with the labels associated with the training examples, and updating the weights associated with the machine learning model through a backpropagation process. Weights can be stored on one or more computer-readable media constituting the instance classifier. The training examples are images of objects captured by one or more imaging devices 130, identified by the object selector 305. In some embodiments, the instance classifier 315 may construct a 3D model of the identified objects based on images of the identified objects, while in other embodiments, the instance classifier 315 detects the identified objects from images of local regions. Subsequently, the instance classifier 315 receives one or more images and detects objects in one or more images. Thus, the instance classifier 315 allows the detection of specific objects in images, while the category classifier 310 identifies the type or category of objects included in one or more images. In one or more embodiments, the instance classifier 315 refines its imagery using subsequently captured images of previously registered objects.

[0072] In various embodiments, object map 320 stores information in a 3D model of a local region to identify objects identified by object selector 305. For example, object map 320 stores a label corresponding to an identified object, along with the location of the identified object within the 3D model of the local region and the size of the identified object (e.g., the bounding box corresponding to the identified object). Object map 320 can also track the position of identified objects relative to the local region. For example, object map 320 updates the 3D model of the local region to identify the positions of one or more objects previously identified by object selector 305 in the 3D model of the local region, thereby allowing object map 320 to update the 3D model of the local region to identify the position of the identified objects relative to head-mounted device 100 within the 3D model of the local region.

[0073] Communication module 325 links object registration module 300 to one or more other components or other devices of head-mounted device 100, such as in the following combinations Figure 6 Further description includes the map building server 625 or console 615. In various embodiments, the communication module 325 couples the object registration module 300 to one or more components via a network. In various embodiments, the communication module 325 may use wired communication protocols, wireless communication protocols, or a combination of wireless and wired transmission protocols to exchange data with components of the head-mounted device 100 or with other components.

[0074] During the deployment of the object registration module 300, the instance classifier 315 can be used to identify registered objects in subsequently captured image data. For example, a head-mounted device (e.g., imaging device 130) with one or more imaging devices (e.g., imaging device 130) Figure 1A Head-mounted devices 100 or Figure 1B The head-mounted device 105 captures subsequent image data of a local region. The head-mounted device can then apply an instance classifier 315 to the subsequent image data to identify registered objects.

[0075] In one or more embodiments, the head-mounted device first applies a category classifier 310 to subsequent image data to identify objects in the subsequent image data. The category classifier 310 can also classify multiple identified objects into one or more of multiple object categories. The head-mounted device can then apply an instance classifier 315 to the identified objects to determine whether one of the multiple identified objects in a local area is a registered object. In some embodiments, the instance classifier 315 can be applied to all identified objects. In other embodiments, the instance classifier 315 can be applied to identified objects in a subset of categories. For example, the object registration module 300 can register many different mugs in a user's home. The object registration module 300 can train multiple instance classifiers 315, each trained to identify one of the multiple registered mugs. During deployment, the object registration module 300 can apply the category classifier 310, which (typically) identifies mugs in a local area. The object registration module 300 can then apply the instance classifier 315 to the multiple identified mugs to identify each individual registered mug.

[0076] After identifying a registered object, the head-mounted device can perform one or more actions on the registered object. One example action includes tracking the location of the identified object within a local area. When the registered object is identified in subsequent image data, the head-mounted device can track the registered object by recording its position relative to the local area. For example, if the user of the head-mounted device prompts the user to remind them of the location of a registered object, the head-mounted device can provide notification of the last known location of the registered object. In another example, the head-mounted device prompts the user to replace the registered object based on information about the registered object (e.g., medication, toothbrush, sponge, etc.) and the time when the registered object was first registered. In yet another example, the head-mounted device retrieves information about the registered object, such as other objects similar to the registered object or information about other objects similar to the registered object (e.g., price, location). Another example action includes providing one or more notifications when a registered object is identified in subsequent image data. In a first example, the head-mounted device can display a tag or marker near the registered object. In a second example, the head-mounted device can categorize the object; for example, the category of the identified object could be "plant." Once the head-mounted device identifies a specific registered plant, it can provide notifications to water the plant.

[0077] Figure 4 This is a flowchart of a method for registering an object with a head-mounted device 100 for identification, according to one or more embodiments. Figure 4 The process illustrated can be performed by a component of the object detection system (e.g., object registration module 300). In other embodiments, other entities can perform the process. Figure 4 This involves some or all of the multiple steps in the process. Implementations may include different or additional steps, or these steps may be performed in a different order.

[0078] As described above Figure 1A and Figure 1B Further described, the head-mounted device 100 receives 405 input from a user to register objects in a local area surrounding the head-mounted device 100. In some embodiments, the head-mounted device 100 receives 405 input by capturing a specific audio phrase or sound from the user via one or more acoustic sensors 180. As another example, the head-mounted device 100 receives 405 input when one or more imaging devices 130 configured to capture images of a local area surrounding the head-mounted device 100 capture a specific posture or movement of a part of the user's body. In other embodiments, the head-mounted device 100 receives 305 input from the user through other actions performed by the user (e.g., physically touching a part of the head-mounted device 100, etc.).

[0079] In response to receiving input 405 from the user, the head-mounted device 100 operates in object registration mode. In object registration mode, when the head-mounted device is operating, the head-mounted device 100 captures data (e.g., image data) for subsequent detection of objects identified by the user. As described above... Figure 1A and Figure 2 Further described, the head-mounted device 100 includes an eye-tracking unit 200 that determines the position and orientation of a user's single or binocular eyes when the user wears the head-mounted device 100. Based on the position and orientation of the user's single or binocular eyes, the eye-tracking unit 200 determines the user's gaze direction towards a local area. For example, for each of the user's eyes, the eye-tracking unit 200 captures the distance between the center of the pupil of that eye and the reflection from the cornea of ​​that eye, and determines the angle of each of the user's eyes based on the distance captured for each of the user's eyes. Based on the angle of the user's eyes, the eye-tracking unit 200 determines the user's gaze direction. For example, when the user's eyes have a defined angle, the user's gaze direction is the location where light rays from the centers of the user's eyes intersect in a local area. In various embodiments, the eye-tracking unit 200 determines the user's gaze direction and the point of fixation of the user's gaze. In such embodiments, the eye-tracking unit 200 determines eye convergence and depth of fixation, as well as the gaze direction. The gaze depth indicator represents the depth of the user's gaze relative to the position pointed to by the head-mounted device 100 worn by the user.

[0080] Additionally, the head-mounted device 100 includes one or more imaging devices 130, each configured to capture images of a local region surrounding the head-mounted device 100, as further described above in conjunction with FIG. 1. The images captured by the one or more imaging devices 130 may include one or more objects in the local region surrounding the head-mounted device 100. The head-mounted device 100 detects objects in the captured one or more images. For example, a category classifier 310 is applied to the captured images to detect candidate objects in one or more images, as described above in conjunction with FIG. 1. Figure 3 Further described. In some embodiments, the category classifier 310 is included in the head-mounted device 100, for example, as described above. Figure 3 Further description is provided in the object registration module 300. Alternatively, the head-mounted device 100 registers with a server (e.g., a map building server 625) or as described below. Figure 6 The console 615, described further, transmits images captured by one or more imaging devices 130, and the server or console 615 applies a category classifier 310 to the captured images. In the preceding example, the server or console 615 sends the objects detected by the category classifier 310 to the head-mounted device 100. As described above... Figure 3Further described, the regions of the image identified by the category classifier 310 are candidate objects because those regions include one or more objects that the category classifier 310 is trained to identify. In some embodiments, the category classifier 310 also outputs a category identifying the type of the candidate object for each candidate object. The application of the category classifier 310 identifies regions within the image that include objects in one or more categories, thereby enabling the differentiation of candidate objects in one or more images from the background of one or more images.

[0081] In various embodiments, the category classifier 310 determines a bounding box for each candidate object detected by the category classifier. The bounding box of a candidate object specifies the boundary of the region of the image that includes the candidate object; therefore, the region within the bounding box of the image includes the candidate object, while the region outside the bounding box is not a candidate object. The category classifier 310 determines the size of the bounding box of each candidate object based on features of the region of the image that includes the candidate object, so different candidate objects can be surrounded by bounding boxes of different sizes. Furthermore, in some embodiments, the category classifier 310 determines the size of the bounding box of the candidate object without receiving input or interaction from the user, thereby simplifying the identification of the region of the image that includes the object. In various embodiments, the category classifier 310 identifies the coordinates within the bounding box image of each candidate object detected in the image and associates a unique candidate object identifier with each bounding box. The candidate object identifier allows the head-mounted device 100 to subsequently identify different regions in the image that include the candidate object.

[0082] Based on the gaze direction determined for the user 410 and candidate objects identified in the local region, the head-mounted device 100 identifies objects 415 in the local region. In various embodiments, the object registration module 300 of the head-mounted device 100 identifies objects 415 as candidate objects in a region corresponding to the location pointed to by the user's gaze in the 3D model of the local region. For example, an identified object is an object included in a bounding box in the 3D model of the local region that intersects with a ray corresponding to the user's gaze direction. In various embodiments, the head-mounted device 100 considers the user's gaze depth and the user's gaze direction to identify objects 415. For example, the head-mounted device identifies objects 415 corresponding to bounding boxes in the 3D model of the local region that intersect with a ray corresponding to the user's gaze direction and are within a threshold distance of the user's gaze depth. In other embodiments, the head-mounted device 100 identifies objects 415 as candidate objects in a region of the image corresponding to the location pointed to by the user's gaze in the local region. For example, an identified object is an object included in a bounding box that also includes the location of the user's gaze direction. In the preceding example, head-mounted device 100 identifies 415 objects included in a bounding box intersecting with the user's gaze direction. In some embodiments, object registration module 300 identifies 415 candidate objects included in a bounding box corresponding to a location where the user's gaze is directed for at least a threshold amount of time. In other embodiments, object registration module 300 identifies 415 candidate objects closest to the user's gaze. This allows for the identification of objects in a local area based on the direction of the user's gaze within that local area, thereby simplifying object identification for the user.

[0083] In various embodiments, the head-mounted device 100 visually distinguishes an identified object from other candidate objects in a localized area and from the background of that localized area. For example, the head-mounted device 100 overlays a bounding box including the identified object onto the identified object, showing the boundary of the identified object to the user via display element 120, while display element 120 does not show bounding boxes surrounding other objects in the localized area. As another example, the head-mounted device 100 overlays a specific color or shadow onto the identified object via display element 120. In yet another example, the head-mounted device 100 displays icons, images, or text near the identified object via display element 120 to distinguish the identified object from other objects in the localized area. By visually distinguishing the identified object from the rest of the localized area, the head-mounted device 100 indicates to the user which object in the localized area has been identified 415.

[0084] In response to the identification of an identified object 415, the imaging device 130 of the head-mounted device 100 captures one or more images of the identified object 420. In some embodiments, the head-mounted device 100 prompts a user to change the position of the identified object relative to one or more of the plurality of imaging devices 130 to capture different images of the identified object. For example, the head-mounted device 100 displays instructions to the user via one or more display elements 120 to modify the position of the identified object from its current position or to modify the position of the imaging device 130 relative to the identified object, and the imaging device 130 captures one or more images of the identified object 420 at the modified position of the identified object relative to the imaging device 130. Modifying the position of the identified object relative to one or more imaging devices 130 allows different images to include different perspectives of the identified object, thereby providing additional information about the characteristics of the identified object. The object registration module 300 stores the captured images.

[0085] In various embodiments, the head-mounted device 100 receives a tag from a user for an identified object and stores the tag in association with each captured image of the identified object. The head-mounted device 100 may receive the tag for the identified object from the user after the object is identified 415 and before one or more images of the object are captured 420. Alternatively, the head-mounted device 100 receives the tag for the identified object after the object is identified 415 and after one or more images of the object are captured 420. Receiving the tag for the identified object from the user allows the user to provide a name or other identifier to make it easier to remember or identify the identified object.

[0086] In various embodiments, the object registration module 300 modifies one or more of the captured images to more clearly include the identified objects. For example, the object registration module 300 modifies the captured image of the identified objects by removing portions of the image outside the bounding boxes corresponding to the identified images. In various embodiments, when modifying the captured images, the bounding boxes corresponding to the identified objects in a 3D model of a local region surrounding the head-mounted device are mapped onto the captured images using the view matrix and projection matrix of the imaging device 130 that captured the images, thereby allowing the captured images to be cropped based on the dimensions of the bounding boxes of the identified objects in the 3D model of the local region. The object registration module 300 stores the modified images associated with the tags of the identified objects. Modifying the captured images allows the object registration module 300 to remove portions of the captured images that are not related to the identified objects.

[0087] Head-mounted device 100 uses one or more captured images of identified objects to train instance classifier 315 to detect identified objects in subsequently captured images. In various embodiments, instance classifier 315 is a machine learning model that includes a set of weights. These weights are parameters used by the machine learning model to transform input data received by the model into output data. For instance classifier 315, the input data includes one or more images of objects, and the output is labels applied to the objects. Weights can be generated through a training process, thereby training the machine learning model based on a set of training examples and labels associated with the training examples. The training examples used by object registration module 300 are captured images of identified objects stored in association with labels of identified objects, where the labels of the identified objects include the labels of the training examples. In various embodiments, the training process includes: applying the machine learning model to the training examples, comparing the output of the machine learning model with the labels associated with the training examples, and updating the weights associated with the machine learning model through a backpropagation process. The weights may be stored on one or more computer-readable media constituting the instance classifier. Subsequently, instance classifier 315 receives one or more images and outputs regions within the images that include identified objects. Therefore, instance classifier 315 allows the detection of specific objects from an image, while category classifier 310 identifies regions that include object categories found in one or more images, without distinguishing between different objects within a common category. This allows instance classifier 315 to detect specific objects from images of local regions captured by imaging device 130, thereby allowing head-mounted device 100 to track specific objects or identify specific objects within local regions.

[0088] In various embodiments, head-mounted device 100 displays to a user an indication that instance classifier 315 has been trained based on captured images of identified objects. For example, head-mounted device 100 displays a prompt or message to the user via one or more display elements 120, the prompt or message including text or images confirming that instance classifier 315 can detect identified objects. In various embodiments, the prompt or message may be displayed near the identified object or at a specific location on display element 120. As another example, head-mounted device 100 displays a specific color or pattern overlaid on the identified object by display element 120 to indicate that instance classifier 315 has been trained. In other embodiments, head-mounted device 100 displays the indication in response to storing one or more images of captured identified objects. Alternatively, the prompt or message may be an audio signal played by one or more speakers 160, a tactile signal causing movement of one or more portions of the frame 110 of the head-mounted device, or otherwise presented to the user via head-mounted device 100.

[0089] Figure 5 This is an example of a head-mounted device 100 registering objects for detection according to one or more embodiments. Figure 5 An example local area 500 surrounding the head-mounted device 100 is shown. Figure 5 In the example, local region 500 includes objects 505, 510, and 515. However, in other embodiments, local region 500 includes a different number of objects. Each of the multiple objects is within the field of view of one or more imaging devices 130 of the head-mounted device, which are positioned to capture an image of local region 500.

[0090] As described above Figure 3 and Figure 4 Further described, the head-mounted device 100 (or a computing device communicatively coupled to the head-mounted device 100) applies a category classifier 310 to an image of a local region 500 captured by one or more imaging devices 130. The category classifier 310 identifies regions within the image of the local region 500 that include candidate objects. The category classifier 310 identifies regions of the local region image that include one or more objects having one or more categories identified by the category classifier 310 during training, thus each region of the image of the local region 500 corresponds to a candidate object. In various embodiments, the category classifier 310 also identifies the category or type of each candidate object identified within the local region 500.

[0091] exist Figure 5 In the example, the category classifier 310 determines a bounding box for each detected candidate object. The bounding box of a candidate object specifies the boundary of the region of the image that includes the candidate object, where the region of the image within the bounding box includes the candidate object, and the region of the image outside the bounding box is not a candidate object. Figure 5 In the example, the category classifier determines a bounding box 520 corresponding to object 505, a bounding box 525 corresponding to object 510, and a bounding box 530 corresponding to object 515. In some embodiments, the head-mounted device 100 displays the bounding boxes 520, 525, and 530 to a user via display element 120 to identify candidate objects identified in the local region 500. In other embodiments, the head-mounted device 100 does not display the bounding boxes 520, 525, and 530, but instead stores the coordinates corresponding to each bounding box 520, 525, and 530 in an image of the local region 500.

[0092] In addition, the head-mounted device 100 determines the gaze direction of 535 users, as described above. Figure 2 and Figure 4Further described. The user's gaze direction indicates the location of the local area to which the user's gaze is directed. In various embodiments, the head-mounted device 100 includes an eye-tracking unit 200 that determines the user's gaze direction based on captured images of the user's eyes when the user wears the head-mounted device 100, as described above. Figure 2 Further description.

[0093] Head-mounted device 100 (e.g., object registration module 300) compares the user's gaze direction within a local region 500 with bounding boxes of candidate objects identified in the image of local region 500. In response to determining that the user's gaze direction falls within a region of local region 500 corresponding to a candidate object, head-mounted device 100 identifies the candidate object. For example, head-mounted device 100 identifies an object as a candidate object in response to determining that the location of the user's gaze direction in the local region corresponds to a bounding box in the image of local region 500 that includes the candidate object. In some embodiments, head-mounted device 100 identifies objects within a bounding box that includes a location where the user's gaze is directed for at least a threshold amount of time. Figure 5 In the example, head-mounted device 100 determines the location 540 of the user's gaze direction. Since location 540 is within bounding box 525, head-mounted device 100 identifies an object 510 included within bounding box 525. In some embodiments, head-mounted device 100 identifies object 510 in response to the user's gaze direction location 540 being within bounding box 525 for at least a threshold amount of time.

[0094] As described above Figure 4 Further described, in response to the identification of object 510, one or more imaging devices 130 of the head-mounted device 100 capture one or more images of object 510. The head-mounted device 100 stores the captured images of the identified object and trains an instance classifier 315 to detect object 510 from the image of local region 500. The instance classifier allows object 510 to be specifically detected from the image of local region 500, thereby allowing object 510 to be distinguished from other objects in local region 500 that have the same category as object 510. As described above... Figure 4 Further described, in various embodiments, the head-mounted device 100 prompts the user to capture images of the object 510 from different orientations relative to the imaging device 130, thus capturing different perspectives of the object 510 in different images to improve the accuracy of detecting the identified object (object 510) in the image of the local region 500.

[0095] Object recognition based on the user's gaze direction simplifies object recognition by reducing the complexity of user input. When the head-mounted device 100 or a device coupled to it identifies a region within the image of the local region 500 corresponding to a candidate object and determines the size of the bounding box for each candidate object, object recognition can be achieved without receiving user input to adjust or crop the image of the local region 500. This reduction in the number and complexity of user input simplifies object recognition, increasing the likelihood of the identified object being used for subsequent user interaction, thus allowing the head-mounted device 100 to offer additional functionality to the user.

[0096] Figure 6 The system 600 includes a head-mounted device 605 according to one or more embodiments. In some embodiments, the head-mounted device 605 may be... Figure 1A Head-mounted device 100, or Figure 1B The head-mounted device 105. The system 600 can operate in artificial reality environments (e.g., virtual reality environments, augmented reality environments, mixed reality environments, or some combination thereof). Figure 6 The illustrated system 600 includes a head-mounted device 605, an input / output (I / O) interface 610 coupled to a console 615, a network 620, and a map-building server 625. Although Figure 6 The illustrated example system 600 includes a head-mounted device 605 and an I / O interface 610; however, in other embodiments, system 600 may include any number of these components. For example, multiple head-mounted devices may be present, each having an associated I / O interface 610, wherein each head-mounted device and I / O interface 610 communicates with a console 615. In alternative configurations, system 600 may include different and / or additional components. Furthermore, in some embodiments, [the following is a continuation of the previous sentence, but the translation is incomplete]. Figure 6 The functions described by one or more components shown can be different from those in combination. Figure 6 The described methods are distributed among the various components. For example, some or all of the functions of the console 615 may be provided by the head-mounted device 605.

[0097] Head-mounted device 605 includes a display assembly 630, an optical block 635, one or more position sensors 640, and a DCA 645. Head-mounted device 605 also includes an audio system 650. Some embodiments of head-mounted device 605 have [unclear meaning - possibly related to a combination of components]. Figure 6 The components described are different from those in the original text. Additionally, in other embodiments, the components are combined... Figure 6The functions provided by the various components described may be distributed differently among the components of the head-mounted device 605, or may be embodied in separate components far from the head-mounted device 605.

[0098] Display component 630 displays content to the user based on data received from console 615. Display component 630 uses one or more display elements (e.g., display element 120) to display content. Display elements may be, for example, electronic displays. In various embodiments, display component 630 includes a single display element or multiple display elements (e.g., displays for each of the user's eyes). Examples of electronic displays include: liquid crystal displays (LCDs); organic light-emitting diode (OLED) displays, active-matrix organic light-emitting diode (AMOLED) displays, waveguide displays, some other type of display, or some combination thereof. Note that in some embodiments, display element 120 may also include some or all of the functions of optical block 635.

[0099] Optical block 635 can amplify image light received from an electronic display, correct optical errors associated with the image light, and present corrected image light to one or both eye-correcting zones of head-mounted device 605. In various embodiments, optical block 635 includes one or more optical elements. Example optical elements included in optical block 635 include: an aperture; a Fresnel lens; a convex lens; a concave lens; a filter; a reflective surface; or any other suitable optical element that affects the image light. Furthermore, optical block 635 can include combinations of different optical elements. In some embodiments, one or more optical elements in optical block 635 may have one or more coatings, such as a partial reflective coating or an anti-reflective coating.

[0100] The amplification and focusing of image light by optical block 635 makes the electronic display physically smaller, lighter, and consumes less power compared to larger displays. Furthermore, the amplification increases the field of view of the content displayed on the electronic display. For example, the field of view of the displayed content is such that almost the entire user's field of view (e.g., approximately 110 degrees diagonally) is used to present the displayed content, and in some cases, the entire user's field of view is used to present the displayed content. Additionally, in some embodiments, the amplification amount can be adjusted by adding or removing optical elements.

[0101] In some embodiments, the optical block 635 may be designed to correct one or more types of optical errors. Examples of optical errors include barrel or pincushion distortion, longitudinal or lateral chromatic aberration. Other types of optical errors may include spherical aberration; chromatic aberration; or errors due to lens field curvature, astigmatism; or any other type of optical error. In some embodiments, the content provided to the electronic display for display is pre-distorted, and the optical block 635 corrects the distortion when it receives image light from the electronic display (which is generated based on the content).

[0102] Position sensor 640 is an electronic device that generates data indicating the position of head-mounted device 605. Position sensor 640 generates one or more measurement signals in response to movement of head-mounted device 605. Position sensor 190 is an embodiment of position sensor 640. Examples of position sensor 640 include one or more IMUs, one or more accelerometers, one or more gyroscopes, one or more magnetometers, another suitable type of sensor for detecting motion, or some combination thereof. Position sensor 640 may include multiple accelerometers for measuring translational motion (forward / backward, up / down, left / right) and multiple gyroscopes for measuring rotational motion (e.g., pitch, yaw, roll). In some embodiments, the IMU rapidly samples the measurement signals and calculates an estimated position of head-mounted device 605 based on the sampled data. For example, the IMU integrates the measurement signals received from the accelerometers over time to estimate a velocity vector, and integrates the velocity vector over time to determine an estimated position of a reference point on head-mounted device 605. The reference point is a point that can be used to describe the position of head-mounted device 605. Although a reference point can generally be defined as a point in space, this reference point is actually defined as a point within the head-mounted device 605.

[0103] The DCA 645 generates depth information for a portion of a local region. The DCA includes one or more imaging devices and a DCA controller. The DCA 645 may also include an illuminator. The operation and structure of the DCA 645 are referenced above. Figure 1A As described above. In various embodiments, the DCA 645 includes an object registration module 300, as described above. Figures 3 to 5 As further described, the object registration module 300 identifies objects in a local region around the head-mounted device 605 based on the user's gaze direction and trains an instance classifier to subsequently detect the identified objects based on the image of the local region.

[0104] Audio system 650 provides audio content to a user of head-mounted device 605. Audio system 650 may include one or more acoustic sensors, one or more transducers, and an audio controller. Audio system 650 may provide spatialized audio content to the user. In some embodiments, audio system 650 may request acoustic parameters from map-building server 625 via network 620. The acoustic parameters describe one or more acoustic properties of a local area (e.g., room impulse response, reverberation time, reverberation level, etc.). Audio system 650 may provide, for example, information describing at least a portion of the local area from DCA 645 and / or location information of head-mounted device 605 from location sensor 640. Audio system 650 may use one or more acoustic parameters received from map-building server 625 to generate one or more sound filters and use said sound filters to provide audio content to the user.

[0105] I / O interface 610 is a device that allows a user to send action requests to console 615 and receive responses from console 615. An action request is a request to perform a specific action. For example, an action request may be an instruction to start or stop capturing image or video data, or an instruction to perform a specific action within an application. I / O interface 610 may include one or more input devices. Example input devices include a keyboard, mouse, game controller, or any other suitable device for receiving action requests and transmitting them to console 615. Action requests received by I / O interface 610 are transmitted to console 615, which performs the action corresponding to the action request. In some embodiments, I / O interface 610 includes an IMU that captures calibration data indicating an estimated position of I / O interface 610 relative to its initial position. In some embodiments, I / O interface 610 may provide haptic feedback to a user based on instructions received from console 615. For example, haptic feedback can be provided when an action request is received, or when the console 615 performs an action, the console 615 sends instructions to the I / O interface 610, thereby causing the I / O interface 610 to generate haptic feedback.

[0106] The console 615 provides content to the head-mounted device 605 for processing based on information received from one or more of the following: DCA 645; head-mounted device 605; and I / O interface 610. Figure 6 In the example shown, console 615 includes application repository 655, tracing module 660, and engine 665. Some embodiments of console 615 have a combination with... Figure 6 These modules or components are different modules or components. Similarly, the functions further described below can be combined with... Figure 6The described methods are distributed among the various components of the console 615 in different ways. In some embodiments, the functions of the console 615 discussed herein can be implemented in the head-mounted device 605 or a remote system.

[0107] Application repository 655 stores one or more applications for execution by console 615. An application is a set of instructions that, when executed by a processor, generate content to be presented to a user. The content generated by the application may be in response to input received from the user in response to movement via head-mounted device 605 or I / O interface 610. Examples of applications include: game applications, conferencing applications, video playback applications, or other suitable applications.

[0108] Tracking module 660 uses information from DCA 645, one or more position sensors 640, or a combination thereof to track the movement of head-mounted device 605 or I / O interface 610. For example, tracking module 660 determines the position of a reference point of head-mounted device 605 in a map of a local area based on information from head-mounted device 605. Tracking module 660 can also determine the position of objects or virtual objects. Additionally, in some embodiments, tracking module 660 can use data portions from position sensors 640 indicating the position of head-mounted device 605 and a representation of a local area from DCA 645 to predict the future position of head-mounted device 605. Tracking module 660 provides engine 665 with the estimated or predicted future position of head-mounted device 605 or I / O interface 610.

[0109] Engine 665 executes the application and receives positioning information, acceleration information, velocity information, predicted future positioning, or some combination thereof from the head-mounted device 605 via tracking module 660. Engine 665 determines the content to be presented to the user on the head-mounted device 605 based on the received information. For example, if the received information indicates that the user has looked to the left, engine 665 generates content for the head-mounted device 605 that reflects the user's movement within a virtual local area or a local area (enhanced with additional content). Additionally, in response to an action request received from I / O interface 610, engine 665 performs an in-application action on console 615 and provides feedback to the user that the action has been performed. This feedback can be visual or auditory feedback via head-mounted device 605, or haptic feedback via I / O interface 610.

[0110] Network 620 couples the head-mounted device 605 and / or console 615 to the map building server 625. Network 620 may include any combination of local area networks and / or wide area networks using both wireless communication systems and / or wired communication systems. For example, network 620 may include the Internet and mobile phone networks. In one embodiment, network 620 uses standard communication technologies and / or protocols. Therefore, network 620 may include links using technologies such as Ethernet, 802.11, worldwide interoperability for microwave access (WiMAX), 2G / 3G / 9G mobile communication protocols, digital subscriber line (DSL), asynchronous transfer mode (ATM), InfiniBand, PCI Express Advanced Switching, etc. Similarly, networking protocols used on Network 620 may include Multiprotocol Label Switching (MPLS), Transmission Control Protocol / Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), Simple Mail Transfer Protocol (SMTP), File Transfer Protocol (FTP), etc. Data exchanged through Network 620 may be represented using technologies and / or formats including binary image data (e.g., Portable Network Graphics (PNG)), Hypertext Markup Language (HTML), Extensible Markup Language (XML), etc.In addition, conventional encryption techniques can be used to encrypt all or some links, such as Secure Sockets Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), and Internet Protocol Security (IPsec).

[0111] Map building server 625 may include a database storing virtual models describing multiple spaces, wherein a location in the virtual model corresponds to the current configuration of a local area of ​​head-mounted device 605. Map building server 625 receives information describing at least a portion of the local area and / or location information of the local area from head-mounted device 605 via network 620. Users can adjust privacy settings to allow or prevent head-mounted device 605 from sending information to map building server 625. Map building server 625 determines the location in the virtual model associated with the local area of ​​head-mounted device 605 based on the received information and / or location information. Map building server 625 determines (e.g., retrieves) one or more acoustic parameters associated with the local area, in part based on the determined location in the virtual model and any acoustic parameters associated with the determined location. Map building server 625 may send the location of the local area and the values ​​of any acoustic parameters associated with the local area to head-mounted device 605.

[0112] One or more components in system 600 may include a privacy module that stores one or more privacy settings for user data elements. The user data elements describe a user or head-mounted device 605. For example, a user data element may describe the user's physical characteristics, actions performed by the user, the user's location on head-mounted device 605, the position of head-mounted device 605, the user's head-related transfer function (HRTF), etc. The privacy settings (or "access settings") of the user data elements may be stored in any suitable manner, such as being stored in association with the user data element, stored in an index on an authorization server, stored in another suitable manner, or any suitable combination thereof.

[0113] Privacy settings for user data elements specify how user data elements (or specific information associated with user data elements) can be accessed, stored, or otherwise used (e.g., viewed, shared, modified, copied, performed, displayed, or identified). In some embodiments, privacy settings for user data elements may specify a "blacklist" of entities that may be denied access to certain information associated with the user data element. Privacy settings associated with user data elements may specify any appropriate granularity for allowing or denying access. For example, some entities may have permission to ascertain the existence of a specific user data element, some entities may have permission to view the content of a specific user data element, and some entities may have permission to modify a specific user data element. Privacy settings may allow users to permit other entities to access or store user data elements for a limited period of time.

[0114] Privacy settings allow users to specify one or more geographic locations from which user data elements can be accessed. Access to or denial of access to user data elements can depend on the geographic location of the entity attempting to access the user data element. For example, a user can allow access to a user data element and specify that the user data element is only accessible to an entity while the user is in a specific location. If the user leaves that specific location, the user data element may no longer be accessible to that entity. As another example, a user can specify that a user data element is only accessible to entities within a threshold distance of the user (such as another user of a headset in the same local area as the user). If the user subsequently changes location, the entity with access to the user data element may lose access, while a new set of entities may gain access when they come within the user's threshold distance.

[0115] System 600 may include one or more authorization / privacy servers for implementing privacy settings. A request from an entity for a specific user data element may identify the entity associated with the request, and if the authorization server determines, based on the privacy settings associated with the user data element, that the entity is authorized to access the user data element, it may send the user data element only to that entity. If the requesting entity is not authorized to access the user data element, the authorization server may prevent the requested user data element from being retrieved or from being sent to the entity. Although this disclosure describes implementing privacy settings in a particular manner, this disclosure contemplates implementing privacy settings in any suitable manner.

[0116] Additional configuration information The above description of embodiments has been presented for illustrative purposes and is not intended to be exhaustive or to limit the patent rights to the precise forms disclosed. Those skilled in the art will understand that many modifications and variations are possible in light of the foregoing disclosure.

[0117] Several embodiments of the algorithms and symbolic representations of operations on information are described in parts of this specification. Those skilled in the art of data processing commonly use these algorithmic descriptions and representations to effectively communicate the substance of their work to others skilled in the art. Although these operations are described functionally, computationally, or logically, they should be understood as being implemented by computer programs or equivalent circuits, microcode, etc. Furthermore, without loss of generality, it is sometimes convenient to refer to the arrangement of these operations as modules. The described operations and their associated modules can be embodied in software, firmware, hardware, or any combination thereof.

[0118] Any step, operation, or process described herein may be performed or implemented individually or in combination with other devices using one or more hardware or software modules. In one embodiment, a software module is implemented using a computer program product comprising a computer-readable medium containing computer program code that can be executed by a computer processor to perform any or all of the described steps, operations, or processes.

[0119] The embodiments may also relate to an apparatus for performing the operations described herein. This apparatus may be specifically constructed for the desired purpose, and / or this apparatus may include a general-purpose computing device selectively activated or reconfigured by a computer program stored in a computer. Such a computer program may be stored in a non-transitory tangible computer-readable storage medium that can be coupled to a computer system bus, or in any type of medium suitable for storing electronic instructions. Furthermore, any computing system mentioned in this specification may include a single processor, or may employ an architecture employing a multiprocessor design to achieve increased computing power.

[0120] The embodiments may also relate to products generated by the computational processes described herein. Such products may include information generated by the computational processes, wherein the information is stored on a non-transitory tangible computer-readable storage medium, and such products may include any embodiment of the computer program product or other combination of data described herein.

Claims

1. A method comprising: Input from the user is received at the head-mounted device worn by the user to register one or more objects in a local area around the head-mounted device for subsequent detection; The head-mounted device determines the user's gaze direction in the local area based on captured information describing the user's single or double eyes; Objects in the local area are identified based on the determined gaze direction of the user; One or more images of the identified object are captured from one or more imaging devices included in the head-mounted device and configured to capture images of the local region; The captured one or more images are stored in association with tags corresponding to the identified objects.

2. The method according to claim 1, wherein, Identifying objects in the local region based on the determined gaze direction of the user includes: One or more regions of the one or more images of the local region captured by the one or more imaging devices are identified as candidate objects; Identify the regions in the 3D model of the local region that correspond to each of the candidate objects; and Identify objects in the 3D model of the local region that correspond to candidate objects within the region pointed to by the user's gaze direction.

3. The method according to claim 2, wherein, The objects identified in the 3D model of the local region that correspond to candidate objects within the region pointed to by the user's gaze direction include: The head-mounted device determines the bounding box corresponding to each candidate object in the 3D model of the local region without input from the user; and Identify candidate objects included in a bounding box at a location including the user's gaze direction; and preferably The candidate objects identified within the bounding box at a location including the user's gaze direction include: Identify candidate objects included in a bounding box that is located at a position including the user's gaze direction for at least a threshold amount of time.

4. The method according to claim 2 or 3, wherein, Identifying one or more regions of the one or more images of the local region captured by the one or more imaging devices as candidate objects includes: A category classifier is applied to one or more images of the local region, the category classifier identifying regions of the image that include at least one object.

5. The method according to any one of the preceding claims, wherein, The tag corresponding to the identified object is received from the user after the one or more images of the identified object are captured, or after the object is identified and before the one or more images of the identified object are captured.

6. The method according to any one of the preceding claims, wherein, Capturing one or more images of the identified object from one or more imaging devices included in the head-mounted device and configured to capture images of the local region includes: Capture multiple images of the identified object, each of the multiple images corresponding to a different position of the identified object relative to the one or more imaging devices.

7. The method according to any one of the preceding claims, further comprising: Based on the captured images of the identified objects, an instance classifier is trained to detect the identified objects within the images of the local region. And preferably, the method further includes: The user is shown an indication that the instance classifier has been trained via the display element of the head-mounted device.

8. A head-mounted device, comprising: frame; One or more display elements are coupled to the frame, each display element being configured to generate image light for presentation to the user; One or more imaging devices coupled to the frame, the one or more imaging devices being configured to capture images of a local region surrounding the frame; An eye-tracking unit configured to determine the user's gaze direction based on captured information describing the user's single or binocular eyes; as well as An object registration module, comprising a processor and a non-transitory computer-readable storage medium having instructions encoded thereon, the instructions causing the processor, when executed by the processor, to: Receive input from the user to register one or more objects in the local area surrounding the head-mounted device for subsequent detection; Objects in the local area are identified based on the determined gaze direction of the user; Capture one or more images of the identified object from the one or more imaging devices; as well as The captured one or more images are stored in association with tags corresponding to the identified objects.

9. The head-mounted device according to claim 8, wherein, Identifying objects in the local region based on the determined gaze direction of the user includes: One or more regions of the one or more images of the local region captured by the one or more imaging devices are identified as candidate objects; Identify the regions in the 3D model of the local region that correspond to each of the candidate objects; and Identify objects in the 3D model of the local region that correspond to candidate objects within the region pointed to by the user's gaze direction.

10. The head-mounted device according to claim 9, wherein, The objects identified in the 3D model of the local region that correspond to candidate objects within the region pointed to by the user's gaze direction include: The head-mounted device determines the bounding box corresponding to each candidate object in the 3D model of the local region without input from the user; and Identify candidate objects included in a bounding box at a location including the user's gaze direction; and preferably The candidate objects identified within the bounding box at a location including the user's gaze direction include: Identify candidate objects included in a bounding box that is located at a position including the user's gaze direction for at least a threshold amount of time.

11. The head-mounted device according to claim 9 or 10, wherein, Identifying one or more regions of the one or more images of the local region captured by the one or more imaging devices as candidate objects includes: A category classifier is applied to one or more images of the local region, the category classifier identifying regions of the image that include at least one object.

12. The head-mounted device according to any one of claims 8 to 11, wherein, The tag corresponding to the identified object is received from the user after one or more images of the identified object are captured.

13. The head-mounted device according to any one of claims 8 to 12, wherein, The tag corresponding to the identified object is received from the user after the object is identified and before the one or more images of the identified object are captured.

14. The head-mounted device according to any one of claims 8 to 13, wherein, Capturing one or more images of the identified object from one or more imaging devices included in the head-mounted device and configured to capture images of the local region includes: Capture multiple images of the identified object, each of the multiple images corresponding to a different position of the identified object relative to the one or more imaging devices.

15. The head-mounted device according to any one of claims 8 to 14, wherein, The instructions encoded on the non-transitory computer-readable storage medium also cause the processor to: Based on the captured images of the identified objects, an instance classifier is trained to detect the identified objects within the images of the local region; and preferably... The instructions encoded on the non-transitory computer-readable storage medium further enable the processor to: The user is shown an indication that the instance classifier has been trained via the display element of the head-mounted device.

Images