Displaying information based on gaze

Abstract

Systems and techniques for displaying information are described herein. For example, a device for displaying information is provided. The device can include at least one memory; and at least one processor coupled to the at least one memory and configured to: detect an object in an image of a scene obtained from a first camera; determine, based on an image of a user obtained from a second camera, that the user is gazing at a representation of the object displayed at a display; and based on determining that the user is gazing at the representation of the object displayed at the display, display, via the display, information associated with the object.

Description

Technical Field

[0001] This disclosure relates in its entirety to displaying information. For example, aspects of this disclosure include systems and techniques for displaying information based on gaze (e.g., a user's gaze). Background Technology

[0002] Devices can use displays to visually provide information, such as by displaying images, videos, and / or text. Many devices include displays; displays can be an important way for such devices to provide information to users. For example, smartphones, tablets, and extended reality (XR) devices (including augmented reality (AR), mixed reality (MR), and virtual reality (VR) devices) all include displays. Summary of the Invention

[0003] The following is a simplified summary of the invention relating to one or more aspects disclosed herein. Therefore, this summary should not be considered an exhaustive overview relating to all conceived aspects, nor should it be considered to identify key or decisive elements relating to all conceived aspects or to depict the scope associated with any particular aspect. Accordingly, the following summary presents certain concepts in a simplified form relating to one or more aspects of the mechanisms disclosed herein, preceding the detailed description that follows.

[0004] Systems and techniques for displaying information are described. According to at least one example, a method for displaying information is provided. The method includes: detecting an object in an image of a scene obtained from a first camera; determining, based on an image of a user obtained from a second camera, a representation of the object that the user is viewing, displayed on a monitor; and displaying information associated with the object via the monitor based on the determination that the user is viewing the object displayed on the monitor.

[0005] In another example, an apparatus for displaying information is provided, the apparatus comprising: at least one memory; and at least one processor (e.g., configured in a circuit) coupled to the at least one memory. The at least one processor is configured to: detect an object in an image of a scene obtained from a first camera; determine, based on an image of a user obtained from a second camera, a representation of the object that the user is viewing, displayed on a monitor; and, based on the determination that the user is viewing the object displayed on the monitor, display information associated with the object via the monitor.

[0006] In another example, a non-transitory computer-readable medium is provided having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to: detect an object in an image of a scene obtained from a first camera; determine, based on an image of a user obtained from a second camera, a representation of the object that the user is looking at and displayed on a monitor; and, based on the determination that the user is looking at the object displayed on the monitor, display information associated with the object via the monitor.

[0007] In another example, an apparatus for displaying information is provided. The apparatus includes: components for detecting an object in an image of a scene obtained from a first camera; components for determining, based on an image of a user obtained from a second camera, that the user is looking at a representation of the object displayed on a monitor; and components for displaying information associated with the object via the monitor based on the determination that the user is looking at the representation of the object displayed on the monitor.

[0008] In another example, an apparatus for displaying information is provided, the apparatus comprising: a first camera; a second camera; a display; at least one memory; and at least one processor coupled to the at least one memory and configured to: detect an object in an image of a scene obtained from the first camera; determine, based on an image of a user obtained from the second camera, a representation of the object that the user is looking at displayed on the display; and, based on the determination that the user is looking at the object displayed on the display, display information associated with the object via the display.

[0009] In some aspects, one or more of the devices described herein are, may be part of, or may include: extended reality devices (e.g., virtual reality (VR) devices, augmented reality (AR) devices, or mixed reality (MR) devices), vehicles (or computing devices, systems, or components of vehicles), mobile devices (e.g., mobile phones or so-called "smartphones," tablet computers, or other types of mobile devices), smart or connected devices (e.g., Internet of Things (IoT) devices), wearable devices, personal computers, laptop computers, video servers, televisions (e.g., network-connected televisions), robotic devices or systems, or other devices. In some aspects, each device may include one image sensor (e.g., a camera) or multiple image sensors (e.g., multiple cameras) for capturing one or more images. In some aspects, each device may include one or more displays for displaying one or more images, notifications, and / or other displayable data. In some aspects, each device may include one or more speakers, one or more light-emitting devices, and / or one or more microphones. In some aspects, each device may include one or more sensors. In some cases, one or more sensors may be used to determine the location of the device, the state of the device (e.g., tracking state, operating state, temperature, humidity level, and / or another state), and / or for other purposes.

[0010] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to define the scope of the claimed subject matter. This subject matter should be understood with reference to the appropriate portions of the entire specification, any or all drawings, and each claim.

[0011] The foregoing and other features and aspects will become more apparent from the following description, claims and accompanying drawings. Attached Figure Description

[0012] The following description, with reference to the accompanying drawings, details exemplary examples of this application: Figure 1 These are illustrations of example extended reality (XR) systems according to various aspects of this disclosure; Figure 2 This is an illustration of another example of an extended reality (XR) system according to various aspects of this disclosure; Figure 3 This is an illustration of yet another example of an extended reality (XR) system according to various aspects of this disclosure; Figure 4 This is a block diagram illustrating the architecture of an example extended reality (XR) system according to some aspects of this disclosure; Figure 5This is a diagram illustrating an example environment in which an example device, according to various aspects of this disclosure, can display information based on a user's gaze; Figure 6 This is a diagram illustrating another example environment in which an example device, according to various aspects of this disclosure, can display information based on a user's gaze; Figure 7 This is a block diagram illustrating an example device for displaying information based on user gaze, according to various aspects of this disclosure; Figure 8 This is a flowchart illustrating another example process for displaying information based on user gaze, according to various aspects of this disclosure; Figure 9 This is a block diagram illustrating examples of deep learning neural networks that can be used to perform various tasks, based on some aspects of the disclosed techniques; Figure 10 This is a block diagram illustrating examples of convolutional neural networks (CNNs) according to various aspects of this disclosure; and Figure 11 This is a block diagram illustrating an example computing device architecture that can implement the various technologies described herein. Detailed Implementation

[0013] Certain aspects of this disclosure are provided below. Some of these aspects may be applied independently, and some may be applied in combination, as will be apparent to those skilled in the art. Specific details are set forth in the following description for purposes of explanation in order to provide a thorough understanding of the various aspects of this application. However, it will be apparent that various aspects may be practiced without these specific details. The accompanying drawings and descriptions are not intended to be limiting.

[0014] The following description provides only exemplary aspects and is not intended to limit the scope, applicability, or configuration of this disclosure. Rather, the following description of exemplary aspects will provide those skilled in the art with a description that can be used to implement the exemplary aspects. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the spirit and scope of this application as set forth in the appended claims.

[0015] The terms “exemplary” and / or “example” are used herein to mean “serving as an example, instance, or illustration.” Any aspect described herein as “exemplary” and / or “example” is not necessarily to be construed as superior to or better than other aspects. Similarly, the term “aspects of this disclosure” does not require that all aspects of this disclosure include the features, advantages, or modes of operation discussed.

[0016] Extended reality (XR) systems may include VR systems that facilitate interaction with virtual reality (VR) environments, AR systems that facilitate interaction with augmented reality (AR) environments, MR systems that facilitate interaction with mixed reality (MR) environments, and / or other XR systems. For example, VR provides a fully immersive experience in a three-dimensional (3D) computer-generated VR environment or video that depicts a virtual version of a real-world environment. VR content may, in some cases, include VR videos that can be captured and rendered at very high quality, potentially providing a truly immersive virtual reality experience. Virtual reality applications may include games, training, education, sports videos, online shopping, and more. VR content may be rendered and displayed using VR systems or devices such as VR HMDs or other VR headsets that completely cover the user's eyes during the VR experience.

[0017] AR is a technology that delivers virtual or computer-generated content (referred to as AR content) onto a user's view of a physical, real-world scene or environment. AR content can include any virtual content, such as text, video, images, graphic content, location data (e.g., Global Positioning System (GPS) data or other location data), sound, any combination thereof, and / or other augmenting content. AR systems are designed to enhance (or augment) rather than replace a person's current perception of reality. For example, a user may see a real, stationary or moving physical object through an AR device display, but the user's visual perception of the physical object can be enhanced or augmented by a virtual image of that object (e.g., a real-world car replaced by a virtual image of DeLorean), by AR content added to the physical object (e.g., virtual wings added to a real-world pig), by AR content displayed relative to the physical object (e.g., virtual information displayed near a sign on a building, a virtual monster virtually anchored to one or more images of a real-world table (e.g., placed on top of that real-world table), and / or by displaying other types of AR content. Various types of AR systems can be used for games, entertainment, and / or other applications.

[0018] MR technology can combine aspects of VR and AR to provide users with immersive experiences. For example, in an MR environment, real-world and computer-generated objects can interact (e.g., a real person can interact with a virtual person as if the virtual person were a real person). Additionally or alternatively, MR can include VR headsets with AR capabilities; for example, an MR system can perform video pass-through (to mimic AR glasses) by transmitting images (and / or videos) of some real-world objects (such as keyboards and / or monitors) and / or taking into account real-world geometry (e.g., walls, tables). For example, in a game, the structure of a room can be retextured according to the game, but the geometry can still be based on the real-world geometry of the room.

[0019] In some cases, XR systems may include optical “see-through” or “transmit” displays (e.g., see-through or transmit AR HMDs or AR glasses), allowing the XR system to display XR content (e.g., AR content) directly onto a real-world view without displaying video content. For example, a user can view a physical object through a display (e.g., glasses or lenses), and the AR system can display AR content onto the display to provide the user with an enhanced visual perception of one or more real-world objects. In one example, the display of an optical see-through AR system may include lenses or glasses positioned in front of each eye (or a single lens or glasses positioned above both eyes). See-through displays allow users to see real-world objects or physical objects directly and can display (e.g., project or otherwise display) an enhanced image or additional AR content of that object to enhance the user's visual perception of the real world.

[0020] XR systems can partially, largely, or completely display visual information (e.g., text, images, and / or video) at a display to fill the user's field of view (e.g., using a see-through or pass-through display). XR systems typically include a display (e.g., a head-mounted display (HMD) or smart glasses), an image capture device adjacent to the display, and a processing device. In such XR systems, the image capture device captures images indicating the user's field of view, the processing device determines or obtains the visual information to be displayed based on the user's field of view and / or objects within it, and the display shows virtual content within the user's field of view.

[0021] Many devices, including handheld devices (such as smartphones and tablets) and extended reality (XR) devices (including augmented reality (AR), mixed reality (MR), and virtual reality (VR) devices), include displays. Such devices can use their respective displays to show visual information to users, such as images, videos, and / or text.

[0022] Users can interact with such devices through various interfaces, such as buttons on the device, buttons on attached or remote controllers, and / or touchscreens. In some cases, using such interfaces may be difficult or inconvenient for the user. For example, if a user is holding a handheld device and viewing a scene through it (e.g., in pass-through display operation mode), touching the touchscreen of the handheld device may be difficult or inconvenient. Additionally, touching such a touchscreen may obstruct part of the display, which may be undesirable. As another example, if a user is wearing a head-mounted display (HMD) of an XR device, the user may not want to hold the controllers. Furthermore, buttons on such an HMD may be inconvenient because they may be small and / or not in the user's view when the user is wearing the HMD.

[0023] This document describes systems, apparatuses, methods (also referred to as processes), and computer-readable media (collectively referred to herein as "systems and techniques") for displaying information based on user gaze. The systems and techniques described herein detect user gaze and determine the information to be displayed based on the user's gaze.

[0024] For example, systems and technologies can be implemented in XR devices. These systems and technologies enable XR devices to receive input from users based on their eyes (e.g., based on their gaze). For example, a user can use an XR device with AR or MR capabilities (e.g., allowing a user to view a scene and visual information displayed by the XR device, for example, using a perspective display or a pass-through display). Systems and technologies can determine what the user is looking at in the scene (through a perspective display or in a representation of the scene on a pass-through display). Systems and technologies can further determine the information to be displayed based on what the user is looking at.

[0025] As another example, the system and technology can be implemented in a handheld device (e.g., a smartphone or tablet). The system and technology enable the handheld device to receive input from the user based on the user's eyes. For example, a user can use the handheld device with AR or MR capabilities (e.g., allowing a user to view a scene and visual information displayed by the handheld device using the device's display in pass-through operation mode). The system and technology can determine what the user is looking at in the scene (e.g., through the handheld device in pass-through operation mode). The system and technology can further determine the information to be displayed based on what the user is looking at.

[0026] For example, if a user is looking at text in their first language, the system and technology can determine to translate the text into a second language (e.g., from a language the user cannot understand to a language the user can understand). As another example, if a user is looking at a product's barcode, the system and technology can determine to display information from an internet search for the product (e.g., pricing information and / or reviews). As yet another example, if a user is looking at a website's Quick Response (QR) code or Uniform Resource Locator (URL), the system and technology can determine to display information from the website.

[0027] In another example, if a user is viewing contact information such as a phone number, email address, or social media handle, the system and technology can determine to display information about the communication. For example, the system and technology could determine to display an inquiry about calling the phone number, a draft text message to the phone number, a draft email addressing the email address, a draft social media message associated with the social media handle, or an inquiry about saving contact information.

[0028] As another example, if a user is looking at an object or person, the system and technology can determine to display an identifier of the object or person to the user (e.g., the name or label of the object or person). The identifier could be the language chosen by the user, such as the user's own language or a language the user is learning.

[0029] As another example, if a user is looking at an object, the system and technology can determine to display information related to the recorded object. For example, a user may be looking at food they intend to eat. The system and technology can display a query about recording the food (e.g., using a food intake app). Additionally or alternatively, a user may be retrieving an object's inventory. The system and technology can determine to display a count of the objects and / or a query about the count of the recorded objects.

[0030] As another example, two users may be viewing a scene. At least one of the users may be viewing the scene through a monitor. The monitor may be a perspective monitor or a pass-through monitor. Alternatively, a scene-facing camera may capture images and / or video of the scene and display the images and / or video at the monitor regardless of whether the monitor is located within the scene. For example, the scene-facing camera may be located at one location in the scene, and the monitor may be located at another location. This configuration may be referred to as a remote pass-through configuration. The system and technology may also include one or more user-facing cameras that capture images of one or both users (specifically, the users' eyes). The system and technology may determine the gaze of one or both users and determine the information to be displayed at the monitor based on the gaze. For example, a parent may be viewing a scene with a child. The child may be viewing the scene through a monitor (e.g., in a perspective display configuration, a pass-through display configuration, or a remote pass-through configuration). The system and technology may capture images of the parent's eyes and determine the parent's gaze. The system and technology may further determine objects in the scene. The system and technology may determine that the parent is looking at a specific object in the scene. Systems and technologies can determine which information related to a specific object to display on a monitor (e.g., for children to watch) based on parental gaze.

[0031] Systems and techniques (as described in the examples above or in other use cases) can use perspective displays or pass-through displays to overlay information onto what the user is looking at (e.g., between the user's eyes and the content the user is looking at in the scene within the user's field of view). For example, if the user is looking at text in a language unknown to the user, the system and techniques can overlay that text with a translation in a language known to the user. As another example, if the user is looking at an object, the system and techniques can overlay the object's identifier onto the object within the user's field of view.

[0032] Additionally or alternatively, the system and techniques may display information within the user's field of view that is adjacent to the content the user is looking at (e.g., next to the content the user is looking at). In some aspects, the system and techniques may display information overlaid on the background of the scene (e.g., as determined using depth detection techniques such as time-of-flight, stereoscopic imaging, structured light, and / or monocular depth detection). In some aspects, the system and techniques may display information overlaid on visually uniform portions of the scene (e.g., blank walls determined by analysis of images of the scene). In some aspects, the system and techniques may display information overlaid on non-interesting portions of the scene (e.g., as determined by tracking the user's gaze over time).

[0033] Additionally or alternatively, the system and technology may determine to stop displaying information and / or determine not to display information based on the user's gaze. For example, if the user is looking at an object in the scene, the system and technology may determine to display information based on determining that the user is looking at the object. The user may stop looking at the object; for example, the user may look at another object in the scene. The system and technology may determine that the user is no longer looking at the object, and determine to stop displaying information based on the user no longer looking at the object. Additionally or alternatively, the system and technology may determine to display other information based on the user being looking at another object in the scene.

[0034] In some aspects, systems and technologies can initiate actions in response to determining that a user is interested in an object in the scene. Actions can be object-based. In some aspects, systems and technologies can further initiate actions in response to instructions from the user. In some cases, information displayed on the monitor can be a prompt for the instruction. In some aspects, systems and technologies can interpret the user's vocalizations, hand gestures, head movements, and / or eye movements into instructions.

[0035] For example, in response to determining that a user is interested in text in a scene (e.g., based on the user looking at the text), the system and technology may display information related to the text (e.g., a translation of the text or information from a text-based internet search). The system and technology may detect the user's hand gestures (e.g., based on an image of the user's hand) or the user's head movements (e.g., based on data from an inertial measurement unit (IMU) of a head-mounted display). The system and technology may interpret the hand gestures or head movements as instructions for the user to hear the text (or text-related information) spoken. In response to interpreting the hand gestures or head movements, the system and technology may generate audio of the text being spoken.

[0036] As another example, in response to determining that a user is interested in a barcode in a scene, the system and technology can display information about the product associated with the barcode (e.g., the price of the product from an online retailer). The system and technology can detect the user's vocalization and interpret the vocalization as an instruction that the user wishes to purchase the product from the online retailer. In response to interpreting the vocalization, the system and technology can initiate a transaction to purchase the product from the online retailer.

[0037] As another example, in response to determining that a user is interested in a phone number in a scene, the system and technology can display prompts asking the user whether they want to save the phone number or call it. The system and technology can detect the user's eye movements (e.g., blinking and / or sustained gaze at the phone number). The system and technology can interpret these eye movements as an indication that the user wants to call the phone number. In response to interpreting the eye movements, the system and technology can initiate a phone call to the phone number.

[0038] In some aspects, the system and techniques can be implemented in a device including a scene-oriented camera, a user-oriented camera, and a display. The scene-oriented camera captures images of the scene, and the system and techniques can identify objects in the scene based on these images. The user-oriented camera captures images of the user (e.g., the user's eyes), and the system and techniques can determine the user's gaze based on these images. The system and techniques can determine, based on the user's gaze, that the user is interested in one of the identified objects in the scene. For example, the system and techniques can compare the user's gaze relative to the scene with the position of an object in the scene. Having determined that the user is interested in an object, the system and techniques can determine the information to be displayed to the user based on this interest. The system and techniques can further display the information at the display. The display can be positioned in the user's field of view between the user's eyes and the scene (e.g., such that the system and techniques can overlay information onto objects within the user's field of view for display).

[0039] By displaying information based on the user's gaze, systems and technologies enable devices (e.g., XR devices and handheld devices) to receive user input in a more user-friendly manner compared to other interfaces such as buttons and touchscreens. Therefore, devices implementing these systems and technologies can be more user-friendly than other devices.

[0040] Various aspects of this application will be described below with reference to the accompanying drawings.

[0041] Figure 1 This is an illustration of an example extended reality (XR) system 100 according to various aspects of the present disclosure. As shown, the XR system 100 includes an XR device 102. The XR device 102 can implement aspects of extended reality (including virtual reality (VR), augmented reality (AR), and / or mixed reality (MR)) such as image capture, object detection, gaze tracking, view tracking, computation, and / or display. For example, the XR device 102 may include one or more scene-facing cameras that can capture images of a scene in which a user 108 uses the XR device 102. The XR device 102 can detect objects in the scene based on the images of the scene. Furthermore, the XR device 102 may include one or more user-facing cameras that can capture images of the user 108's eyes. The XR device 102 can determine the user 108's gaze based on the images of the user 108. The XR device 102 can determine objects of interest in the scene based on the user 108's gaze. The XR device 102 can acquire and / or render information (e.g., text, images, and / or videos based on objects of interest). XR device 102 can display information to user 108 (e.g., within user 108's field of view 110).

[0042] XR device 102 can display information to be viewed by user 108 within user 108's field of view 110. For example, in a "perspective" configuration, XR device 102 may include a transparent surface (e.g., optical glass) such that information can be displayed on the transparent surface (e.g., by projection onto the transparent surface) to overlay the information onto a scene as viewed through the transparent surface. In a "transparent" configuration, XR device 102 may include a scene-facing camera that captures images of the scene for user 108. XR device 102 can display captured images or videos of the scene as captured by the scene-facing camera, along with information overlaid on the images or videos of the scene.

[0043] In various examples, XR device 102 may be or may include a head-mounted display (HMD), a virtual reality head-mounted device, and / or smart glasses. XR device 102 may include one or more cameras (including scene-oriented cameras and / or user-oriented cameras), a GPU, one or more sensors (e.g., one or more inertial measurement units (IMUs), image sensors, and / or microphones) and / or one or more output devices (e.g., speakers, displays, and / or smart glasses).

[0044] Figure 2 This is an illustration of an example extended reality (XR) system 200 according to various aspects of the present disclosure. As shown, the XR system 200 includes an XR device 202, an accessory device 204, and a communication link 206 between the XR device 202 and the accessory device 204. The XR device 202 can implement aspects of extended reality (including virtual reality (VR), augmented reality (AR), and / or mixed reality (MR)), such as image capture, view tracking, and / or display. For example, the XR device 202 may include one or more scene-oriented cameras that can capture images of a scene in which a user 208 uses the XR device 202. Furthermore, the XR device 202 may include one or more user-oriented cameras that can capture images of the user 208's eyes. The XR device 202 can provide the scene images and / or the user 208's images to the accessory device 204 (e.g., via the communication link 206).

[0045] The accessory device 204 enables computational aspects of extended reality, including, for example, object detection, gaze tracking, information gathering, and / or information generation. For instance, the accessory device 204 can receive images of the scene and / or the eyes of user 208. The accessory device 204 can detect objects in the scene based on the received images of the scene. Furthermore, the accessory device 204 can determine the gaze of user 208 based on images received by user 208 (e.g., the eyes of user 208). The accessory device 204 can determine objects of interest in the scene based on the gaze of user 208. The accessory device 204 can acquire and / or render information (e.g., text, images, and / or video based on the objects of interest). The accessory device 204 can provide information to XR device 202 (e.g., via communication link 206). XR device 202 can display information to user 208 (e.g., within user 208's field of view 210).

[0046] XR device 202 can display information to be viewed by user 208 within user 208's field of view 210. For example, in a "perspective" configuration, XR device 202 may include a transparent surface (e.g., optical glass) such that information can be displayed on the transparent surface (e.g., by projection onto the transparent surface) to overlay information onto a scene as viewed through the transparent surface. In a "transparent" configuration, XR device 202 may include a scene-facing camera that captures images of the scene for user 208. XR device 202 can display captured images or videos of the scene as captured by the scene-facing camera, along with information overlaid on the images or videos of the scene.

[0047] In various examples, XR device 202 may be or may include a head-mounted display (HMD), a virtual reality headset, and / or smart glasses. XR device 202 may include one or more cameras (including scene-oriented cameras and / or user-oriented cameras), a GPU, one or more sensors (e.g., one or more inertial measurement units (IMUs), image sensors, and / or microphones) and / or one or more output devices (e.g., speakers, displays, and / or smart glasses). Companion device 204 may be or may include a smartphone, laptop computer, tablet computer, personal computer, gaming system, server computer or server equipment (e.g., an edge or cloud-based server, a personal computer acting as a server equipment, or a mobile device acting as a server equipment), any other computing device, and / or combinations thereof. Communication link 206 may be based on any suitable wireless protocol (such as, for example, IEEE 802.11 (Wi-Fi), IEEE 802.15, or Bluetooth). ®The communication link 206 can be a direct wireless connection between the XR device 202 and the companion device 204 in some cases. In other cases, the communication link 206 can be via one or more intermediate devices, such as routers or switches and / or across a network.

[0048] Figure 3 This is an illustration of an example extended reality (XR) system 300 according to various aspects of the present disclosure. As shown, the XR system 300 includes a handheld device 302, which includes a display 304. In some cases, the handheld device 302 may implement aspects of extended reality (including virtual reality (VR), augmented reality (AR), and / or mixed reality (MR)) such as image capture, object detection, gaze tracking, view tracking, computation, and / or display. For example, the handheld device 302 may include one or more scene-facing cameras that capture images of a scene in which a user 308 uses the handheld device 302. The handheld device 302 may detect objects in the scene based on the images of the scene. Furthermore, the handheld device 302 may include one or more user-facing cameras that capture images of the user 308's eyes. The handheld device 302 may determine the user 308's gaze based on the images of the user 308. The handheld device 302 may determine objects of interest in the scene based on the user 308's gaze. The handheld device 302 can acquire and / or render information (e.g., text, images, and / or video based on an object of interest). The handheld device 302 can display the information to the user 308 at the display 304 (e.g., within the user 308's field of view 310).

[0049] Handheld device 302 can display information to be viewed by user 308 within user 308's field of view 310. Handheld device 302 can operate in a "transparent" configuration. For example, handheld device 302 may include a scene-facing camera that captures images of the scene as seen by user 308. Handheld device 302 can display captured images or videos of the scene as seen by the scene-facing camera, along with information overlaid on the images or videos of the scene. Handheld device 302 can display information to be viewed by user 308 within user 308's field of view 310. As another example, in a "perspective" configuration, handheld device 302 may include a transparent surface (e.g., optical glass) such that information can be displayed on the transparent surface to overlay information onto the scene as seen through the transparent surface.

[0050] Handheld device 302 and / or display 304 may be or may include a handheld device, smartphone, tablet, or other computing device with a display. Handheld device 302 includes one or more cameras (including scene-oriented cameras and / or user-oriented cameras), a GPU, one or more sensors (e.g., one or more inertial measurement units (IMUs), image sensors, and / or microphones) and / or one or more output devices (e.g., speakers, displays, and / or smart glasses).

[0051] Figure 4 This is a diagram illustrating the architecture of an example extended reality (XR) system 400 according to some aspects of this disclosure. The XR system 400 can execute XR applications and implement XR operations. For example, the XR system 400 can implement aspects of extended reality (including virtual reality (VR), augmented reality (AR), and / or mixed reality (MR)) such as image capture, object detection, gaze tracking, view tracking, computation, and / or display. Figure 1 XR system 100, Figure 2 XR system 200 and / or Figure 3 Any of the 300 XR systems can achieve Figure 4 The architecture of the XR system 400.

[0052] In this exemplary example, the XR system 400 includes one or more image sensors 402, accelerometers 404, gyroscopes 406, storage devices 408, input devices 410, displays 412, computing components 414, XR engines 424, image processing engines 426, rendering engines 428, and communication engines 430. It should be noted that... Figure 4 The components 402 to 432 shown are non-limiting examples provided for illustrative and explanatory purposes, and other examples may include those with... Figure 4 The components shown may be more numerous, fewer, or different than those shown. For example, in some cases, the XR system 400 may include one or more other sensors (e.g., one or more inertial measurement units (IMUs), radar, light detection and ranging (LIDAR) sensors, radio detection and ranging (RADAR) sensors, sound detection and ranging (SODAR) sensors, sound navigation and ranging (SONAR) sensors, audio sensors, etc.), one or more display devices, one or more other processing engines, one or more other hardware components, and / or Figure 4 One or more other software and / or hardware components not shown herein. While various components of the XR system 400 (such as image sensor 402) may be referred to in the singular herein, it should be understood that the XR system 400 may include any of the components discussed herein (e.g., multiple image sensors 402).

[0053] Display 412 may be or may include glass, screen, lens, projector and / or other display mechanism that allows users to see a real-world environment and also allows XR content to be overlaid on, overlapped with, mixed with or otherwise displayed on the real-world environment.

[0054] XR system 400 may include input device 410 or be able to communicate with such input device (wired or wirelessly). Input device 410 may include any suitable input device, such as a touchscreen, pen or other pointing device, keyboard, mouse, buttons or keys, microphone for receiving voice commands, gesture input device for receiving gesture commands, video game controller, steering wheel, joystick, set of buttons, trackball, remote control, any other input device discussed herein, or any combination thereof. In some cases, image sensor 402 may capture images that can be processed for interpreting gesture commands.

[0055] The XR system 400 can also communicate with one or more other electronic devices (wired or wireless). For example, the communication engine 430 can be configured to manage connections and communicate with one or more electronic devices. In some cases, the communication engine 430 may correspond to... Figure 11 The communication interface is 1126.

[0056] In some embodiments, image sensor 402, accelerometer 404, gyroscope 406, storage device 408, display 412, computing component 414, XR engine 424, image processing engine 426, and rendering engine 428 may be part of the same computing device. For example, in some cases, image sensor 402, accelerometer 404, gyroscope 406, storage device 408, display 412, computing component 414, XR engine 424, image processing engine 426, and rendering engine 428 may be integrated into HMDs, extended reality glasses, smartphones, laptops, tablets, gaming systems, and / or any other computing device. However, in some embodiments, image sensor 402, accelerometer 404, gyroscope 406, storage device 408, display 412, computing component 414, XR engine 424, image processing engine 426, and rendering engine 428 may be part of two or more independent computing devices. For example, in some cases, some of the components 402 to 432 may be part of or implemented by a computing device, and the remaining components may be part of or implemented by one or more other computing devices. For instance, in a discrete-sensory XR system, XR system 400 may include a first device (e.g., an HMD) that includes a display 412, an image sensor 402, an accelerometer 404, a gyroscope 406, and / or one or more computing components 414. XR system 400 may also include a second device that includes additional computing components 414 (e.g., implementing an XR engine 424, an image processing engine 426, a rendering engine 428, and / or a communication engine 430). In this example, the second device may generate virtual content based on information or data (e.g., images, sensor data, such as measurements from accelerometer 404 and gyroscope 406) and may provide the virtual content to the first device for display at the first device. The second device may be or may include a smartphone, laptop computer, tablet computer, personal computer, gaming system, server computer or server equipment (e.g., an edge or cloud-based server, a personal computer acting as a server equipment, or a mobile device acting as a server equipment), any other computing device and / or combinations thereof.

[0057] Storage device 408 can be any storage device used for storing data. Furthermore, storage device 408 can store data from any component of the XR system 400. For example, storage device 408 can store data from image sensor 402 (e.g., image or video data), data from accelerometer 404 (e.g., measurements), data from gyroscope 406 (e.g., measurements), data from computing component 414 (e.g., processing parameters, preferences, virtual content, rendered content, scene maps, tracking and positioning data, object detection data, privacy data, XR application data, facial recognition data, occlusion data, etc.), data from XR engine 424, data from image processing engine 426, and / or data from rendering engine 428 (e.g., output frames). In some examples, storage device 408 may include a buffer for storing frames processed by computing component 414.

[0058] Computing component 414 may be or may include a central processing unit (CPU) 416, a graphics processing unit (GPU) 418, a digital signal processor (DSP) 420, an image signal processor (ISP) 422, a neural processing unit (NPU) 432 that implements one or more trained neural networks, and / or other processors. Computing component 414 may perform various operations such as image enhancement, computer vision, graphics rendering, extended reality operations (e.g., tracking, localization, pose estimation, map building, content anchoring, content rendering, prediction, etc.), image and / or video processing, sensor processing, recognition (e.g., text recognition, face recognition, object recognition, feature recognition, tracking or pattern recognition, scene recognition, occlusion detection, etc.), trained machine learning operations, filtering, and / or any of the various operations described herein. In some examples, computing component 414 may implement (e.g., control, operate, etc.) an XR engine 424, an image processing engine 426, and a rendering engine 428. In other examples, computing component 414 may also implement one or more other processing engines.

[0059] Image sensor 402 may include any image and / or video sensor or capture device. In some examples, image sensor 402 may be part of a multi-camera assembly, such as a dual-camera assembly. Image sensor 402 may include one or more scene-oriented cameras and one or more user-oriented cameras. For example, image sensor 402 may include one or more scene-oriented cameras configured to face the scene (e.g., away from the user). Additionally, image sensor 402 may include one or more user-oriented cameras configured to face the user. The user-oriented camera may be configured to capture an image of the user's eye. Image sensor 402 may capture image and / or video content (e.g., raw image and / or video data), which may then be processed by computing component 414, XR engine 424, image processing engine 426, and / or rendering engine 428, as described herein.

[0060] In some examples, image sensor 402 may capture image data and may generate an image (also called a frame) based on that image data and / or provide the image data or frame to XR engine 424, image processing engine 426, and / or rendering engine 428 for processing. The image or frame may include a video frame in a video sequence or a still image. The image or frame may include an array of pixels representing a scene. For example, the image may be: a red-green-blue (RGB) image with red, green, and blue color components per pixel; a lightness, redness, blueness (YCbCr) image with a lightness component and two chromaticity (color) components (redness and blueness) per pixel; or any other suitable type of color or monochrome image. In some aspects, image sensor 402 may include an infrared (IR) sensor that captures light outside the visible spectrum. The IR sensor may be used for low-light settings. As an example, XR system 400 may include an HMD (e.g., including display 412) that may include a baffle to block light from the scene from reaching the user's eyes. As another example, XR system 400 may include sunglasses (e.g., including display 412) that block light from the scene from reaching the user's eyes. Image sensor 402 may include an IR sensor (e.g., a user-facing IR sensor) for capturing an image of the user's eyes even when light from the scene is blocked.

[0061] In some cases, image sensor 402 (and / or other cameras of XR system 400) may be configured to also capture depth information. For example, in some implementations, image sensor 402 (and / or other cameras) may include an RGB depth (RGB-D) camera. In some cases, XR system 400 may include one or more depth sensors (not shown) that are separate from image sensor 402 (and / or other cameras) and can capture depth information. For example, such depth sensors may acquire depth information independently of image sensor 402. In some examples, depth sensors may be physically mounted in the same general location or orientation as image sensor 402, but may operate at a different frequency or frame rate than image sensor 402. In some examples, depth sensors may take the form of a light source that projects a structured or textured light pattern (which may include one or more narrowband lights) onto one or more objects in a scene. Depth information can then be obtained by utilizing the geometric deformation of the projected pattern caused by the surface shape of the objects. In one example, depth information may be obtained from a stereo sensor, such as a combination of an infrared structured light projector and an infrared camera registered to a camera (e.g., an RGB camera).

[0062] The XR system 400 may also include other sensors among its one or more sensors. The one or more sensors may include one or more accelerometers (e.g., accelerometer 404), one or more gyroscopes (e.g., gyroscope 406), and / or other sensors. The one or more sensors may provide velocity, orientation, and / or other positioning-related information to the computing component 414. For example, accelerometer 404 may detect the acceleration of the XR system 400 and may generate an acceleration measurement based on the detected acceleration. In some cases, accelerometer 404 may provide one or more translation vectors (e.g., up / down, left / right, forward / backward) that may be used to determine the positioning or attitude of the XR system 400. Gyroscope 406 may detect and measure the orientation and angular velocity of the XR system 400. For example, gyroscope 406 may be used to measure the pitch, roll, and yaw of the XR system 400. In some cases, gyroscope 406 may provide one or more rotation vectors (e.g., pitch, yaw, roll). In some examples, the image sensor 402 and / or the XR engine 424 may use measurements obtained by the accelerometer 404 (e.g., one or more translation vectors) and / or measurements obtained by the gyroscope 406 (e.g., one or more rotation vectors) to calculate the attitude of the XR system 400. As previously noted, in other examples, the XR system 400 may also include other sensors such as an inertial measurement unit (IMU), a magnetometer, gaze and / or eye-tracking sensors, machine vision sensors, smart scene sensors, voice recognition sensors, impact sensors, vibration sensors, positioning sensors, tilt sensors, etc.

[0063] As noted above, in some cases, one or more sensors may include at least one IMU. An IMU is an electronic device that uses a combination of one or more accelerometers, one or more gyroscopes, and / or one or more magnetometers to measure specific forces, angular velocities, and / or orientations of the XR system 400. In some examples, one or more sensors may output measurement information associated with the capture of images by the image sensor 402 (and / or other cameras of the XR system 400) and / or depth information obtained using one or more depth sensors of the XR system 400.

[0064] The XR engine 424 can use the output of one or more sensors (e.g., accelerometer 404, gyroscope 406, one or more IMUs and / or other sensors) to determine the attitude of the XR system 400 (also known as head attitude) and / or the attitude of the image sensor 402 (or other cameras of the XR system 400). In some cases, the attitude of the XR system 400 and the attitude of the image sensor 402 (or other cameras) can be the same. The attitude of the image sensor 402 refers to the attitude of the image sensor 402 relative to a reference frame (e.g., relative to...). Figure 2 The camera pose (210) is determined by the positioning and orientation of the field of view. In some implementations, the camera pose can be determined with respect to 6 degrees of freedom (6DoF), which refers to three translational components (e.g., which can be given by X (horizontal), Y (vertical), and Z (depth) coordinates relative to a reference frame such as the image plane) and three angular components (e.g., roll, pitch, and yaw relative to the same reference frame). In some implementations, the camera pose can be determined with respect to 3 degrees of freedom (3DoF), which refers to three angular components (e.g., roll, pitch, and yaw).

[0065] In some cases, a device tracker (not shown) may use measurements from one or more sensors and image data from image sensor 402 to track the pose (e.g., 6DoF pose) of the XR system 400. For example, the device tracker may fuse visual data from the image data (e.g., using a visual tracking solution) with inertial data from the measurements to determine the position and motion of the XR system 400 relative to the physical world (e.g., a scene) and a map of the physical world. As described below, in some examples, when tracking the pose of the XR system 400, the device tracker may generate a three-dimensional (3D) map of the scene (e.g., the real world) and / or generate updates to the 3D map for that scene. 3D map updates may include, for example, but not limited to, new or updated features and / or features or landmarks associated with the scene and / or the 3D map of that scene, positioning updates identifying or updating the position of the XR system 400 within the scene and the 3D map of that scene, etc. The 3D map provides a digital representation of the scene in the real / physical world. In some examples, 3D maps can anchor location-based objects and / or content to real-world coordinates and / or objects. XR system 200 can use mapped scenes (e.g., scenes in the physical world represented by a 3D map and / or scenes associated with that 3D map) to merge the physical and virtual worlds and / or merge virtual content or objects with the physical environment.

[0066] In some cases, the XR system 400 can also track the user's hands and / or fingers to allow the user to interact with and / or control virtual content in the virtual environment. For example, the XR system 400 can track the posture and / or movement of the user's hands and / or fingertips to identify or translate the user's interaction with the virtual environment. User interaction may include, for example, but not limited to, moving virtual content items, resizing virtual content items, selecting input interface elements in the virtual user interface (e.g., a virtual representation of a mobile phone, a virtual keyboard, and / or other virtual interfaces), and providing input through the virtual user interface.

[0067] Figure 5This is an illustration of an example environment 500 in which an example device 520, according to various aspects of this disclosure, can display information 528 based on the gaze 532 of a user 530. Generally, device 520 may use one or more scene-oriented cameras 522 to capture images of scene 502. Device 520 may detect objects in the image of scene 502 (e.g., object 504, object 506, object 508, text 510, quick response (QR) code 512, and / or Uniform Resource Locator (URL) 516). Furthermore, device 520 may use one or more user-oriented cameras 524 to capture images of the eyes of user 530. Device 520 may track the gaze 532 of user 530. Device 520 may determine the object of interest of user 530 based on the gaze 532 of user 530. Device 520 may then determine the information 528 to be displayed at display 526 based on the object of interest.

[0068] Scene 502 may include any scene, indoors or outdoors. Scene 502 may be or may include anything captured by an image from a scene-oriented camera 522. Scene 502 may include any number of objects, such as people, animals, vehicles, buildings, mountains, trees, signs, documents, text, QR codes, barcodes, contact information, URLs, etc. For example, such as Figure 5 As illustrated, scene 502 may include several objects in the foreground or background of an image. Alternatively, in some cases, scene 502 may include one object (e.g., a document). In such cases, an object may include more objects (e.g., more or more lines of text). An object may be or may include anything that can be detected as an object by device 520. Object 504, object 506, object 508, text 510, QR code 512, phone number 514, and URL 516 are provided as examples of objects.

[0069] Device 520 may be an XR device or a handheld device (e.g., capable of AR or MR). Device 520 may be or may include a head-mounted display (HMD), a virtual reality headset, and / or smart glasses, for example, Figure 1 XR device 102 and / or Figure 2 The XR device 202 and accompanying device 204 can be examples of device 520. Alternatively, device 520 can be or may include a handheld device, smartphone, tablet, or other computing device with a display, for example, Figure 3 The handheld device 302 can be an example of device 520. Furthermore, device 520 can implement... Figure 4The architecture of the XR system 400. In some aspects (e.g., when the device 520 is implemented as an HMD, VR headset, or smart glasses), the display 526 may include two screens or lenses for displaying images according to the principle of stereoscopic depth. Alternatively, in some aspects (e.g., when the device 520 is implemented as a handheld device), the device 520 may include a single display. In either case, the device 520 can implement AR or MR by displaying information (e.g., text, images, and / or video) in the user 530's field of view (e.g., between the user 530's eyes and scene 502).

[0070] Device 520 may include one or more scene-oriented cameras 522. The scene-oriented cameras 522 may be configured to point towards scene 502 (e.g., away from user 530). For example, the scene-oriented cameras 522 may be pointed opposite to display 526, such that when user 530 is viewing display 526, the scene-oriented cameras 522 are away from user 530 and pointed towards scene 502. The scene-oriented cameras 522 may periodically capture images of scene 502, for example, at a certain frame rate (e.g., 30 frames per second (FPS) or 60 FPS).

[0071] Device 520 can detect objects in an image of scene 502. For example, device 520 can implement a trained object detection model. Device 520 can use the trained object detection model to identify objects in an image of scene 502 captured by a scene-oriented camera 522. For example, device 520 can detect each of object 504, object 506, object 508, text 510, QR code 512, phone number 514, and / or URL 516. In some aspects, device 520 can classify the objects (e.g., device 520 can determine the category to which the object belongs).

[0072] In some respects, device 520 can detect objects based on a configurable list of objects of interest (e.g., user 530's). For example, user 530 can edit the list of objects that user 530 may be interested in, including things such as: text, text in the target language, text about the target topic, contact information, barcodes, barcodes of certain types of products, QR codes, food items, products, and specific types of things (e.g., tasks that the user can be assigned to count or inventory).

[0073] Device 520 may include one or more user-facing cameras 524. The user-facing cameras 524 may be configured to point at user 530. For example, the user-facing cameras 524 may be pointed in the same direction as display 526 and / or at an angle to view user 530's eyes while user 530 is viewing display 526. The user-facing cameras 524 may periodically capture images of user 530, for example, at a certain frame rate (e.g., 30 frames per second (FPS) or 60 FPS).

[0074] In some aspects, scene-oriented camera 522 and / or user-oriented camera 524 may be configured to capture images whenever device 520 is powered on (including when device 520 is displaying information on display 526, when display 526 is not displaying information, and / or when display 526 is locked). Scene-oriented camera 522 and / or user-oriented camera 524 may be, or may include, cameras that can be referred to as “always-on cameras” or “always-sensing cameras.” For example, scene-oriented camera 522 and / or user-oriented camera 524 may be capturing images (e.g., at a certain frame capture rate) whenever device 520 is active (or “powered on”). In such aspects, camera 522 and / or camera 524 may be capturing images regardless of whether display 526 is displaying information. For example, display 526 may be turned off, while camera 522 and / or camera 524 may still be actively sensing. For example, device 520 may use camera 522 to detect QR codes or text codes. If device 520 detects an object (e.g., an object from a list of objects of interest) in an image captured by camera 522, device 520 may unlock (e.g., using camera 524) and / or activate display 526 to display information.

[0075] In other respects, the scene-oriented camera 522 and / or the user-oriented camera 524 may be activated by an application running on the device 520. For example, one or more operations described herein (including the scene-oriented camera 522 and / or the user-oriented camera 524 capturing images) may be performed by the device 520 in response to an application running on the device 520 initiating and / or controlling an operation.

[0076] In some aspects, display 526 may be or may include a perspective display. For example, user 530 may view scene 502 through display 526. In other aspects, display 526 may be or may include a perspective display. For example, device 520 may use scene-oriented camera 522 to capture images of scene 502 and display the captured images at display 526, so that user 530 views a representation of scene 502 (including objects therein) at display 526.

[0077] In either case, the user-facing camera 524 can capture an image of the user 530's eyes, and the device 520 can determine the user 530's gaze 532 relative to scene 502. Furthermore, the device 520 can determine which object in scene 502 the user 530 is gazing at. For example, if the display 526 includes a perspective display, the user 530 can gaze at object 504 through the display 526. The user-facing camera 524 can capture an image of the user 530's eyes, and the device 520 can determine the location within scene 502 where the user 530 is gazing. Additionally or alternatively, the device 520 can determine which object in scene 502 (e.g., among detected objects) the user 530 is gazing at (e.g., based on the relationship between objects in the scene and gaze 532).

[0078] As another example, in the case where display 526 includes a pass-through display, user 530 may gaze at a representation of an object as displayed on display 526. For example, display 526 may display real-time video data captured by scene-facing camera 522. User-facing camera 524 may capture an image of user 530's eyes, and device 520 may determine which point on display 526 user 530 is gazing at. Device 520 may determine the location in scene 502 that user 530 is gazing at based on the representation of scene 502 displayed on display 526 and based on which point on display 526 user 530 is gazing at. Additionally or alternatively, device 520 may determine which object in scene 502 user 530 is gazing at (e.g., among detected objects) (e.g., based on the location on display 526 user 530 is gazing at and based on which objects are displayed at which point on display 526).

[0079] Having detected an object (e.g., based on an image of scene 502) and determined user 530's gaze 532 (e.g., based on an image of user 530), device 520 can determine the object of interest for user 530. For example, device 520 can determine the object that user 530 is currently gazing at from among the detected objects. Device 520 can determine that the object that user 530 is currently gazing at is the object of interest for user 530.

[0080] Device 520 may determine, based on the object of interest, to display information 528 on display 526. Determining to display information 528 on display 526 may include determining the content of the information to be displayed and determining the positioning of display 526 for displaying information 528.

[0081] For example, if device 520 determines that user 530 is looking at object 504, device 520 may determine to display information 528 related to object 504. For example, if object 504 is a tree, device 520 may determine to display information about the tree, such as the tree's species, the common name of the tree's species, and / or information about the tree's maintenance.

[0082] As another example, if device 520 determines that user 530 is looking at object 506, and object 506 is a landmark (e.g., a mountain, building, statue, etc.), then device 520 may determine to display information about the landmark. For example, device 520 may determine to display the name of the landmark, the date associated with the landmark, and / or deliver text, images, and / or video information about the importance of the landmark.

[0083] As yet another example, if device 520 determines that user 530 is looking at text 510, then device 520 may determine to translate text 510 from a first language to a second language. The first and / or second language may be based on user selection. For example, the user may be visiting a foreign country and may want all text to be translated into the user's language. As another example, the user may be learning a language and may want to see text in the user's language translated into the language the user is learning.

[0084] As another example, if device 520 determines that user 530 is looking at QR code 512, device 520 can display information based on QR code 512. For example, if QR code 512 is associated with a URL, device 520 can display that URL.

[0085] As another example, if device 520 determines that user 530 is looking at URL 516, device 520 can display information based on URL 516. For example, URL 516 may be associated with a website, and device 520 may obtain information from the website (e.g., a descriptor or image of a landing page). Device 520 can then display this information to user 530.

[0086] In some aspects, device 520 may include a memory and may store information in that memory. Device 520 may select information 528 to be displayed from the stored information. Additionally or alternatively, device 520 may include a communication interface and may obtain information 528 from a remote source (e.g., a web server via a web search). For example, device 520 may perform an image-based search (e.g., based on an image of object 504 or object 506) to obtain information 528. As another example, device 520 may obtain information from a website (e.g., a website associated with the URL associated with QR code 512 and / or URL 516).

[0087] In some aspects, device 520 may display information 528 overlaid on the object of interest within the user 530's field of view (e.g., between the user 530's eyes and the object of interest in scene 502). For example, if the user 530 is looking at text 510, device 520 may display the translated text over text 510 within the user 530's field of view. As another example, if the user 530 is looking at object 504, device 520 may display the identifier of object 504 between the user 530's eyes and object 504 within the user 530's field of view. Additionally or alternatively, device 520 may display information 528 adjacent to the object of interest within the user 530's field of view. In some aspects, device 520 may display information 528 overlaid on the background of scene 502 (e.g., as determined using depth detection techniques such as time-of-flight, stereoscopic imaging, structured light, and / or monocular depth detection). In some aspects, device 520 may display information 528 over a visually uniform portion of the scene (e.g., as determined by analysis of images of the scene). In other aspects, device 520 may display information 528 over a portion of the scene that is not of interest (e.g., as determined by tracking the user's gaze over time).

[0088] Additionally or alternatively, device 520 may determine to stop displaying information 528 and / or determine not to display information 528 based on gaze 532. For example, if user 530 is gazing at QR code 512 in scene 502, device 520 may determine to display information 528 based on determining gaze 532 at QR code 512. User 530 may stop gazing at QR code 512; for example, user 530 may gaze at text 510 in scene 502. Device 520 may determine that user 530 is no longer gazing at QR code 512, and determine to stop displaying information 528 based on user 530 no longer gazing at QR code 512. Additionally or alternatively, device 520 may determine to display other information based on user 530 gazing at text 510.

[0089] In some aspects, device 520 may take action based on an object of interest and / or based on information 528. In some aspects, device 520 may further initiate action in response to an instruction from user 530. In some cases, information 528 displayed at display 526 may be or may include prompts for instructions to take action. For example, information 528 may include some information about an object of interest. If user 530 gazes at information 528 (e.g., for several seconds), device 520 may display additional information about the object of interest. As another example, information 528 may include information about a website. If device 520 determines that user 530 is gazing at the information, device 520 may open the website and obtain and display additional information from it.

[0090] In some aspects, device 520 can interpret a user's vocalizations, hand gestures, head movements, and / or eye movements into commands (e.g., related to taking further action). For example, device 520 may include a microphone and may capture and / or interpret the vocalizations of user 530 into commands. Additionally or alternatively, scene-facing camera 522 may capture images of user 530's hands, and device 520 may track the hands and interpret hand gestures into commands. Additionally or alternatively, device 520 may include an inertial measurement unit (IMU) and may determine and interpret the movement of user 530's head into commands. Additionally or alternatively, device 520 may determine head movements using simultaneous localization and mapping (SLAM) techniques based on images captured by scene-facing camera 522. Additionally or alternatively, user-facing camera 524 may capture images of user 530's eyes, and device 520 may track eye movements (including blinking and / or pointing) and interpret eye movements into commands.

[0091] For example, device 520 can detect text 510 in scene 502. Furthermore, device 520 can determine that gaze 532 is on text 510, and therefore text 510 is an object of interest for user 530. In response to determining that user 530 is interested in text 510, device 520 can overlay information 528 (e.g., a translation of text 510) onto text 510 within user 530's field of view for display. Device 520 can detect hand gestures of user 530 and interpret the gestures as indications that user 530 wants additional information about text 510. In response to interpreting the hand gestures, device 520 can obtain and display additional information about text 510.

[0092] As another example, device 520 can detect QR code 512 in scene 502. Furthermore, device 520 can determine that gaze 532 is on QR code 512, and therefore QR code 512 is an object of interest for user 530. In response to determining that user 530 is interested in QR code 512, device 520 can display information 528 about a website associated with QR code 512. Device 520 can detect a vocalization by user 530 and interpret the vocalization as an instruction from user 530 to device 520 to open a browser and access a website. In response to interpreting the vocalization, device 520 can open a browser to the website and display information from the website on display 526.

[0093] As another example, device 520 can detect phone number 514 in scenario 502. Furthermore, device 520 can determine that gaze 532 is on phone number 514, and therefore phone number 514 is an object of interest for user 530. In response to determining that user 530 is interested in phone number 514, device 520 can display prompts regarding whether user 530 wants to save phone number 514 as a contact, send a text message to phone number 514, or call phone number 514 to inquire about user 530. Device 520 can detect eye movements of user 530 (e.g., a portion of the gaze prompt). Device 520 can interpret the eye movements as an indication that user 530 wants to send a text message to phone number 514. In response to interpreting the eye movements, device 520 can generate a blank text message for phone number 514.

[0094] In some respects, device 520 may initiate one or more of the operations described herein in response to an instruction from user 530. For example, initially, display 526 may not be displaying information 528. If display 526 is a perspective display, display 526 may not display information 528, and if display 526 is a perspective display, display 526 may only display the image / video captured by scene-oriented camera 522 without any additional information 528. User 530 may point scene-oriented camera 522 at an object of interest (e.g., text 510) and provide an instruction (e.g., using a wake word). For example, user 530 may say, “Wake word, translate.” Device 520 may use user-oriented camera 524 to capture an image of user 530 (e.g., for facial recognition authentication). Device 520 may use scene-oriented camera 522 to capture an image of text 510. Device 520 may then translate text 510 according to the instruction.

[0095] Figure 6 This is an illustration of an example environment 600 in which an example device 606, according to various aspects of this disclosure, can display information based on the gaze of a user 618. Generally, device 606 may use one or more scene-facing cameras 612 to capture images of scene 602. Device 606 may detect objects (e.g., object 604) in the image of scene 602. Furthermore, device 606 may use one or more user-facing cameras 614 to capture images of the eyes of user 618. Device 606 may track the gaze 624 of user 618. Device 606 may determine the object of interest of user 618 based on the gaze 624 of user 618. Device 606 may then determine the information to be displayed at display 608 based on the object of interest. Device 606 may be... Figure 5 Example of device 520. Device 606 can be a handheld device, for example, device 606 can be... Figure 3Example of a handheld device 302.

[0096] Device 606 may be located within the field of view 620 of user 618. Device 606 may occupy a portion 622 of the field of view 620 of user 618, for example, device 606 may be located between user 618 and scene 602, and block a portion 622 of the field of view 620 of user 618 in scene 602.

[0097] The scene-oriented camera 612 may have a field of view 616 of scene 602. The device 606 may display an image at a display 608 based on an image captured by the scene-oriented camera 612. For example, the device 606 may display an image of the field of view 616 captured by the scene-oriented camera 612.

[0098] The field of view 616 of the scene-oriented camera 612 (and / or the image displayed on the display 608) may differ from a portion 622 of the field of view 620. Some XR devices may seek to replicate a portion 622 of the field of view 620 using the image displayed on the display 608. However, device 606 may display an image on the display 608 that does not seek to replicate a portion 622 of the field of view 620, for example, by displaying a wider field of view than portion 622. Additionally or alternatively, the scene-oriented camera 612 may be pointed in a direction different from the field of view 620 of user 618. Additionally or alternatively, the line of sight between user 618 (e.g., user 618's eyes) and representation 610 (e.g., as illustrated by gaze 624) may differ from the line of sight between user 618 (e.g., user 618's eyes) and object 604 in scene 602 (e.g., as illustrated by gaze 626).

[0099] Device 606 can detect and / or track the gaze 624 of user 618. Device 606 can determine that user 618 is gazing at a representation 610 of object 604, such as that displayed on display 608. Device 606 can determine that user 618 is interested in object 604 based on the user's gaze representation 610. Device 606 can determine the information to be displayed on display 608 based on determining that user 618 is interested in object 604 and / or based on determining that user 618 is gazing at a representation 610 of object 604, such as that displayed on display 608. After determining the display information based on object 604, device 606 can perform any and / or all of the operations described above with respect to device 520.

[0100] Figure 7 This is a block diagram illustrating an example device 700 for displaying information based on user gaze, according to various aspects of this disclosure. Device 700 includes a scene-oriented camera 702, a user-oriented camera 704, and a display 706. The scene-oriented camera 702 can be connected to... Figure 5 Scene-oriented camera 522 and / or Figure 6 The scene-oriented camera 612 is the same as, substantially similar to, and / or can perform the same or substantially the same operations as, it is similar to. The user-oriented camera 704 is similar to... Figure 5 User-facing camera 524 and / or Figure 6 The user-facing camera 614 is the same as, substantially similar to, and / or can perform the same or substantially the same operation as, it is compatible with. The display 706 is compatible with... Figure 5 The monitor 526 and / or Figure 6 The display 608 is the same as, substantially similar to, and / or can perform the same or substantially the same operations as, those of the display. The device 700 may additionally include one or more processors that can perform operations associated with the object detector 712, classifier 714, gaze detector 722, correlator 730, information acquirer 732, renderer 734, gesture determiner 740, motion determiner 742, gesture determiner 744, and motion determiner 746, each of which may be implemented in software and / or as one or more machine learning models.

[0101] Device 700 may acquire image 710 (e.g., from scene-oriented camera 702). Image 710 may be an image of the scene (e.g., a view of scene 502 as shown by...). Figure 5 Scene-oriented camera 522 and / or Figure 6 (Images captured by the scene-oriented camera 612).

[0102] Object detector 712 can detect objects in image 710. Object detector 712 can be or may include a convolutional neural network (CNN) or one or more visual transformers (e.g., having a detection architecture), such as, for example, a single-look detector (SSD), you only look once (YOLO), or a faster region convolutional neural network (faster RCNN). In some aspects, object detector 712 can be trained (e.g., through a backpropagation training process) to detect objects in an image.

[0103] In some aspects, the object detector 712 may output object location 716, which may be or may include image coordinates (e.g., bounding boxes) of pixels representing objects in the image 710. Additionally or alternatively, object location 716 may be or may include spatial coordinates (e.g., relative to the scene-facing camera 702 or relative to a reference coordinate system).

[0104] In some aspects, device 700 may include a classifier 714 that classifies objects detected by object detector 712. For example, classifier 714 may assign labels to detected objects, thereby labeling objects as people, animals, vehicles, text, landmarks, etc. In some aspects, classifier 714 may be trained (e.g., through supervised and / or unsupervised training processes) to determine the category of objects. In some aspects, classifier 714 may be included in object detector 712. In other aspects, classifier 714 may be omitted from device 700.

[0105] A gaze detector 722 can determine a gaze 724 based on an image 720 of the user's eyes (e.g., captured by a user-facing camera 704). The gaze detector 722 may be or may include a CNN classifier. The output head of such a CNN may be or may include (gaze angle, eye visibility, etc.). In some aspects, the gaze detector 722 may be trained (e.g., via a backpropagation training process) to determine the user's gaze based on an image of the user's eyes.

[0106] Gaze 724 may be or may include an indication of a user's gaze. In some aspects, gaze 724 may be represented by an angle (e.g., a gaze relative to directly in front). For example, gaze 724 may be represented by azimuth and elevation (or pitch and yaw). Additionally or alternatively, gaze 724 may be relative to image coordinates. For example, gaze 724 may relate to the coordinates of a display (e.g., display 706, through which a scene is viewed in a perspective or pass-through configuration). Additionally or alternatively, gaze 724 may be relative to the scene (e.g., via a relative or reference coordinate system).

[0107] Correlator 730 can correlate gaze 724 with an object (e.g., described by object location 716). In some aspects, correlator 730 can compare object location 716 with gaze 724 and determine the object to which gaze 724 is directed. For example, when both object location 716 and gaze 724 are relative to image coordinates of display 706, correlator 730 can correlate the object with gaze 724 based on the correspondence of image coordinates. As another example, when both object location 716 and gaze 724 are relative to a relative or reference coordinate system, correlator 730 can model a scene that includes objects in the scene and gaze 724 within the scene. In some aspects, correlator 730 can be or include a machine learning model trained to correlate objects with gazes. In some aspects, correlator 730 can determine the object of interest for the user of device 700.

[0108] In response to determining an object of interest for a user of device 700, information acquirer 732 may acquire information based on that object. In some aspects, information acquirer 732 may acquire information from local memory (e.g., included in or coupled to device 700). Additionally or alternatively, information acquirer 732 may acquire information from a remote device (e.g., a server) via a communication connection (e.g., via a communication network, such as the Internet).

[0109] Renderer 734 can render information 736 to be displayed at monitor 706. For example, renderer 734 determines the location and / or manner in which information 736 is displayed at monitor 706.

[0110] In some aspects, device 700 may include a gesture determiner 740 and / or a gesture determiner 744 capable of detecting gestures. For example, gesture determiner 740 may track a user's hand in image 710 and interpret the gesture based on hand movements. As another example, gesture determiner 744 may track a user's eyes in image 720 and interpret the gesture based on eye movements. In some aspects, gesture determiner 740 may be or may include a machine learning model trained to interpret gestures based on images of the hand. In some aspects, gesture determiner 744 may be or may include a machine learning model trained to interpret gestures based on images of the eyes. In some aspects, one or both of gesture determiner 740 and gesture determiner 744 may be omitted.

[0111] In some aspects, device 700 may include motion determiners 742 and / or 746, which may determine actions based on gestures interpreted by gesture determiners 740 and / or 744, respectively. For example, motion determiner 742 may take an action in response to a gesture (e.g., a hand gesture) determined by gesture determiner 740. For example, object detector 712 may detect a telephone number in image 710. Correlator 730 may determine that gaze 724 is at the telephone number. Information acquirer 732 may cause renderer 734 and display 706 to display information 736 including an inquiry about calling the telephone number. Motion determiner 742 may initiate a telephone call to the telephone number in response to gesture determiner 740 interpreting a hand gesture as an indication that the user wishes to call the telephone number.

[0112] Figure 8This is a flowchart illustrating a process 800 for displaying information based on a user's gaze, according to various aspects of this disclosure. One or more operations of process 800 may be performed by a computing device (or apparatus) or a component of a computing device (e.g., chipset, codec, etc.). The computing device may be a mobile device (e.g., a mobile phone), a network-connected wearable device such as a watch, an extended reality (XR) device such as a virtual reality (VR) device or an augmented reality (AR) device, a vehicle or a component or system of a vehicle, a desktop computing device, a tablet computing device, a server computer, a robotic device, and / or any other computing device with the resource capability to perform process 800. One or more operations of process 800 may be implemented as software components that execute and run on one or more processors.

[0113] At box 802, the computing device (or one or more components thereof) can detect objects in an image of the scene obtained from the first camera. For example, Figure 6 The device 606 can detect objects 604 in an image of scene 602 captured by a scene-oriented camera 612.

[0114] In some aspects, the first camera may be or may include a scene-oriented camera. In some aspects, the scene-oriented camera may be configured to capture images by default when the device is active. For example, the scene-oriented camera may be or may include a "perpetually active camera" or a "constantly active camera." In some aspects, the computing device (or one or more components thereof) may be configured to initiate an application on the device to activate the scene-oriented camera. For example, the scene-oriented camera may not be "perpetually active," but may be activated by an application.

[0115] In some aspects, a computing device (or one or more components thereof) may be configured to detect multiple objects in a scene based on an image of the scene. For example, device 606 may detect object 604 and other objects in scene 602. In some aspects, objects may be detected based on a list of objects of interest. For example, object 604 may be selected from all objects detected based on the list of objects of interest. In some aspects, the list of objects of interest may include: text; contact information; barcodes; quick response (QR) codes; and food items. For example, object 604 may be selected from multiple objects detected by device 606 based on the fact that object 604 is a tree on the list of objects of interest. In some aspects, objects of interest may be text. The text may be in a target language. For example, the text in the list of objects of interest is text in a target language. For example, text 510 may be detected based on the target language. Figure 5 Text 510. In some respects, the list of objects of interest can be user-configurable. For example, user 530 may be able to configure the list of objects of interest.

[0116] At box 804, the computing device (or one or more components thereof) may determine, based on an image of the user obtained from a second camera, a representation of an object displayed on a display. For example, device 606 may detect a gaze 624 of user 618 based on an image of user 618 captured by a user-facing camera 614. Device 606 may further determine that user 618 is looking at a representation 610 of object 604, such as that displayed on display 608.

[0117] In some aspects, the second camera may be or may include a user-facing camera. In some aspects, the second camera may capture images of the user's eyes. In some aspects, the user-facing camera may be configured to capture images by default when the device is active. For example, the user-facing camera may be or may include a "perpetually open camera" or a "constantly sensing camera." In some aspects, the computing device (or one or more components thereof) may be configured to initiate an application on the device to activate the user-facing camera. In some aspects, the computing device (or one or more components thereof) may be configured to initiate an application to activate at least one of the first or second cameras. For example, the user-facing camera may not be "perpetually open," but may be activated by an application.

[0118] At box 806, the computing device (or one or more components thereof) may display information associated with an object via the display based on determining that a user is looking at a representation of an object displayed on the display. For example, based on device 606 determining that user 618 is looking at a representation 610 of object 604, such as that displayed on display 608, device 606 may determine to display information associated with object 604 on display 608.

[0119] In some aspects, a computing device (or one or more components thereof) can determine that an object is of interest to the user by determining, based on gaze, the representation of an object displayed on a display that the user is looking at. For example, device 520 can determine that QR code 512 is of interest to user 530 by determining the location of gaze 532 at the representation of QR code 512 displayed on display 526. In some aspects, the display can be or may include a pass-through display configured to be positioned between the user and the scene within the user's field of view. For example, display 526 can be a pass-through display. In some aspects, the pass-through display can be part of an XR system and / or a handheld device. As another example, device 606 can determine that object 604 is of interest to user 618 by determining the location of gaze 624 at the representation 610 of object 604 displayed on display 608. In some aspects, the display can be or may include a handheld display configured to be positioned between the user and the scene within the user's field of view. For example, display 608 may be positioned within user 618's field of view 620 between user 618 and scene 602.

[0120] In some aspects, the computing device (or one or more components thereof) may include a first camera, a second camera, and a display. For example, device 606 may include a scene-oriented camera 612, a user-oriented camera 614, and a display 608. In some aspects, the first camera is configured to capture a field of view of the scene; the display fills a portion of the user's field of view; and the field of view of the scene differs from that portion of the user's field of view. For example, display 608 may occupy a portion 622 of the user 618's field of view 620. However, portion 622 of the field of view 620 may differ from the entire field of view 620 of the user 618. In some aspects, the line of sight between the user and an object (e.g., as illustrated by gaze 626) may differ from the line of sight between the user and a representation of the object displayed on the display (e.g., as illustrated by gaze 624). For example, the line of sight (e.g., a straight line) between the user 618's eye and object 604 may differ from the line of sight between the user 618's eye and a representation 610 of object 604 at display 608.

[0121] In some aspects, a computing device (or one or more components thereof) may be configured to initiate an action in response to determining that an object is an object of interest to a user or determining that the user is looking at an object such as one displayed on a monitor. In some aspects, the action may be object-based. Actions may be or may include: translating text, wherein the object comprises text; preparing communication, wherein the object comprises contact information and the communication is based on the contact information; requesting data from a server, wherein the object comprises a barcode or quick response (QR) code and the requested data is based on the barcode or QR code; identifying the object; and / or recording the object. For example, device 520 may be configured to translate text 510, prepare a telephone call or text message for telephone number 514, request data associated with telephone number 514 and / or URL 516 from a server, identify object 504 and / or object 506, and / or record the object.

[0122] In some respects, the action may be initiated in response to a command from the user. In other respects, the information displayed on the screen may be, or may include, prompts related to the command.

[0123] In some aspects, a computing device (or one or more components thereof) may be configured to interpret at least one of the following as instructions: a user's vocalization; a user's hand gesture; a user's head movement; or a user's eye movement. For example, device 520 may include a microphone and may capture and interpret vocalizations of device 520 as instructions. As another example, scene-oriented camera 522 may capture an image of user 530's hand and interpret the hand gesture as instructions. As yet another example, device 520 may include an inertial measurement unit (IMU) and may detect and interpret head movements as instructions. As yet another example, device 520 may use Simultaneous Localization and Mapping (SLAM) technology, where images are captured by scene-oriented camera 522 to detect head movements and interpret the head movements as instructions. As yet another example, device 520 may detect eye movements based on images captured by user-oriented camera 524 and interpret the eye movements as instructions.

[0124] In some examples, as previously noted, the methods described herein (e.g., Figure 8 The process 800 and / or other methods described herein may be performed wholly or partially by a computing device or apparatus. In one example, one or more of these methods may be performed by... Figure 1 XR system 100 and / or XR device 102, Figure 2 XR system 200 and / or XR equipment 202 and supporting equipment 204, Figure 3 XR system 300 and / or handheld device 302, Figure 4 XR system 400, Figure 5 Equipment 520 Figure 6 The device 606 or is executed by another system or device. In another example, these methods (e.g., Figure 8 One or more of the processes 800 and / or other methods described herein may be used by Figure 11 The computing device architecture 1100 shown is implemented wholly or in part. For example, it has Figure 11The computing device architecture 1100 shown may include, or be included in, components of, XR system 100, XR device 102, XR system 200, XR device 202, accessory device 204, XR system 300, handheld device 302, XR system 400, device 520, and / or device 606, and may implement the operation of process 800 and / or other processes described herein. In some cases, the computing device or apparatus may include various components such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the steps of the processes described herein. In some examples, the computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive Internet Protocol (IP) based data or other types of data.

[0125] A component capable of implementing a computing device in a circuit. For example, the component may include electronic circuitry or other electronic hardware, and / or may be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, graphics processing unit (GPU), digital signal processor (DSP), central processing unit (CPU), and / or other suitable electronic circuitry), and / or may include computer software, firmware, or any combination thereof for performing the various operations described herein, and / or may be implemented using computer software, firmware, or any combination thereof for performing the various operations described herein.

[0126] Process 800 and / or other processes described herein are illustrated as logic flowcharts, whose operations represent sequences of operations that can be implemented in hardware, computer instructions, or combinations thereof. In the context of computer instructions, each operation represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the described operation. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a particular function or implement a particular data type. The order in which the operations are described is not intended to be construed as limiting, and any number of the described operations can be combined in any order and / or in parallel to implement the process.

[0127] Additionally, process 800 and / or other processes described herein may be executed under the control of one or more computer systems configured using executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that executes jointly on one or more processors, implemented in hardware, or a combination thereof. As noted above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising multiple instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0128] As noted above, various aspects of this disclosure may utilize machine learning models or systems.

[0129] Figure 9 This is an exemplary example of a neural network 900 (e.g., a deep learning neural network) that can be used to implement machine learning-based object detection, object classification, gaze detection, image recognition (e.g., face recognition, object recognition, scene recognition, etc.), feature extraction, gaze detection, gaze prediction, feature segmentation, implicit neural representation generation, rendering, classification, authentication, and / or automation.

[0130] Input layer 902 includes input data. In an exemplary example, input layer 902 may include data representing images from scene-oriented camera 522 and / or user-oriented camera 524. Neural network 900 includes multiple hidden layers 906a, 906b to 906n. Hidden layers 906a, 906b to 906n comprise "n" hidden layers, where "n" is an integer greater than or equal to one. Multiple hidden layers can be made to include as many layers as needed for a given application. Neural network 900 also includes an output layer 904 that provides the output of the processing performed by hidden layers 906a, 906b to 906n. In an exemplary example, output layer 904 may provide object detection, object classification, gaze detection, and / or gaze prediction.

[0131] The neural network 900 may be or may include a multi-layer neural network with interconnected nodes. Each node may represent a piece of information. The information associated with these nodes is shared between different layers, and each layer retains information while processing it. In some cases, the neural network 900 may include a feedforward network, in which case there are no feedback connections in which the network's output is fed back into itself. In some cases, the neural network 900 may include a recurrent neural network, which may have loops that allow information to be carried across nodes when reading from the input.

[0132] Information can be exchanged between nodes through node-to-node interconnects between layers. Nodes in input layer 902 can activate the node set in the first hidden layer 906a. For example, as shown, each input node in input layer 902 is connected to each node in the first hidden layer 906a. Nodes in the first hidden layer 906a can transform the information of each input node by applying an activation function to the input node information. The information derived from this transformation can then be passed to nodes in the next hidden layer 906b, activating those nodes, which can then perform their own specified functions. Example functions include convolution, upsampling, data transformation, and / or any other suitable function. The output of hidden layer 906b can then activate nodes in the next hidden layer, and so on. Finally, the output of hidden layer 906n can activate one or more nodes in output layer 904, providing the output at those nodes. In some cases, although a node in neural network 900 (e.g., node 908) is shown as having multiple output lines, the node has a single output and all lines shown as outputs from the node represent the same output value.

[0133] In some cases, each node or the interconnection between nodes may have weights, which are a set of parameters derived from the training of the neural network 900. Once the neural network 900 is trained, it can be called a trained neural network, which can be used to perform one or more operations. For example, the interconnection between nodes may represent a piece of information about what the interconnected nodes have learned. This interconnection may have tunable numerical weights that can be tuned (e.g., based on the training dataset), allowing the neural network 900 to adapt to the input and learn as more and more data is processed.

[0134] The neural network 900 can be pre-trained to process features from the data in the input layer 902 using different hidden layers 906a, 906b to 906n, so as to provide an output through the output layer 904. In an example where the neural network 900 is used to identify features in an image, the neural network 900 can be trained using training data that includes both images and labels, as described above. For example, training images can be input into the network, where each training image has a label indicating features in the image (for feature segmentation machine learning systems) or a label indicating the category of activity in each image. In an example where object classification is used for illustrative purposes, the training images may include images of the number 2, in which case the label of the image may be [0 0 1 0 0 0 0 0 0 0].

[0135] In some cases, the neural network 900 can use a training process called backpropagation to adjust the weights of its nodes. As noted above, the backpropagation process can include forward pass, loss function, back pass, and weight update. For each training iteration, forward pass, loss function, back pass, and parameter update are performed. For each set of training images, this process can be repeated up to a certain number of iterations until the neural network 900 is trained well enough to accurately tune the weights of each layer.

[0136] For an example of identifying objects in an image, the forward pass may include passing a training image through a neural network 900. The weights are initially randomized before training the neural network 900. As an illustrative example, the image may include a numerical array representing the pixels of the image. Each number in the array may include a value from 0 to 255 describing the intensity of the pixel at that location in the array. In one example, the array may include a 28×28×3 numerical array with 28 rows and 28 columns of pixels and 3 color components (such as red, green, and blue, or lightness and two chroma components, etc.).

[0137] As noted above, for the first training iteration of the Neural Network 900, the output will likely include values ​​due to the weights being randomly selected during initialization without prioritizing any particular class. For example, if the output is a vector with probabilities that an object includes different classes, the probability values ​​for each class may be equal or at least very similar (e.g., for ten possible classes, each class may have a probability value of 0.1). With the initial weights, the Neural Network 900 cannot determine low-level features and therefore cannot make an accurate determination of what the object's classification might be. A loss function can be used to analyze the error in the output. Any suitable loss function can be defined, such as cross-entropy loss. Another example of a loss function includes mean squared error (MSE), which is defined as... The loss can be set to equal E. 总计 The value of .

[0138] For the first training image, the loss (or error) will be high because the actual value will be significantly different from the predicted output. The goal of training is to minimize the loss so that the predicted output matches the training labels. The Neural Network 900 performs backpropagation by determining which inputs (weights) contribute most to the network's loss and can adjust the weights to reduce and eventually minimize the loss. The derivative of the loss with respect to the weights (denoted as...) can be calculated. dL / dW ,in WThese are the weights at a specific layer, used to determine the weights that contribute the most to the network's loss. After calculating the derivative, a weight update can be performed by updating all the weights of the filter. For example, weights can be updated so that they change in the opposite direction of the gradient. A weight update can be represented as... ,in w Indicates weight, w i Let represent the initial weights, and η represent the learning rate. The learning rate can be set to any suitable value, where a high learning rate includes larger weight updates, while a lower value indicates smaller weight updates.

[0139] Neural Network 900 can include any suitable deep network. An example includes a Convolutional Neural Network (CNN), which includes an input layer and an output layer, with multiple hidden layers between them. The hidden layers of a CNN include a series of convolutional layers, non-linear layers, pooling layers (for downsampling), and fully connected layers. Neural Network 900 can include any other deep network besides CNNs, such as autoencoders, deep belief networks (DBNs), recurrent neural networks (RNNs), etc.

[0140] Figure 10 This is an exemplary example of a Convolutional Neural Network (CNN) 1000. The input layer 1002 of the CNN 1000 includes data representing an image or frame. For example, the data may include a numerical array representing pixels of an image, where each number in the array includes a value from 0 to 255 describing the pixel intensity at that location in the array. Using the previous example from above, the array may include a 28×28×3 numerical array with 28 rows and 28 columns of pixels and 3 color components (e.g., red, green, and blue, or lightness and two chroma components, etc.). The image may be passed through a convolutional hidden layer 1004, an optional non-linear activation layer, a pooling hidden layer 1006, and a fully connected layer 1008 (which may be hidden) to obtain an output at the output layer 1010. Although... Figure 10 Only one hidden layer from each hidden layer is shown in the diagram, but those skilled in the art will understand that multiple convolutional hidden layers, non-linear layers, pooling hidden layers, and / or fully connected layers may be included in a CNN 1000. As previously described, the output may indicate a single category of an object, or may include probabilities that best describe the category of an object in an image.

[0141] The first layer of a CNN 1000 can be a convolutional hidden layer 1004. The convolutional hidden layer 1004 analyzes the image data from the input layer 1002. Each node in the convolutional hidden layer 1004 is connected to a region of the input image called a receptive field (pixel). The convolutional hidden layer 1004 can be thought of as one or more filters (each filter corresponding to a different activation or feature map), where each convolutional iteration of the filter is a node or neuron in the convolutional hidden layer 1004. For example, the region of the input image covered by the filter at each convolutional iteration will be the receptive field of the filter. In an exemplary example, if the input image comprises a 28×28 array and each filter (and its corresponding receptive field) is a 5×5 array, then there will be 24×24 nodes in the convolutional hidden layer 1004. Each connection between a node and its receptive field learns weights and, in some cases, learns an overall bias, such that each node learns to analyze its specific local receptive field in the input image. Each node in the convolutional hidden layer 1004 will have the same weights and biases (called shared weights and shared biases). For example, the filter has a weight (digital) array and the same depth as the input. For the image frame example, the filter would have a depth of 3 (based on the three color components of the input image). An exemplary example of the filter array size is 5×5×3, corresponding to the size of the receptive field of a node.

[0142] The convolutional property of the convolutional hidden layer 1004 is due to the fact that each node of the convolutional layer is applied to its corresponding receptive field. For example, the filter of the convolutional hidden layer 1004 may begin at the top left corner of the input image array and may convolve around the input image. As noted above, each convolutional iteration of the filter can be considered as a node or neuron of the convolutional hidden layer 1004. In each convolutional iteration, the value of the filter is multiplied by the corresponding number of original pixel values ​​of the image (e.g., a 5×5 filter array is multiplied by a 5×5 array of input pixel values ​​at the top left corner of the input image array). The multiplications from each convolutional iteration can be summed to obtain the sum of that iteration or node. Next, the process continues at the next position in the input image based on the receptive field of the next node in the convolutional hidden layer 1004. For example, the filter may move a step size (called stride) to the next receptive field. The stride may be set to 1 or any other suitable amount. For example, if the stride is set to 1, the filter will move 1 pixel to the right in each convolutional iteration. Processing the filter at each unique location in the input volume produces a number representing the filter result at that location, thus determining a sum value for each node of the convolutional hidden layer 1004.

[0143] The mapping from the input layer to the convolutional hidden layer 1004 is called an activation map (or feature map). An activation map includes node-specific values ​​representing the filter results at each location of the input volume. Activation maps may include arrays containing various sums of values ​​produced by the filter for each iteration of the input volume. For example, if a 5×5 filter is applied to each pixel of a 28×28 input image (with a stride of 1), the activation map would consist of a 24×24 array. The convolutional hidden layer 1004 may include several activation maps to identify multiple features in the image. Figure 10 The example shown includes three activation maps. Using the three activation maps, the convolutional hidden layer 1004 can detect three different types of features, each of which is detectable across the entire image.

[0144] In some examples, nonlinear hidden layers can be applied after the convolutional hidden layer 1004. Nonlinear layers can be used to introduce nonlinearity into a system that has already computed linear operations. An exemplary example of a nonlinear layer is the Corrected Linear Unit (ReLU) layer. The ReLU layer applies the function f(x) = max(0, x) to all values ​​in the input volume, which changes all negative activations to 0. Therefore, ReLU can add nonlinearity to the CNN 1000 without affecting the receptive field of the convolutional hidden layer 1004.

[0145] A pooling hidden layer 1006 can be applied after the convolutional hidden layer 1004 (and, in use, after the non-linear hidden layer). The pooling hidden layer 1006 is used to simplify the information in the output of the convolutional hidden layer 1004. For example, the pooling hidden layer 1006 can take each activation map output from the convolutional hidden layer 1004 and use a pooling function to generate a condensed activation map (or feature map). Max pooling is an example of a function performed by the pooling hidden layer. The pooling hidden layer 1006 uses other forms of pooling functions, such as average pooling, L2 norm pooling, or other suitable pooling functions. Pooling functions (e.g., max pooling filters, L2 norm filters, or other suitable pooling filters) are applied to each activation map included in the convolutional hidden layer 1004. Figure 10 In the example shown, three pooling filters are used to convolve the three activation maps in the hidden layer 1004.

[0146] In some examples, max pooling can be used by applying a max pooling filter (e.g., of size 2×2) with a stride (e.g., equal to the dimension of the filter, such as stride 2) to the activation map output from convolutional hidden layer 1004. The output from the max pooling filter includes the maximum number in each sub-region of the filter convolution. Using a 2×2 filter as an example, each unit in the pooling layer summarizes a region of 2×2 nodes from the previous layer (each node is a value in the activation map). For example, four values ​​(nodes) in the activation map will be analyzed by the 2×2 max pooling filter at each iteration of the filter, with the maximum of the four values ​​being output as the "maximum" value. If such a max pooling filter is applied to an activation filter of 24×24 nodes from convolutional hidden layer 1004, the output from pooling hidden layer 1006 will be an array of 12×12 nodes.

[0147] In some examples, L2 norm pooling filters may also be used. L2 norm pooling filters involve calculating the square root of the sum of squares of the values ​​in a 2×2 region (or other suitable region) of the activation map (instead of calculating the maximum value as done in max pooling), and using the calculated value as the output.

[0148] Pooling functions (e.g., max pooling, L2 norm pooling, or other pooling functions) determine whether a given feature is found anywhere within a region of an image, discarding the exact location information. This can be done without affecting the results of feature detection, because once a feature has been found, its exact location is less important than its approximate location relative to other features. Max pooling (and other pooling methods) offers the benefit of having far fewer pooling features, thus reducing the number of parameters required in subsequent layers of a CNN 1000.

[0149] The final connection in the network is a fully connected layer, which connects each node from the pooling hidden layer 1006 to each output node in the output layer 1010. Using the example above, the input layer comprises 28×28 nodes encoding the pixel intensity of the input image, the convolutional hidden layer 1004 comprises 3×24×24 hidden feature nodes based on applying a 5×5 local receptive field (for filtering) to three activation maps, and the pooling hidden layer 1006 comprises a layer of 3×12×12 hidden feature nodes based on applying a max-pooling filter to a 2×2 region across each of the three feature maps. Extending this example, the output layer 1010 may comprise ten output nodes. In this example, each node of the 3×12×12 pooling hidden layer 1006 is connected to each node of the output layer 1010.

[0150] The fully connected layer 1008 takes the output of the previous pooling hidden layer 1006 (which should represent an activation map of high-level features) and determines the features most relevant to a particular class. For example, the fully connected layer 1008 can determine the high-level features most relevant to a particular class and may include weights (nodes) for the high-level features. The product between the weights of the fully connected layer 1008 and the pooling hidden layer 1006 can be computed to obtain the probabilities for different classes. For example, if the CNN 1000 is used to predict that the object in an image is a person, there will be high values ​​in the activation map representing the high-level features of a person (e.g., two legs, a face at the top of the object, two eyes at the top left and top right of the face, a nose in the middle of the face, a mouth at the bottom of the face, and / or other features common to people).

[0151] In some examples, the output from output layer 1010 may include an M-dimensional vector (M=10 in the previous example). M indicates the number of classes from which CNN 1000 must choose when classifying objects in an image. Other example outputs may also be provided. Each number in the M-dimensional vector represents the probability that an object belongs to a certain class. In an exemplary example, if the 10-dimensional output vector represents objects of ten different classes as [0 0 0.05 0.8 0 0.15 0 0 0 0], then the vector indicates a 5% probability that the image is an object of the third class (e.g., a dog), an 80% probability that the image is an object of the fourth class (e.g., a person), and a 15% probability that the image is an object of the sixth class (e.g., a kangaroo). The probability of a class can be considered as the confidence level that an object is part of that class.

[0152] Figure 11 An example computing device architecture 1100 is illustrated, illustrating example computing devices that can implement the various technologies described herein. In some examples, the computing device may include a mobile device, a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a video server, a vehicle (or a computing device within a vehicle), or other devices. For example, computing device architecture 1100 may include, implement, or be included in any of the following: Figure 1 XR system 100 and / or XR device 102, Figure 2 XR system 200 and / or XR equipment 202 and supporting equipment 204, Figure 3 XR system 300 and / or handheld device 302, Figure 4 XR system 400, Figure 5 Device 520 and / or Figure 6Device 606. Additionally or alternatively, computing device architecture 1100 may be configured to execute process 800 and / or other processes described herein.

[0153] The components of computing device architecture 1100 are shown to communicate electrically with each other using a connection 1112, such as a bus. The example computing device architecture 1100 includes a processing unit (CPU or processor) 1102 and a computing device connection 1112 that couples various computing device components, including computing device memories 1110 (such as read-only memory (ROM) 1108 and random access memory (RAM) 1106), to the processor 1102.

[0154] The computing device architecture 1100 may include a cache of high-speed memory that is directly connected to, very close to, or integrated into the processor 1102. The computing device architecture 1100 may copy data from memory 1110 and / or storage device 1114 to cache 1104 for fast access by the processor 1102. In this way, the cache can provide performance improvements by avoiding latency for the processor 1102 while waiting for data. These and other modules may control or be configured to control the processor 1102 to perform various actions. Other computing device memory 1110 may also be used. Memory 1110 may include various different types of memory with different performance characteristics. The processor 1102 may include any general-purpose processor and hardware or software services configured to control the processor 1102 (such as services 11116, 1118, and 3120 stored in storage device 1114), as well as dedicated processors in which software instructions are incorporated into the processor design. The processor 1102 may be a self-contained system containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors can be symmetric or asymmetric.

[0155] To enable user interaction with computing device architecture 1100, input device 1122 can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice input, etc. Output device 1124 can also be one or more of a variety of output mechanisms known to those skilled in the art, such as a display, projector, television, speaker equipment, etc. In some instances, multi-mode computing devices enable users to provide multiple types of input to communicate with computing device architecture 1100. Communication interface 1126 typically controls and manages user input and computing device output. There are no limitations on operation on any particular hardware arrangement, and therefore the underlying features here can be easily replaced to obtain improved hardware or firmware arrangements as they are developed.

[0156] Storage device 1114 is a non-volatile memory and may be a hard disk or other type of computer-readable medium capable of storing computer-accessible data, such as a magnetic tape cassette, flash memory card, solid-state memory device, digital multifunction disk, magnetic tape cartridge, random access memory (RAM) 1106, read-only memory (ROM) 1108, and hybrid forms thereof. Storage device 1114 may include services 1116, 1118, and 1120 for controlling processor 1102. Other hardware or software modules are envisioned. Storage device 1114 may be connected to computing device connection 1112. In one aspect, a hardware module performing a particular function may include software components stored in a computer-readable medium connected to necessary hardware components, such as processor 1102, connection 1112, output device 1124, etc., to perform that function.

[0157] With reference to a given parameter, property, or condition, the term "substantially" may mean that a person skilled in the art would understand that a given parameter, property, or condition is satisfied with a small degree of variance (such as, for example, within acceptable manufacturing tolerances). For example, depending on the specific parameter, property, or condition that is substantially satisfied, the parameter, property, or condition may be satisfied at least 90%, at least 95%, or even at least 99%.

[0158] Various aspects of this disclosure are applicable to any suitable electronic device (such as a security system, smartphone, tablet, laptop, vehicle, drone, or other device) that includes or is coupled to one or more active depth sensing systems. Although devices having or coupled to a light projector are described below, various aspects of this disclosure are applicable to devices having any number of light projectors and are therefore not limited to any particular device.

[0159] The term "device" is not limited to one or a specific number of physical objects (such as a smartphone, a controller, a processing system, etc.). As used herein, a device can be any electronic device having one or more parts that implement at least some parts of this disclosure. Although the following description and examples use the term "device" to describe various aspects of this disclosure, the term "device" is not limited to a specific configuration, type, or number of objects. Additionally, the term "system" is not limited to multiple components or specific aspects. For example, a system may be implemented on one or more printed circuit boards or other substrates and may have movable or static components. Although the following description and examples use the term "system" to describe various aspects of this disclosure, the term "system" is not limited to a specific configuration, type, or number of objects.

[0160] Specific details are provided in the foregoing description to provide a thorough understanding of the aspects and examples presented herein. However, those skilled in the art will understand that these aspects can be practiced without these specific details. For clarity, in some cases, the technology may be presented as comprising individual functional blocks, including functional blocks comprising devices, device components, steps or routines in methods embodied in software or a combination of hardware and software. Additional components may be used in addition to those shown in the figures and / or described herein. For example, circuits, systems, networks, processes and other components may be shown as components in block diagram form to avoid obscuring these aspects in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures and techniques may be shown without unnecessary detail to avoid obscuring aspects.

[0161] Various aspects described above can be presented as processes or methods, depicted as flowcharts, diagrams, data flow graphs, structure diagrams, or block diagrams. While flowcharts may describe operations as sequential processes, many operations within an operation can be executed in parallel or concurrently. Furthermore, the order of operations can be rearranged. A process terminates when its operations are completed, but it may have additional steps not included in the diagrams. Processes can correspond to methods, functions, procedures, subroutines, subroutines, etc. When a process corresponds to a function, its termination may correspond to the function returning to its calling function or the main function.

[0162] The processes and methods described in the examples above can be implemented using stored computer-executable instructions or computer-executable instructions otherwise obtainable from a computer-readable medium. Such instructions may include, for example, instructions and data that configure, cause or otherwise configure, a general-purpose computer, special-purpose computer, or processing device to perform a function or group of functions. The portion of the computer resources used may be accessible via a network. Computer-executable instructions may be, for example, binary files, intermediate format instructions (such as assembly language), firmware, source code, etc.

[0163] The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. Computer-readable media can include non-transitory media in which data can be stored and which do not include carrier waves and / or transient electronic signals propagating wirelessly or over a wired connection. Examples of non-transitory media include, but are not limited to, magnetic disks or magnetic tapes, optical storage media (such as compact discs (CDs) or digital versatile discs (DVDs)), flash memory, magnetic disks or optical disks, USB devices equipped with non-volatile memory, network storage devices, any suitable combinations thereof, etc. Computer-readable media may store code and / or machine-executable instructions thereon, which may represent procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, classes, or any combination of instructions, data structures, or program statements. Code segments can be coupled to other code segments or hardware circuitry by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, independent variables, parameters, data, etc., can be transmitted, forwarded, or sent through any suitable means, including memory sharing, message passing, token passing, network transmission, etc.

[0164] In some respects, computer-readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media explicitly exclude media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0165] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented as software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing necessary tasks may be stored in a computer-readable or machine-readable medium. A processor performs the necessary tasks. Typical examples of form factors include laptop computers, smartphones, mobile phones, tablet devices, or other small form factor personal computers, personal digital assistants, rack-mounted devices, standalone devices, etc. The functionality described herein may also be embodied in peripheral devices or interlocking cards. By further example, such functionality may also be implemented on circuit boards of different chips or different processes executed on a single device.

[0166] Instructions, media for delivering such instructions, computing resources for executing them, and other structures for supporting such computing resources are example components for providing the functionality described in this disclosure.

[0167] In the foregoing description, aspects of this application have been described with reference to their specific aspects, but those skilled in the art will recognize that this application is not limited thereto. Therefore, although illustrative aspects of this application have been described in detail herein, it is to be understood that the inventive concepts can be implemented and employed in various other ways, and the appended claims are not intended to be construed as including these variations unless limited by prior art. The various features and aspects of the applications described above can be used individually or in combination. Furthermore, aspects can be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of this specification. Therefore, the specification and drawings should be considered illustrative rather than restrictive. For illustrative purposes, the methods are described in a particular order. It should be understood that, in alternative aspects, the methods may be performed in a different order than described.

[0168] Those skilled in the art will understand that the less than (“<”) and greater than (“>”) symbols or terms used herein may be replaced with less than or equal to (“≤”) and greater than or equal to (“≥”) symbols without departing from the scope of this description.

[0169] When a component is described as being “configured” to perform certain operations, such configuration can be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., microprocessors or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0170] The phrase “coupled to” means any component that is physically connected directly or indirectly to another component, and / or any component that communicates directly or indirectly with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0171] Claim language or other languages ​​that state "at least one of" and / or "one or more of" in a set indicate that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, claim language stating "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language stating "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, A and B and C, or any repetition is information or data (e.g., A and A, B and B, C and C, A and A and B, etc.), or any other ordering, repetition, or combination of A, B, and C. The language "at least one of" and / or "one or more of" in a set does not limit the set to the items listed in the set. For example, the language of a claim stating "at least one of A and B" or "at least one of A or B" may refer to A, B, or A and B, and may additionally include items not listed in the set of A and B. The phrases "at least one" and "one or more" are used interchangeably herein.

[0172] Claims using phrases such as "at least one processor, the at least one processor being configured to," "at least one processor being configured to," "one or more processors, the one or more processors being configured to," or "one or more processors being configured to," or other languages, indicate that one or more processors (in any combination) are capable of performing associated operations. For example, a claim using the phrase "at least one processor, the at least one processor being configured to: X, Y, and Z" means that a single processor can be used to perform operations X, Y, and Z; or that multiple processors are each assigned a specific subset of tasks to perform operations X, Y, and Z, such that the multiple processors together perform X, Y, and Z; or that a group of multiple processors work together to perform operations X, Y, and Z. In another example, a claim using the phrase "at least one processor, the at least one processor being configured to: X, Y, and Z" could mean that any single processor can perform only at least one subset of operations X, Y, and Z.

[0173] When referring to one or more elements that perform functions (e.g., steps of a method), one element may perform all functions, or more than one element may jointly perform these functions. When more than one element jointly performs these functions, each function does not need to be performed by every single element (e.g., different functions may be performed by different elements), and / or each function does not need to be performed by only one element as a whole (e.g., different elements may perform different sub-functions of a function). Similarly, when referring to one or more elements configured to cause another element (e.g., a device) to perform functions, one element may be configured to cause another element to perform all functions, or more than one element may be jointly configured to cause another element to perform these functions.

[0174] When referring to an entity that performs or is configured to perform functions (e.g., steps of a method) (e.g., any entity or device described herein), the entity may be configured to cause one or more elements (individually or collectively) to perform those functions. One or more components of the entity may include at least one memory, at least one processor, at least one communication interface, another component configured to perform one or more of those functions, and / or any combination thereof. When referring to an entity that performs functions, the entity may be configured to cause one component to perform all functions, or to cause more than one component to perform those functions collectively. When the entity is configured to cause more than one component to perform those functions collectively, each function does not need to be performed by every single component (e.g., different functions may be performed by different components), and / or each function does not need to be performed by only one component as a whole (e.g., different components may perform different sub-functions of a function).

[0175] The various exemplary logic blocks, modules, circuits, and algorithm steps described in conjunction with the aspects disclosed herein can be implemented as electronic hardware, computer software, firmware, or combinations thereof. To clearly illustrate this interchangeability between hardware and software, various exemplary components, blocks, modules, circuits, and steps have been broadly described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of this application.

[0176] The techniques described herein can also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques can be implemented in any of a variety of devices, such as general-purpose computers, wireless communication devices (mobile phones), or integrated circuit devices with multiple uses, including applications in wireless communication devices (mobile phones) and other devices. Any feature described as a module or component can be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, these techniques can be implemented at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, perform one or more of the methods described above. The computer-readable data storage medium can form part of a computer program product, which may include packaging material. The computer-readable medium may include memory or data storage media, such as random access memory (RAM) (such as synchronous dynamic random access memory (SDRAM)), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or conveys program code in the form of instructions or data structures that can be accessed, read and / or executed by a computer, such as propagated signals or waves.

[0177] The program code can be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such processors can be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; however, in alternatives, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, as used herein, the term "processor" may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or means suitable for implementing the techniques described herein.

[0178] The exemplary aspects of this disclosure include: Aspect 1. An apparatus for displaying information, the apparatus comprising: a first camera; a second camera; a display; at least one memory; and at least one processor, the at least one processor being coupled to the at least one memory and configured to: detect an object in an image of a scene obtained from the first camera; detect a user's gaze relative to the scene based on an image of a user obtained from the second camera; determine that the object is an object of interest for the user based on a relationship between the user's gaze and the object; and display information associated with the object via the display based on the determination that the object is the object of interest for the user.

[0179] Aspect 2. The device according to aspect 1, wherein the first camera includes a scene-oriented camera.

[0180] Aspect 3. The device according to aspect 2, wherein the scene-oriented camera is configured to capture images by default when the device is active.

[0181] Aspect 4. The device according to aspect 2, wherein the at least one processor is further configured to initiate an application that activates the scene-oriented camera.

[0182] Aspect 5. The device according to any one of Aspects 1 to 4, wherein the second camera includes a user-facing camera.

[0183] Aspect 6. The device according to aspect 5, wherein the user-facing camera is configured to capture images by default when the device is active.

[0184] Aspect 7. The device according to aspect 5, wherein the at least one processor is further configured to initiate an application that activates the user-facing camera.

[0185] Aspect 8. The device according to any one of Aspects 1, 2 or 5, wherein the at least one processor is further configured to initiate an application that activates at least one of the first camera or the second camera.

[0186] Aspect 9. The device according to any one of Aspects 1 to 8, wherein, in order to determine that the object is the user's object of interest, the at least one processor is configured to determine, based on the gaze, that the user is gazing at the object in the scene through the display.

[0187] Aspect 10. The device according to aspect 9, wherein the display includes a perspective display configured to be positioned in the user's field of view between the user and the scene.

[0188] Aspect 11. The device according to aspect 10, wherein the perspective display is part of an extended reality (XR) system.

[0189] Aspect 12. The device according to any one of aspects 1 to 8, wherein, in order to determine that the object is the user's object of interest, the at least one processor is configured to determine, based on the gaze, a representation of the object that the user is gazing at displayed on the display.

[0190] Aspect 13. The device according to aspect 12, wherein the display includes a pass-through display configured to be positioned in the user's field of view between the user and the scene.

[0191] Aspect 14. The device according to aspect 13, wherein the transparent display is part of at least one of: an extended reality (XR) system; or a handheld device.

[0192] Aspect 15. The device according to any one of Aspects 1 to 8, wherein, in order to determine that the object is the user's object of interest, the at least one processor is configured to determine, based on the gaze, that the user is gazing at the object in the scene.

[0193] Aspect 16. The device according to any one of aspects 1 to 15, wherein the at least one processor is further configured to initiate an action in response to determining that the object is the user's object of interest.

[0194] Aspect 17. The device according to aspect 16, wherein the action is based on the object.

[0195] Aspect 18. The device according to aspect 17, wherein the action is at least one of: translating text, wherein the object includes the text; preparing communication, wherein the object includes contact information and the communication is based on the contact information; requesting data from a server, wherein the object includes a barcode or a quick response (QR) code and the requested data is based on the barcode or the QR code; identifying the object; or recording the object.

[0196] Aspect 19. The device according to any one of Aspects 16 to 18, wherein the action is further initiated in response to an instruction from the user.

[0197] Aspect 20. The device according to aspect 19, wherein the information displayed on the display includes prompts related to the instructions.

[0198] Aspect 21. The device according to aspect 20, the device further comprising interpreting at least one of the following as the instruction: the user's vocalization; the user's hand gesture; the user's head movement; or the user's eye movement.

[0199] Aspect 22. The device according to any one of aspects 1 to 21, wherein the at least one processor is further configured to detect a plurality of objects in the scene based on an image of the scene.

[0200] Aspect 23. The device according to any one of aspects 1 to 22, wherein the object is detected based on a list of objects of interest.

[0201] Aspect 24. The device according to aspect 23, wherein the list of objects of interest includes: text; contact information; barcodes; quick response (QR) codes; and food items.

[0202] Aspect 25. A method for displaying information, the method comprising: detecting an object in an image of a scene obtained from a first camera of a device; detecting a user's gaze relative to the scene based on an image of a user obtained from a second camera of the device; determining that the object is an object of interest for the user based on a relationship between the user's gaze and the object; and displaying information associated with the object via a display of the device based on the determination that the object is the object of interest for the user.

[0203] Aspect 26. The method according to aspect 25, wherein the first camera includes a scene-oriented camera.

[0204] Aspect 27. The method according to aspect 26, wherein the scene-oriented camera is configured to capture images by default when the device is active.

[0205] Aspect 28. The method according to aspect 26, the method further comprising initiating an application to activate the scene-oriented camera on the device.

[0206] Aspect 29. The method according to any one of Aspects 25 to 28, wherein the second camera includes a user-facing camera.

[0207] Aspect 30. The method according to aspect 29, wherein the user-facing camera is configured to capture images by default when the device is active.

[0208] Aspect 31. The method according to aspect 29, the method further comprising initiating an application to activate the user-facing camera on the device.

[0209] Aspect 32. The method according to any one of Aspects 25, 26 or 29, the method further comprising initiating an application to activate at least one of the first camera or the second camera.

[0210] Aspect 33. The method according to any one of Aspects 25 to 32, wherein determining that the object is the user's object of interest includes determining, based on the gaze, that the user is gazing at the object in the scene through the display.

[0211] Aspect 34. The method according to aspect 33, wherein the display includes a perspective display configured to be positioned in the user's field of view between the user and the scene.

[0212] Aspect 35. The method according to aspect 34, wherein the perspective display is part of an extended reality (XR) system.

[0213] Aspect 36. The method according to any one of Aspects 25 to 32, wherein determining that the object is the user's object of interest includes determining, based on the gaze, a representation that the user is gazing at the object displayed on the display.

[0214] Aspect 37. The method according to aspect 36, wherein the display includes a pass-through display configured to be positioned in the user's field of view between the user and the scene.

[0215] Aspect 38. The method according to aspect 37, wherein the transparent display is part of at least one of: an extended reality (XR) system; or a handheld device.

[0216] Aspect 39. The method according to any one of Aspects 25 to 32, wherein determining that the object is the user's object of interest includes determining, based on the gaze, that the user is gazing at the object in the scene.

[0217] Aspect 40. The method according to any one of aspects 25 to 39, the method further comprising initiating an action in response to determining that the object is the user's object of interest.

[0218] Aspect 41. The method according to aspect 40, wherein the action is based on the object.

[0219] Aspect 42. The method according to aspect 41, wherein the action is at least one of: translating text, wherein the object includes the text; preparing communication, wherein the object includes contact information and the communication is based on the contact information; requesting data from a server, wherein the object includes a barcode or a quick response (QR) code and the requested data is based on the barcode or the QR code; identifying the object; or recording the object.

[0220] Aspect 43. The method according to any one of aspects 40 to 42, wherein the action is initiated in further response to an instruction from the user.

[0221] Aspect 44. The method according to aspect 43, wherein the information displayed on the display includes prompts related to the instructions.

[0222] Aspect 45. The method according to aspect 44, the method further comprising interpreting at least one of the following as the instruction: the user's vocalization; the user's hand gesture; the user's head movement; or the user's eye movement.

[0223] Aspect 46. The method according to any one of aspects 25 to 45, the method further comprising detecting a plurality of objects in the scene based on an image of the scene.

[0224] Aspect 47. The method according to any one of Aspects 25 to 46, wherein the object is detected based on a list of objects of interest.

[0225] Aspect 48. The method according to aspect 47, wherein the list of objects of interest includes: text; contact information; barcodes; quick response (QR) codes; and food items.

[0226] Aspect 49. The method according to aspect 48, wherein the text is in the target language.

[0227] Aspect 50. The method according to aspect 48, wherein the list of objects of interest is user-configurable.

[0228] Aspect 51. The device according to aspect 24, wherein the text is in the target language.

[0229] Aspect 52. The device according to aspect 24, wherein the list of objects of interest is user-configurable.

[0230] Aspect 53. A non-transitory computer-readable storage medium having instructions stored thereon, the instructions causing the at least one processor, when executed, to perform any one of aspects 25 to 50.

[0231] Aspect 54. An apparatus for providing virtual content for display, the apparatus comprising one or more components for performing operations according to any one of Aspects 25 to 50.

[0232] Aspect 55. An apparatus for displaying information, the apparatus comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to: detect an object in an image of a scene obtained from a first camera; determine, based on an image of a user obtained from a second camera, a representation of the object being viewed by the user on a display; and, based on the determination that the user is viewing the object on the display, display information associated with the object via the display.

[0233] Aspect 56. The device according to aspect 55, wherein the first camera includes a scene-oriented camera.

[0234] Aspect 57. The device according to aspect 56, wherein the scene-oriented camera is configured to capture images by default when the device is active.

[0235] Aspect 58. The device according to any one of Aspects 56 or 57, wherein the at least one processor is further configured to initiate an application that activates the scene-oriented camera.

[0236] Aspect 59. The device according to any one of Aspects 55 to 58, wherein the second camera includes a user-facing camera.

[0237] Aspect 60. The device according to aspect 59, wherein the user-facing camera is configured to capture images by default when the device is active.

[0238] Aspect 61 The device according to any one of Aspects 59 or 60, wherein the at least one processor is further configured to initiate an application to activate the user-facing camera.

[0239] Aspect 62. The device according to any one of aspects 55 to 61, wherein the at least one processor is further configured to initiate an application to activate at least one of the first camera or the second camera.

[0240] Aspect 63. The device according to any one of aspects 55 to 62, wherein the device includes a handheld device.

[0241] Aspect 64. The device according to any one of aspects 55 to 63, wherein the device comprises the first camera, the second camera, and the display.

[0242] Aspect 65. The device according to any one of Aspects 55 to 64, wherein: the first camera is configured to capture the field of view of the scene; the display occupies a portion of the user's field of view; and the field of view of the scene is different from the portion of the user's field of view.

[0243] Aspect 66. The device according to any one of aspects 55 to 65, wherein the line of sight between the user and the object is different from the line of sight between the user and the representation of the object displayed by the display.

[0244] Aspect 67. The device according to any one of aspects 55 to 66, wherein the at least one processor is further configured to initiate an action in response to determining that the object is the user's object of interest.

[0245] Aspect 68. The device according to aspect 67, wherein the action is based on the object.

[0246] Aspect 69. The device according to aspect 68, wherein the action is at least one of: translating text, wherein the object includes the text; preparing communication, wherein the object includes contact information and the communication is based on the contact information; requesting data from a server, wherein the object includes a barcode or a quick response (QR) code and the requested data is based on the barcode or the QR code; identifying the object; or recording the object.

[0247] Aspect 70. The device according to any one of aspects 67 to 69, wherein the action is further initiated in response to an instruction from the user.

[0248] Aspect 71. The device according to aspect 70, wherein the information displayed on the display includes prompts related to the instructions.

[0249] Aspect 72. The device according to aspect 71, the device further comprising interpreting at least one of the following as the instruction: the user's vocalization; the user's hand gesture; the user's head movement; or the user's eye movement.

[0250] Aspect 73. The device according to any one of aspects 55 to 72, wherein the at least one processor is further configured to detect a plurality of objects in the scene based on an image of the scene.

[0251] Aspect 74. The device according to any one of aspects 55 to 73, wherein the object is detected based on a list of objects of interest.

[0252] Aspect 75. The device according to aspect 74, wherein the list of objects of interest includes: text; contact information; barcodes; quick response (QR) codes; and food items.

[0253] Aspect 76. The device according to aspect 75, wherein the text is in the target language.

[0254] Aspect 77. The device according to any one of aspects 75 or 76, wherein the list of objects of interest is user-configurable.

[0255] Aspect 78. A method for displaying information, the method comprising: detecting an object in an image of a scene obtained from a first camera of a device; detecting a user's gaze relative to the scene based on an image of a user obtained from a second camera of the device; determining that the object is an object of interest for the user based on a relationship between the user's gaze and the object; and displaying information associated with the object via a display of the device based on the determination that the object is the object of interest for the user.

[0256] Aspect 79. The method according to aspect 78, the method further comprising initiating an action in response to determining that the object is the user's object of interest.

[0257] Aspect 80. The method according to aspect 79, wherein the action is initiated in response to an instruction from the user.

[0258] Aspect 81. The method according to aspect 80, the method further comprising interpreting at least one of the following as the instruction: the user's vocalization; the user's hand gesture; the user's head movement; or the user's eye movement.

[0259] Aspect 82. A method for displaying information, the method comprising: detecting an object in an image of a scene obtained from a first camera; determining, based on an image of a user obtained from a second camera, a representation that the user is looking at the object displayed on a display; and displaying, via the display, information associated with the object based on the determination that the user is looking at the object displayed on the display.

[0260] Aspect 83. A non-transitory computer-readable storage medium having instructions stored thereon, the instructions causing the at least one processor, when executed, to perform any one of aspects 25 to 50 or 78 to 82.

[0261] Aspect 84. An apparatus for providing virtual content for display, the apparatus comprising one or more components for performing operations according to any one of aspects 25 to 50 or 78 to 82.

Claims

1. A device for displaying information, the device comprising: At least one memory; and At least one processor, the at least one processor being coupled to the at least one memory and being configured to: Detect objects in the scene image obtained from the first camera; The representation of the object that the user is looking at, displayed on the monitor, is determined based on the image of the user obtained from the second camera; as well as Based on the indication that the user is looking at the object displayed on the display, information associated with the object is displayed via the display.

2. The device of claim 1, wherein the first camera comprises a scene-oriented camera.

3. The device of claim 2, wherein the scene-oriented camera is configured to capture images by default when the device is active.

4. The device of claim 2, wherein the at least one processor is further configured to initiate an application that activates the scene-oriented camera.

5. The device of claim 1, wherein the second camera includes a user-facing camera.

6. The device of claim 5, wherein the user-facing camera is configured to capture images by default when the device is active.

7. The device of claim 5, wherein the at least one processor is further configured to initiate an application that activates the user-facing camera.

8. The device of claim 1, wherein the at least one processor is further configured to initiate an application that activates at least one of the first camera or the second camera.

9. The device according to claim 1, wherein the device includes a handheld device.

10. The device of claim 1, wherein the device comprises the first camera, the second camera, and the display.

11. The device according to claim 1, wherein: The first camera is configured to capture the field of view of the scene; The display occupies a portion of the user's field of view; and The field of view of the scene is different from a portion of the user's field of view.

12. The device of claim 1, wherein the line of sight between the user and the object is different from the line of sight between the user and the representation of the object displayed by the display.

13. The device of claim 1, wherein the at least one processor is further configured to initiate an action in response to determining that the object is an object of interest to the user.

14. The device of claim 13, wherein the action is based on the object.

15. The device of claim 14, wherein said action is at least one of the following: Translate the text, wherein the object includes the text; Prepare for communication, wherein the object includes contact information, and the communication is based on the contact information; Requesting data from a server, wherein the object includes a barcode or a Quick Response (QR) code, and the requested data is based on the barcode or the QR code; Identify the object; or Record the object.

16. The device of claim 13, wherein the action is further initiated in response to an instruction from the user.

17. The device of claim 16, wherein the information displayed on the display includes prompts related to the instructions.

18. The apparatus of claim 17, further comprising interpreting at least one of the following as the instructions: The user's voice; The user's hand gestures; The user's head movement; or The user's eye movements.

19. The device of claim 1, wherein the at least one processor is further configured to detect a plurality of objects in the scene based on an image of the scene.

20. A method for displaying information, the method comprising: Detect objects in the scene image obtained from the first camera; The representation of the object that the user is looking at, displayed on the monitor, is determined based on the image of the user obtained from the second camera; as well as Based on the indication that the user is looking at the object displayed on the display, information associated with the object is displayed via the display.

Images