Managing the position and transition state of extended content

Abstract

Techniques and systems for displaying content using an extended reality device are provided. For example, the process may include receiving content for display, the content being associated with a first object and a first transition, the first transition indicating a change to be applied to the content based on a trigger condition; outputting content for display associated with the first object; determining that the trigger condition has been met; modifying the content for presentation based on the first transition in response to the determination that the trigger condition has been met; and outputting the modified content for display.

Description

Technical Field

[0001] This application relates to content for extended reality. For example, aspects of this application relate to systems and techniques for managing the position and transition state of extended content.

Background Art

[0002] Extended reality (XR) technology can be used to present extended (e.g., virtual) content to a user and / or to combine a real-world environment from the physical world with a virtual environment to provide an XR experience to the user. The term XR can encompass virtual reality (VR), augmented reality (AR), mixed reality (MR), and the like. An XR system can enable a user to experience an XR environment by overlaying virtual content on an image of the real-world environment, which the user can view through an XR device (e.g., a head-mounted display (HMD), extended reality glasses, or other device). For example, the XR device can display the environment to the user. The environment is at least partially different from the real-world environment in which the user is located. The user can generally interactively change their view of the environment, for example, by tilting or moving an XR device (e.g., an HMD or other device).

[0003] XR systems may include a “see-through” display that allows a user to see their real-world environment based on light from the real-world environment passing through the display. In some cases, an XR system may include a “pass-through” display, which, through a digital “pass-through” display, allows a user to see their real-world environment, or a virtual environment based on their real-world environment, based on a view of the environment captured by one or more cameras and displayed on the display. A “see-through” XR system or a “pass-through” XR system may be worn by a user while they are engaged in activities in their real-world environment.

[0004] In some cases, users may be able to interact with virtual content, such as virtual text or objects, displayed within their view. These virtual objects may be presented in a way that makes them appear to interact with the environment. For example, virtual text may be displayed so that it appears to be on the walls of the environment, or a virtual ball may appear to bounce off real objects in the environment. In some XR systems, users may be able to create virtual content. As an example, a user may be able to create a virtual notification and anchor (e.g., attach) it to a real-world object. Techniques for managing and transitioning such virtual notifications or other virtual content may be useful. [Overview of the project]

[0005] The following provides a simplified overview of one or more embodiments disclosed herein. Therefore, the following overview should not be considered a broad overview of all intended embodiments, nor should it be considered to identify the main or important elements of all intended embodiments, or to define the scope associated with any particular embodiment. Accordingly, the following overview provides a simplified representation of a specific concept relating to one or more embodiments of the mechanism disclosed herein, prior to the “Modes for Carrying Out the Invention” presented below.

[0006] In one example for explanation, an extended reality device for displaying content is provided. This extended reality device includes at least one memory and at least one processor coupled to the at least one memory. The at least one processor is configured to receive content for display, the content being associated with a first object and a first transition, the first transition indicating a change to be applied to the content based on a trigger condition, output content for display associated with the first object, determine that the trigger condition has been met, modify the content for presentation based on the first transition in response to the determination that the trigger condition has been met, and output the modified content for display.

[0007] As another example, a method for displaying Extended Reality content is provided. The method includes receiving content for display, the content being associated with a first object and a first transition, the first transition indicating a change to be applied to the content based on a trigger condition; outputting content for display associated with the first object; determining that the trigger condition has been met; modifying the content for presentation based on the first transition in response to the determination that the trigger condition has been met; and outputting the modified content for display.

[0008] In another example, a non-temporary computer-readable medium storing instructions is provided, which, when an instruction is executed by at least one processor, causes at least one processor to receive content for display, the content being associated with a first object and a first transition, the first transition indicating a change to be applied to the content based on a trigger condition, output content for display associated with the first object, determine that the trigger condition has been met, and, in response to the determination that the trigger condition has been met, modify the content for presentation based on the first transition and output the modified content for display.

[0009] As another example, a device for displaying extended reality content is provided. The device includes means for receiving content, the content being associated with a first object and a first transition, the first transition indicating a change to be applied to the content based on a trigger condition; means for outputting content for display, associated with a first object; means for determining that a trigger condition has been met; means for modifying the content for presentation based on the first transition in response to the determination that the trigger condition has been met; and means for outputting the modified content for display.

[0010] In some embodiments, the device may include, or be part of, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a mobile device (e.g., a mobile phone or other mobile device), a wearable device (e.g., a network-connected watch or other wearable device), a personal computer, a laptop computer, a server computer, a television, a video gaming console, or other devices. In some embodiments, the device further includes at least one camera for capturing one or more images or video frames. For example, the device may include one camera (e.g., an RGB camera) or multiple cameras for capturing one or more videos containing one or more images and / or video frames. In some embodiments, the device includes a display for displaying one or more images, videos, notifications, or other displayable data. In some embodiments, the device includes a transmitter configured to transmit data or information to at least one device via a transmitting medium. In some embodiments, the processor includes a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), or other processing devices or components.

[0011] This summary is not intended to identify the main or essential features of the claimed subject matter, nor is it intended to be used independently to determine the scope of the claimed subject matter. The subject matter should be understood by referring to the entire specification of this patent, any or all of the drawings, and the appropriate parts of each claim.

[0012] The above, along with other features and examples, will become clearer by referring to the following specification, claims, and accompanying drawings.

[0013] An example for the purpose of explaining this application will be described in detail below with reference to the following figures. [Brief explanation of the drawing]

[0014] [Figure 1] This is a block diagram showing the architecture of an image capture and processing system according to the embodiments of this disclosure. [Figure 2] This figure shows an exemplary Extended Reality (XR) system architecture according to some aspects of the present disclosure. [Figure 3A] This figure shows examples of neural networks, illustrated by several examples. [Figure 3B] This figure shows examples of neural networks, illustrated by several examples. [Figure 3C] This figure shows examples of neural networks, illustrated by several examples. [Figure 3D] This figure shows examples of neural networks, illustrated by several examples. [Figure 4] This figure shows examples of neural networks, illustrated by several examples. [Figure 5] An exemplary view through a pass-through display of an XR system according to the embodiments of this disclosure is illustrated. [Figure 6] The fields of the notice are shown according to the nature of this disclosure. [Figure 7] This is a block diagram of the notification engine according to the aspects of this disclosure. [Figure 8] This is a flowchart illustrating the process for displaying content according to the aspects of this disclosure. [Figure 9A] This is a perspective view showing several examples of head-mounted displays (HMDs). [Figure 9B] Figure 9A shows perspective views illustrating several examples of head-mounted displays (HMDs). [Figure 10A]A perspective view showing the front of a mobile device capable of displaying XR content, by way of several examples. [Figure 10B] A perspective view showing the back of a mobile device 950, according to an aspect of the present disclosure. [Figure 11] A diagram showing an example of a system implementing a specific aspect of the present technology.

Mode for Carrying Out the Invention

[0015] Specific aspects and examples of the present disclosure are provided below. As will be apparent to those skilled in the art, some of these aspects and examples may be applied independently, and some of them may be applied in combination. In the following description, for the purpose of explanation, specific details are set forth in order to provide a thorough understanding of the subject matter of the present application. However, it will be apparent that various examples may be practiced without these specific details. The figures and the specification are not intended to be limiting.

[0016] The following description provides only examples for explanation and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description will enable those skilled in the art to implement examples for explanation. It should be understood that various changes may be made to the functions and arrangements of elements without departing from the spirit and scope of the present application as set forth in the appended claims.

[0017] Extended reality (XR) systems or devices can provide users with virtual content and / or combine the real world or physical environment with a virtual environment (composed of virtual content) to provide users with an XR experience. The real world environment may include real world objects (also called physical objects) such as people, vehicles, buildings, tables, chairs, and / or other real world or physical objects. XR systems or devices can facilitate interaction with different types of XR environments (for example, a user can use an XR system or device to interact with an XR environment). XR systems may include VR systems that facilitate interaction with virtual reality (VR) environments, AR systems that facilitate interaction with augmented reality (AR) environments, MR systems that facilitate interaction with mixed reality (MR) environments, and / or other XR systems. Examples of XR systems or devices include, in particular, head-mounted displays (HMDs) and smart glasses. In some cases, an XR system may track parts of the user (e.g., the user's hands and / or fingertips) to enable the user to interact with items in the virtual content.

[0018] AR is a technology that provides virtual or computer-generated content (referred to as AR content) through a user's view of a physical real-world scene or environment. AR content can include virtual content such as video, images, graphic content, location data (e.g., global positioning system (GPS) data or other location data), sound, any combination thereof, and / or other augmented content. AR systems or devices are designed to enhance (or augment) rather than replace a person's current perception of reality. For example, a user can view an actual stationary or moving physical object through an AR device display, but the user's visual perception of the physical object can be extended or enhanced by virtual images of the object (e.g., a real-world vehicle replaced by a virtual image of a DeLorean), AR content added to the physical object (e.g., virtual wings added to a living animal), AR content displayed with respect to the physical object (e.g., information virtual content displayed near a sign on a building, a virtual coffee cup virtually anchored (e.g., placed thereon) to a real-world table in one or more images), and / or other types of AR content. Various types of AR systems can be used for gaming, entertainment, and / or other applications.

[0019] In some cases, content for an XR system may include notification messages such as reminders, emotions, questions, warnings, advice, commands, and labels. In some cases, notifications may be represented digitally as objects, including text, images, holograms, audio tones, voices, music, and / or video, within the XR system's field of view. Content may be anchored (e.g., associated) with various objects in the environment, such as people, animals, locations, rooms, and objects. These objects may be real-world or virtual objects. In some examples, content may be user-created by the creator, and during authoring, content may be associated with objects. In some cases, it may be useful to define how content can transition associations between multiple objects.

[0020] Systems, apparatus, electronic devices, methods (also called processes), and computer-readable media (collectively referred to herein as “systems and techniques”) for managing the position and transition states of augmented content are described herein. In some cases, content may be anchored to objects, and content for display may be associated with transitions and trigger conditions. In some cases, a transition (e.g., content transition) may be a change in content, and a trigger condition may be a condition that causes a change in content. As an example, content may include a text reminder. This content may be associated with an object in either a virtual or real environment. Continuing the example, reminder content may be associated with an object such as a refrigerator, so that the content is displayed by the XR device when the refrigerator is within the XR device’s field of view.

[0021] In some cases, content can be associated with trigger conditions, such as when the content is read or when a specific action is performed (or not performed). For example, if content is triggered by reading the content, an XR device may track eye movements, and if the XR device detects eye movements consistent with reading the content, the XR device may trigger the content. Triggering content can change how the content is presented. In some cases, information about the content may change. For example, the text of the content may change after the content is triggered. In some cases, how the content is presented may change. For example, text content may, when triggered, later be presented as audio content. In some cases, transitions can associate content with another object. For example, text content associated with an object may, when triggered, be associated with the user's field of view (e.g., head-locked) so that the content remains presented within the user's field of view independently of the object (e.g., locked). Content can become more useful by transitioning it so that it is presented in various forms. For example, transitioning content to different forms can highlight content for important notifications, etc. Furthermore, controlling transitions based on triggers can allow content to adapt to behaviors such as performing (or not performing) an action, which can help make the content more relevant and useful.

[0022] In some cases, content such as notifications may include one or more transitions. To help create a more interactive and useful XR ecosystem, notifications can transition from one state to another rather than remaining static. Allowing notifications to dynamically adjust their states can make them more relevant and / or useful. For example, such dynamically adjustable states can allow notifications to respond to behavior by appearing to respond to and interact with actors. Allowing notifications to transition from one state to another can also be used to indicate when a notification may become more urgent or less urgent.

[0023] Various aspects of this application will be described with reference to the figures.

[0024] Figure 1 is a block diagram showing the architecture of the image capture and processing system 100. The image capture and processing system 100 includes various components used to capture and process images of a scene (e.g., images of scene 110). The image capture and processing system 100 can capture a single image (or photograph) and / or capture a video containing multiple images (or video frames) in a specific sequence. In some cases, the lens 115 and the image sensor 130 may be associated with the optical axis. In an example for one explanation, the photosensitive area of ​​the image sensor 130 (e.g., the photodiode) and the lens 115 can both be centered on the optical axis. The lens 115 of the image capture and processing system 100 is directed toward the scene 110 and receives light from the scene 110. The lens 115 bends the incident light from the scene toward the image sensor 130. The light received by the lens 115 passes through the aperture. In some cases, the aperture (e.g., aperture size) is controlled by one or more control mechanisms 120 and received by an image sensor 130. In some cases, the aperture may have a fixed size.

[0025] One or more control mechanisms 120 may control exposure, focus, and / or zoom based on information from the image sensor 130 and / or from the image processor 150. One or more control mechanisms 120 may include multiple mechanisms and components, for example, control mechanism 120 may include one or more exposure control mechanisms 125A, one or more focus control mechanisms 125B, and / or one or more zoom control mechanisms 125C. One or more control mechanisms 120 may also include additional control mechanisms other than those exemplified, such as control mechanisms for analog gain, flash, HDR, depth of field, and / or other image capture properties.

[0026] The focus control mechanism 125B of the control mechanism 120 can acquire the focus setting. In some examples, the focus control mechanism 125B stores the focus setting in a memory register. Based on the focus setting, the focus control mechanism 125B can adjust the position of the lens 115 relative to the position of the image sensor 130. For example, based on the focus setting, the focus control mechanism 125B can adjust the focus by moving the lens 115 closer to or further away from the image sensor 130 by acting a motor or servo (or other lens mechanism). In some cases, one or more additional lenses, such as microlenses, on each photodiode of the image sensor 130 may be included in the image capture and processing system 100, each of which bends the light received from the lens 115 toward the corresponding photodiode before it reaches the photodiode. The focus setting may be determined via contrast detection autofocus (CDAF), phase detection autofocus (PDAF), hybrid autofocus (HAF), or any combination thereof. The focus setting may be determined using the control mechanism 120, the image sensor 130, and / or the image processor 150. The focus setting may also be referred to as the image capture setting and / or image processing setting. In some cases, the lens 115 may be fixed to the image sensor, and the focus control mechanism 125B may be omitted without departing from the scope of this disclosure.

[0027] The exposure control mechanism 125A of the control mechanism 120 can acquire exposure settings. In some cases, the exposure control mechanism 125A stores the exposure settings in a memory register. Based on these exposure settings, the exposure control mechanism 125A can control the aperture size (e.g., aperture size or f / stop), the duration for which the aperture is open (e.g., exposure time or shutter speed), the duration for which the sensor collects light (e.g., exposure time or electronic shutter speed), the sensitivity of the image sensor 130 (e.g., ISO sensitivity or film sensitivity), the analog gain applied by the image sensor 130, or any combination thereof. The exposure settings may also be referred to as image capture settings and / or image processing settings.

[0028] The zoom control mechanism 125C of the control mechanism 120 can acquire the zoom setting. In some examples, the zoom control mechanism 125C stores the zoom setting in a memory register. Based on the zoom setting, the zoom control mechanism 125C can control the focal length of the lens element assembly (lens assembly) which includes lens 115 and one or more additional lenses. For example, the zoom control mechanism 125C can control the focal length of the lens assembly by operating one or more motors or servos (or other lens mechanisms) to move one or more of those lenses relative to each other. The zoom setting may also be referred to as the image capture setting and / or image processing setting. In some examples, the lens assembly may include a parfocal zoom lens or a variable-focus zoom lens. In some examples, the lens assembly may include a focusing lens (which may be lens 115) that first receives light from scene 110, and that light then passes through an afocal zoom system between the focusing lens (e.g., lens 115) and the image sensor 130 before the light reaches the image sensor 130. An afocal zoom system may include, in some cases, two positive (e.g., converging, convex) lenses with equal or similar (e.g., within a threshold difference range from each other) focal lengths, with a negative (e.g., diverging, concave) lens in between. In some cases, the zoom control mechanism 125C moves one or more of the lenses in the afocal zoom system, such as one or both of the negative and positive lenses. In some cases, the zoom control mechanism 125C can control the zoom by capturing an image from one of several image sensors (e.g., including image sensor 130) using a zoom corresponding to the zoom setting. For example, the image processing system 100 may include a wide-angle image sensor with a relatively low zoom and a telephoto image sensor with a larger zoom. In some cases, based on the selected zoom setting, the zoom control mechanism 125C can capture an image from the corresponding sensor.

[0029] The image sensor 130 includes one or more arrays of photodiodes or other photosensitive elements. Each photodiode measures the amount of light that ultimately corresponds to a particular pixel in the image produced by the image sensor 130. In some cases, different photodiodes may be covered by different filters. In some cases, different photodiodes may be covered by color filters and therefore may measure light that matches the color of the filter covering that photodiode. Various color filter arrays may be used, including Bayer color filter arrays, quad color filter arrays (also called quad Bayer color filter arrays or QFCA), and / or any other color filter arrays. For example, a Bayer color filter includes a red filter, a blue filter, and a green filter, and each pixel in the image is produced based on red light data from at least one photodiode covered by a red filter, blue light data from at least one photodiode covered by a blue filter, and green light data from at least one photodiode covered by a green filter.

[0030] Returning to Figure 1, other types of color filters may use yellow, magenta, and / or cyan (also called "emerald") color filters instead of, or in addition to, red, blue, and / or green filters. In some cases, some photodiodes may be configured to measure infrared (IR) light. In some implementations, a photodiode measuring IR light may not be covered by any filter, thus allowing the IR photodiode to measure both visible light (e.g., color) and IR light. In some examples, an IR photodiode may be covered by an IR filter, allowing IR light to pass through while blocking light from other parts of the frequency spectrum (e.g., visible light, color). Some image sensors (e.g., image sensor 130) may completely lack filters (e.g., color, IR, or any other part of the light spectrum) and instead use different photodiodes across the entire pixel array (sometimes stacked vertically). Different photodiodes across the entire pixel array may respond to light of different wavelengths by having different spectral sensitivity curves. Monochrome image sensors can also suffer from insufficient color depth due to the lack of color filters.

[0031] In some cases, the image sensor 130 may, alternatively or additionally, include an opacity mask and / or reflective mask that prevents light from reaching a particular photodiode or a particular portion of a photodiode at a particular time and / or from a particular angle. In some cases, an opacity and / or reflective mask may be used for phase-detection autofocus (PDAF). In some cases, an opacity and / or reflective mask may be used to prevent portions of the electromagnetic spectrum from reaching the photodiodes of the image sensor (e.g., IR cut filter, UV cut filter, bandpass filter, lowpass filter, highpass filter, etc.). The image sensor 130 may also include an analog gain amplifier for amplifying the analog signal output by the photodiode, and / or an analog-to-digital converter (ADC) for converting the analog signal output from the photodiode (and / or amplified by the analog gain amplifier) ​​into a digital signal. In some cases, certain components or functions considered with respect to one or more of the control mechanisms 120 may be included in the image sensor 130, either instead or additionally. The image sensor 130 may be a charge-coupled device (CCD) sensor, an electron-multiplying CCD (EMCCD) sensor, an active-pixel sensor (APS), a complementary metal-oxide semiconductor (CMOS), an N-type metal-oxide semiconductor (NMOS), a hybrid CCD / CMOS sensor (e.g., sCMOS), or any other combination thereof.

[0032] The image processor 150 may include one or more processors, such as one or more image signal processors (ISPs) (including ISP 154), one or more host processors (including host processor 152), and / or one or more of any other types of processors 1110 considered in relation to the computing system 1100 of Figure 11. The host processor 152 may be a digital signal processor (DSP) and / or other types of processors. In some implementations, the image processor 150 is a single integrated circuit or chip (e.g., referred to as a system-on-a-chip or SoC) including the host processor 152 and ISP 154. In some cases, the chip may include one or more input / output ports (e.g., input / output (I / O) ports 156), central processing units (CPUs), graphics processing units (GPUs), broadband modems (e.g., 3G, 4G, or LTE, 5G, etc.), memory, connectivity components (e.g., Bluetooth®, Global Positioning System (GPS), etc.), any combination thereof, and / or other components.I / O port 156 may include any suitable input / output port or interface in accordance with one or more protocols or specifications, such as an Inter-Integrated Circuit 2 (I2C) interface, an Inter-Integrated Circuit 3 (I3C) interface, a Serial Peripheral Interface (SPI) interface, a General Purpose Input / Output (GPIO) interface, a Mobile Industry Processor Interface (MIPI) (MIPI CSI-2 physical (PHY) layer port or interface), an Advanced High-performance Bus (AHB) bus, any combination thereof, and / or other input / output ports. In an example for one explanation, the host processor 152 may communicate with the image sensor 130 using the I2C port, and the ISP 154 may communicate with the image sensor 130 using the MIPI port.

[0033] The image processor 150 may perform several tasks, such as demosaicing, color space conversion, image frame downsampling, pixel interpolation, automatic exposure (AE) control, automatic gain control (AGC), CDAF, PDAF, automatic white balance, merging image frames to form an HDR image, image recognition, object recognition, feature recognition, input acceptance, output management, memory management, or any combination thereof. The image processor 150 may store image frames and / or processed images in random access memory (RAM) 140 / 1125, read-only memory (ROM) 145 / 1120, cache, memory unit, other storage device, or any combination thereof.

[0034] Various input / output (I / O) devices 160 may be connected to the image processor 150. The I / O devices 160 may include a display screen, keyboard, keypad, touchscreen, trackpad, touch-sensitive surface, printer, any other output device, any other input device, or any combination thereof. In some cases, captions may be input to the image processing device 105B via the physical keyboard or keypad of the I / O device 160, or via the virtual keyboard or keypad of the touchscreen of the I / O device 160. The I / O devices 160 may include one or more ports, jacks, or other connectors that enable wired connections between the image capture and processing system 100 and one or more peripheral devices, through which the image capture and processing system 100 may receive data from and / or transmit data to one or more peripheral devices. The I / O device 160 may include one or more wireless transceivers that enable wireless connectivity between the image capture and processing system 100 and one or more peripheral devices, through which the image capture and processing system 100 may receive data from and / or transmit data to one or more peripheral devices. The peripheral devices may include any of the types of I / O devices 160 considered above, and when coupled to a port, jack, wireless transceiver, or other wired and / or wireless connector, the peripheral device itself may be considered an I / O device 160.

[0035] In some cases, the image capture and processing system 100 may be a single device. In some cases, the image capture and processing system 100 may be two or more separate devices, including an image capture device 105A (e.g., a camera) and an image processing device 105B (e.g., a computing device coupled to the camera). In some implementations, the image capture device 105A and the image processing device 105B may be coupled together, for example, via one or more wires, cables, or other electrical connectors and / or wirelessly via one or more wireless transceivers. In some implementations, the image capture device 105A and the image processing device 105B may be separated from each other.

[0036] As shown in Figure 1, the vertical dashed lines divide the image capture and processing system 100 in Figure 1 into two parts, each representing an image capture device 105A and an image processing device 105B. The image capture device 105A includes a lens 115, a control mechanism 120, and an image sensor 130. The image processing device 105B includes an image processor 150 (including an ISP 154 and a host processor 152), RAM 140, ROM 145, and an I / O device 160. In some cases, certain components shown in the image capture device 105A, such as the ISP 154 and / or the host processor 152, may be included within the image capture device 105A.

[0037] The image capture and processing system 100 may include electronic devices such as mobile or fixed telephone handsets (e.g., smartphones, mobile phones, etc.), desktop computers, laptop or notebook computers, tablet computers, set-top boxes, televisions, cameras, display devices, digital media players, video gaming consoles, video streaming devices, Internet Protocol (IP) cameras, or any other suitable electronic devices. In some examples, the image capture and processing system 100 may include one or more wireless transceivers for wireless communication, such as cellular network communication, 802.11 Wi-Fi communication, wireless local area network (WLAN) communication, or any combination thereof. In some implementations, the image capture device 105A and the image processing device 105B may be different devices. For example, the image capture device 105A may include a camera device, and the image processing device 105B may include a computing device such as a mobile handset, desktop computer, or other computing device.

[0038] While the image capture and processing system 100 is shown to include certain components, those skilled in the art will understand that the image capture and processing system 100 may include more components than those shown in Figure 1. The components of the image capture and processing system 100 may include software, hardware, or one or more combinations of software and hardware. For example, in some implementations, the components of the image capture and processing system 100 may include electronic circuits or other electronic hardware that include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits), and / or may be implemented using them, and / or may include computer software, firmware, or any combination thereof for performing the various operations described herein, and / or may be implemented using them. The software and / or firmware may include one or more instructions that are stored on a computer-readable storage medium and are executable by one or more processors of the electronic devices implementing the image capture and processing system 100.

[0039] In some examples, the Extended Reality (XR) system 200 in Figure 2 may include an image capture and processing system 100, an image capture device 105A, an image processing device 105B, or a combination thereof.

[0040] Figure 2 shows an exemplary Extended Reality (XR) System 200 architecture according to several aspects of the present disclosure. The XR System 200 can run (or execute) XR applications and realize XR behavior. In some examples, the XR System 200 can perform tracking and location, mapping of the environment in the physical world (e.g., a scene), and / or placement and rendering of virtual content on a display 209 (e.g., a screen, a visible plane / area, and / or other display) as part of an XR experience. For example, the XR System 200 can generate a map of the environment in the physical world (e.g., a three-dimensional (3D) map), track the pose (e.g., location and position) of the XR System 200 relative to the environment (e.g., relative to the 3D map of the environment), place and / or anchor virtual content to a specific location(s) on the map of the environment, and render the virtual content on the display 209 so that it appears as if the virtual content is at a location in the environment corresponding to a specific location on the map of the scene where the virtual content is placed and / or anchored. The display 209 may include glass, screens, lenses, projectors, and / or other display mechanisms that allow the user to view a real-world environment, and also allow XR content to be overlaid, overlapped, blended, or displayed on top of it.

[0041] In the example for this explanation, the XR system 200 includes one or more image sensors 202, an accelerometer 204, a gyroscope 206, a memory device 207, a computing component 210, an XR engine 220, an image processing engine 224, a rendering engine 226, and a communication engine 228. Note that the components 202-228 shown in Figure 2 are non-limiting examples provided for illustrative and explanatory purposes, and other examples may include more, fewer, or different components than those shown in Figure 2. For example, in some cases, the XR system 200 may include one or more other sensors (e.g., one or more inertial measurement units (IMUs), radar, light detection and ranging (LIDAR) sensors, radio detection and ranging (RADAR) sensors, sound detection and ranging (SODAR) sensors, acoustic navigation and ranging (SONAR) sensors, audio sensors, etc.), one or more display devices, one or more other processing engines, one or more other hardware components, and / or one or more other software and / or hardware components not shown in Figure 2. Various components of the XR system 200, such as the image sensor 202, may be referred to in the singular herein, but it should be understood that the XR system 200 may include any of the components described herein (e.g., multiple image sensors 202).

[0042] The XR system 200 includes or communicates with an input device 208 (wired or wirelessly). The input device 208 may include any suitable input device, such as a touchscreen, a pen or other pointer device, a keyboard, a mouse, buttons or keys, a microphone for receiving voice commands, a gesture input device for receiving gesture commands, a video game controller, a steering wheel, a joystick, a set of buttons, a trackball, a remote control, any other input device 1145 described herein, or any combination thereof. In some cases, an image sensor 202 may capture an image that can be processed to interpret a gesture command.

[0043] The XR system 200 can also communicate (wired or wirelessly) with one or more other electronic devices. For example, the communication engine 228 may be configured to manage connections and communicate with one or more electronic devices. In some cases, the communication engine 228 may correspond to the communication interface 1140 in Figure 11.

[0044] In some implementations, one or more image sensors 202, accelerometers 204, gyroscopes 206, memory devices 207, computational components 210, XR engines 220, image processing engines 224, and rendering engines 226 may be part of the same computing device. For example, in some cases, one or more image sensors 202, accelerometers 204, gyroscopes 206, memory devices 207, computational components 210, XR engines 220, image processing engines 224, and rendering engines 226 can be integrated into HMDs, extended reality glasses, smartphones, laptops, tablet computers, gaming systems, and / or any other computing devices. However, in some implementations, one or more image sensors 202, accelerometers 204, gyroscopes 206, memory devices 207, computational components 210, XR engines 220, image processing engines 224, and rendering engines 226 may be part of two or more separate computing devices. For example, in some cases, some of the components 202-226 may be part of or implemented by one computing device, while the remaining components may be part of or implemented by one or more other computing devices.

[0045] The storage device 207 can be any storage device(s) that stores data. Furthermore, the storage device 207 can store data from any of the components of the XR system 200. For example, the storage device 207 can store data from the image sensor 202 (e.g., image or video data), data from the accelerometer 204 (e.g., measurements), data from the gyroscope 206 (e.g., measurements), data from the computational component 210 (e.g., processing parameters, preferences, virtual content, rendering content, scene maps, tracking and location data, object detection data, privacy data, XR application data, face recognition data, occlusion data, etc.), data from the XR engine 220, data from the image processing engine 224, and / or data from the rendering engine 226 (e.g., output frames). In some examples, the storage device 207 may include a buffer for storing frames for processing by the computational component 210.

[0046] One or more computational components 210 may include a central processing unit (CPU) 212, a graphics processing unit (GPU) 214, a digital signal processor (DSP) 216, an image signal processor (ISP) 218, and / or other processors (e.g., a neural processing unit (NPU) implementing one or more trained neural networks). Computational components 210 can perform a variety of operations, such as image enhancement, computer vision, graphics rendering, extended reality operations (e.g., tracking, location identification, pose estimation, mapping, content anchoring, content rendering, etc.), image and / or video processing, sensor processing, recognition (e.g., text recognition, face recognition, object recognition, feature recognition, tracking or pattern recognition, scene recognition, occlusion detection, etc.), trained machine learning operations, filtering, and / or any of the various operations described herein. In some examples, computational components 210 may implement an XR engine 220, an image processing engine 224, and a rendering engine 226 (e.g., control, operation, etc.). In other examples, the computational component 210 may also implement one or more other processing engines.

[0047] The image sensor 202 may include any image and / or video sensor or capture device. In some examples, the image sensor 202 may be part of a multi-camera assembly, such as a dual-camera assembly. The image sensor 202 can capture image and / or video content (e.g., raw image and / or video data), which can then be processed by the computational component 210, the XR engine 220, the image processing engine 224, and / or the rendering engine 226, as described herein. In some examples, the image sensor 202 may include the image capture and processing system 100, the image capture device 105A, the image processing device 105B, or a combination thereof.

[0048] In some examples, the image sensor 202 can capture image data, generate images (also called frames) based on the image data, and / or provide the image data or frames to the XR engine 220, image processing engine 224, and / or rendering engine 226 for processing. The images or frames may include video frames of a video sequence or still images. The images or frames may include pixel arrays representing a scene. For example, the images may be red-green-blue (RGB) images having red, green, and blue color components per pixel, luminance, red color difference, blue color difference (YCbCr) images having a luminance component and two color difference (red color difference) components (red color difference and blue color difference) per pixel, or any other preferred type of color or monochrome image.

[0049] In some cases, the image sensor 202 (and / or other cameras in the XR system 200) can also be configured to capture depth information. For example, in some implementations, the image sensor 202 (and / or other cameras) can include an RGB depth (RGB-depth, RGB-D) camera. In some cases, the XR system 200 can include one or more depth sensors (not shown) that are separate from the image sensor 202 (and / or other cameras) and can capture depth information. For example, such depth sensors can acquire depth information independently of the image sensor 202. In some examples, the depth sensors may be physically installed in the same general location as the image sensor 202 but may operate at a different frequency or frame rate than the image sensor 202. In some examples, the depth sensor may take the form of a light source that can project a structured or textured light pattern, which may contain one or more narrowband lights, onto one or more objects in the scene. Depth information can then be acquired by taking advantage of the geometric distortion of the projected pattern caused by the surface shape of the objects. For example, depth information can be obtained from a stereo sensor, such as an infrared structured light projector and a combination of a camera (e.g., an RGB camera) and an infrared camera aligned with it.

[0050] The XR system 200 may also include other sensors within its one or more sensors. These one or more sensors may include one or more accelerometers (e.g., accelerometer 204), one or more gyroscopes (e.g., gyroscope 206), and / or other sensors. These one or more sensors can provide velocity, orientation, and / or other position-related information to the computational component 210. For example, accelerometer 204 can detect acceleration by the XR system 200 and generate acceleration measurements based on the detected acceleration. In some cases, accelerometer 204 may provide one or more translation vectors (e.g., up / down, left / right, forward / backward) that can be used to determine the position or pose of the XR system 200. Gyroscope 206 can detect and measure the orientation and angular velocity of the XR system 200. For example, gyroscope 206 can be used to measure the pitch, roll, and yaw of the XR system 200. In some cases, gyroscope 206 may provide one or more rotation vectors (e.g., pitch, yaw, roll). In some examples, the image sensor 202 and / or the XR engine 220 can use measurements obtained by the accelerometer 204 (e.g., one or more translation vectors) and / or the gyroscope 206 (e.g., one or more rotation vectors) to calculate the pose of the XR system 200. As mentioned above, in other examples, the XR system 200 may also include other sensors such as an inertial measurement unit (IMU), magnetometer, gaze and / or eye-tracking sensor, machine vision sensor, smart scene sensor, speech recognition sensor, collision sensor, shock sensor, position sensor, and tilt sensor.

[0051] As described above, in some cases, one or more sensors may include at least one IMU. An IMU is an electronic device that uses a combination of one or more accelerometers, one or more gyroscopes, and / or one or more magnetometers to measure specific forces, angular velocities, and / or orientations of the XR system 200. In some examples, one or more sensors may output measured information associated with capturing images captured by the image sensor 202 (and / or other cameras of the XR system 200), and / or depth information acquired using one or more depth sensors of the XR system 200.

[0052] The outputs of one or more sensors (e.g., accelerometer 204, gyroscope 206, one or more IMUs, and / or other sensors) may be used by the XR engine 220 to determine the pose of the XR system 200 (also called the head pose) and / or the pose of the image sensor 202 (or other cameras of the XR system 200). In some cases, the pose of the XR system 200 and the pose of the image sensor 202 (or other cameras) may be the same. The pose of the image sensor 202 refers to the position and orientation of the image sensor 202 relative to a reference frame (e.g., relative to scene 110). In some implementations, the camera pose can be determined in 6 degrees of freedom (6DoF), which refer to three translational components (e.g., given by X (horizontal), Y (vertical), and Z (depth) coordinates relative to a reference frame such as an image plane) and three angular components (e.g., roll, pitch, and yaw relative to the same reference frame). In some implementations, camera pose may be determined with respect to three degrees of freedom (3DoF), which refer to three angular components (e.g., roll, pitch, and yaw).

[0053] In some cases, a device tracker (not shown) can track the pose (e.g., 6DoF pose) of the XR system 200 using measurements from one or more sensors and image data from the image sensor 202. For example, the device tracker can fuse visual data from image data (e.g., using a visual tracking solution) with inertial data from measurements to determine the position and motion of the XR system 200 relative to the physical world (e.g., the scene) and a map of the physical world. In some examples, as described below, when tracking the pose of the XR system 200, the device tracker can generate a three-dimensional (3D) map of the scene (e.g., the real world) and / or generate updates to the 3D map of the scene. The 3D map update may include, for example, new or updated features and / or features or landmark points associated with the scene and / or the 3D map of the scene, location-specific updates that identify or update the position of the XR system 200 in the scene and the 3D map of the scene. The 3D map can provide a digital representation of the scene in the real world / physical world. In some examples, a 3D map can anchor location-based objects and / or content to real-world coordinates and / or objects. The XR system 200 can use the mapped scene (for example, a scene of the physical world represented by and / or associated with a 3D map) to merge the physical world and the virtual world, and / or merge virtual content or objects with the physical environment.

[0054] In some embodiments, the pose of the image sensor 202 and / or the entire XR system 200 can be determined and / or tracked by the computational component 210 using a visual tracking solution based on images captured by the image sensor 202 (and / or other cameras of the XR system 200). For example, in some examples, the computational component 210 can perform tracking using computer vision-based tracking, model-based tracking, and / or simultaneous location identification and mapping (SLAM) techniques. For example, the computational component 210 can perform SLAM or communicate (wired or wirelessly) with a SLAM system (not shown). SLAM refers to a class of techniques that create a map of the environment (e.g., a map of the environment modeled by the XR system 200) and simultaneously track the poses of the cameras (e.g., image sensor 202) and / or the XR system 200 against that map. The map may be called a SLAM map and may be three-dimensional (3D). The SLAM technique can be performed using color or grayscale image data captured by the image sensor 202 (and / or other cameras of the XR system 200) and can be used to generate estimates of 6DoF pose measurements from the image sensor 202 and / or the XR system 200. Such a SLAM technique configured to perform 6DoF tracking can be called 6DoF SLAM. In some cases, the output of one or more sensors (e.g., accelerometer 204, gyroscope 206, one or more IMUs, and / or other sensors) can be used to estimate, correct, and / or adjust the estimated pose.

[0055] In some cases, 6DoF SLAM (e.g., 6DoF tracking) can associate features observed from specific input images from image sensor 202 (and / or other cameras) with a SLAM map. For example, 6DoF SLAM can use the association of feature points from the input images to determine the pose (position and orientation) of image sensor 202 and / or XR system 200 relative to the input images. 6DoF mapping can also be performed to update the SLAM map. In some cases, a SLAM map maintained using 6DoF SLAM may include 3D feature points triangulated from two or more images. For example, keyframes can be selected from an input image or video stream to represent an observed scene. For all keyframes, the respective 6DoF camera pose associated with the image can be determined. The pose of image sensor 202 and / or XR system 200 can be determined by projecting features from the 3D SLAM map onto the image or video frame and updating the camera pose from the validated 2D-3D correspondence.

[0056] In one example for illustrative purposes, the computational component 210 can extract feature points from a specific input image (e.g., all input images, a subset of input images, etc.) or from each keyframe. As used herein, feature points (also called alignment points) are specific or identifiable parts of an image, such as a part of a hand or the edge of a table. Features extracted from captured images can represent specific feature points along a three-dimensional space (e.g., coordinates on the X, Y, and Z axes), and all feature points can have associated feature locations. Feature points in a keyframe may match (be the same as or correspond to) feature points in a previously captured input image or keyframe, or they may fail to match. Feature detection can be used to detect feature points. Feature detection may include image processing operations used to examine one or more pixels in an image to determine whether a feature exists in a particular pixel. Feature detection can be used to process the entire captured image or a specific part of an image. For each image or keyframe, once a feature is detected, a local image patch near the feature can be extracted.Features can be extracted using any suitable technique, such as Scale Invariant Feature Transform (SIFT) (which localizes features and generates descriptions of them), Learned Invariant Feature Transform (LIFT), Speed ​​Up Robust Features (SURF), Gradient Location-Orientation Histogram (GLOH), Oriented Fast and Rotated Brief (ORB), Binary Robust Invariant Scalable Keypoints (BRISK), Fast Retina Keypoint (FREAK), KAZE, Accelerated KAZE (AKAZE), Normalized Cross Correlation (NCC), descriptor matching, another suitable technique, or a combination thereof.

[0057] As an example for one explanation, the computational component 210 can extract feature points corresponding to a mobile device, for example. In some cases, feature points corresponding to a mobile device may be tracked to determine the mobile device's pose. As will be explained in more detail below, the mobile device's pose may be used to determine the location for projecting AR media content that can enhance the media content displayed on the mobile device's screen.

[0058] In some cases, the XR system 200 may also track the user's hands and / or fingers to allow the user to interact with and / or control virtual content within the virtual environment. For example, the XR system 200 may track the pose and / or movement of the user's hands and / or fingertips to identify or translate user interactions with the virtual environment. User interactions may include, but are not limited to, moving items in virtual content, resizing items in virtual content, selecting input interface elements in a virtual user interface (e.g., a virtual representation of a mobile phone, a virtual keyboard, and / or other virtual interfaces), and providing input through a virtual user interface.

[0059] A neural network is an example of a machine learning system, and a neural network can include an input layer, one or more hidden layers, and an output layer. Data is provided from the input nodes of the input layer, processed by the hidden nodes of one or more hidden layers, and an output is produced through the output nodes of the output layer. Deep learning networks typically include multiple hidden layers. Each layer of a neural network can include a feature map or activation map, which can include artificial neurons (or nodes). Feature maps can include filters, kernels, etc. Nodes can include one or more weights used to indicate the importance of one or more nodes in the layer. In some cases, a deep learning network can have a series of many hidden layers, with early layers used to determine simple and low-level properties of the input, and later layers building a hierarchy of more complex and abstract properties.

[0060] Deep learning architectures can learn feature hierarchies. When visual data is presented, for example, the first layer might learn to recognize relatively simple features such as edges in the input stream. In another example, when auditory data is presented, the first layer might learn to recognize spectral power at specific frequencies. A second layer, taking the output of the first layer as input, might learn to recognize combinations of features such as simple shapes in the case of visual data, or combinations of sounds in the case of auditory data. For example, higher layers might learn to represent complex shapes in visual data or words in auditory data. Even higher layers might learn to recognize common visual objects or spoken phrases.

[0061] Deep learning architectures can perform particularly well when applied to problems with natural hierarchical structures. For example, classifying electric vehicles might benefit from first learning to recognize wheels, windshields, and other features. These features can then be combined in different ways in higher layers to recognize passenger cars, trucks, and airplanes.

[0062] Neural networks can be designed using various connectivity patterns. In a feedforward network, each neuron in a given layer interacts with neurons in higher layers, passing information from lower to higher layers. As mentioned above, a hierarchical representation can be constructed within the consecutive layers of a feedforward network. Neural networks can also have recursive or feedback (also called top-down) connections. In recursive connections, the output from a neuron in a given layer can be transmitted to another neuron in the same layer. Recursive architectures can be useful when recognizing patterns across two or more chunks of input data delivered to the neural network in a sequence. The connection from a neuron in a given layer to a neuron in a lower layer is called a feedback (or top-down) connection. Networks with many feedback connections can be useful when the recognition of a high-level concept can help identify specific low-level features of the input. The connections between layers in a neural network can be fully connected or locally connected. Various examples of neural network architectures are described below with respect to Figures 3A to 34.

[0063] Neural networks can be designed using various connectivity patterns. In a feedforward network, each neuron in a given layer interacts with neurons in higher layers, passing information from lower to higher layers. As mentioned above, a hierarchical representation can be constructed within the consecutive layers of a feedforward network. Neural networks can also have recursive or feedback (also called top-down) connections. In recursive connections, the output from a neuron in a given layer can be passed to another neuron in the same layer. Recursive architectures can be useful when recognizing patterns across two or more chunks of input data delivered to the neural network in a sequence. The connection from a neuron in a given layer to a neuron in a lower layer is called a feedback (or top-down) connection. Networks with many feedback connections can be useful when the recognition of a high-level concept can help identify specific low-level features of the input.

[0064] The connections between layers of a neural network can be fully connected or locally connected. Figure 3A shows an example of a fully connected neural network 302. In the fully connected neural network 302, neurons in the first layer can transmit their output to all neurons in the second layer, and as a result, each neuron in the second layer receives input from all neurons in the first layer. Figure 3B shows an example of a locally connected neural network 304. In the locally connected neural network 304, neurons in the first layer can be connected to a limited number of neurons in the second layer. More generally, the locally connected layers of the locally connected neural network 304 may be configured such that each neuron in the layer has the same or similar connectivity pattern but can have different connection strengths (e.g., 310, 312, 314, and 316). Because neurons in higher layers of a given region can receive inputs that are tuned through training for the characteristics of a limited portion of all inputs to the network, the connectivity patterns of local connections can create spatially distinct receptive fields within the higher layers.

[0065] An example of a locally connected neural network is a convolutional neural network. Figure 3C shows an example of a convolutional neural network 306. The convolutional neural network 306 may be configured such that the connection strength (e.g., 308) associated with the input for each neuron in the second layer is shared. Convolutional neural networks may be suitable for problems where the spatial location of the input is meaningful. According to aspects of this disclosure, the convolutional neural network 306 may be used to perform one or more aspects of video compression and / or restoration.

[0066] One type of convolutional neural network is the deep convolutional network (DCN). Figure 3D shows a detailed example of a DCN300 designed to recognize visual features from an image 326 input from an image capture device 330, such as an in-vehicle camera. The DCN300 in this example can be trained to identify traffic signs and the numbers written on them. Of course, the DCN300 can be trained for other tasks, such as identifying lane markings or traffic signals.

[0067] DCN300 can be trained using supervised learning. During training, DCN300 may be presented with images such as image 326 of a speed limit sign, and then a forward pass may be computed to produce output 322. DCN300 may include a feature extraction section and a classification section. Upon receiving image 326, the convolutional layer 332 may apply a convolutional kernel (not shown) to image 326 to produce a first set of feature maps 318. As an example, the convolutional kernel for convolutional layer 332 may be a 5x5 kernel that produces a 28x28 feature map. In this example, four different convolutional kernels were applied to image 326 in convolutional layer 332, since four different feature maps are produced in the first set of feature maps 318. Convolutional kernels are also sometimes called filters or convolutional filters.

[0068] A first set of feature maps 318 may be subsampled by a max pooling layer (not shown) to generate a second set of feature maps 320. The max pooling layer reduces the size of the first set of feature maps 318; that is, the size of the second set of feature maps 320, such as 14×14, is smaller than the size of the first set of feature maps 318, such as 28×28. The reduced size provides similar information to subsequent layers while reducing memory consumption. The second set of feature maps 320 may be further convolved through one or more subsequent convolutional layers (not shown) to generate one or more subsequent sets of feature maps (not shown).

[0069] In the example in Figure 3D, a second set of feature maps 320 is convolved to generate a first feature vector 324. Furthermore, the first feature vector 324 is further convolved to generate a second feature vector 328. Each feature in the second feature vector 328 may contain a digit corresponding to a possible feature of image 326, such as "label", "60", and "100". A softmax function (not shown) can be used to convert the digits in the second feature vector 328 into probabilities. Thus, the output 322 of DCN300 is the probability that image 326 contains one or more features.

[0070] In this example, the probabilities in output 322 for "label" and "60" are higher than the probabilities for other outputs in output 322 such as "30", "40", "50", "70", "80", "90", and "100". Before training, the outputs 322 generated by DCN300 may be inaccurate. Therefore, an error can be calculated between output 322 and the target output. The target output is the ground truth of image 326 (e.g., "label" and "60"). The weights of DCN300 can then be adjusted so that the output 322 of DCN300 approaches the target output.

[0071] To adjust the weights, a learning algorithm may compute a gradient vector for the weights. The gradient may indicate the amount by which the error will increase or decrease if the weights are adjusted. In the top layer, the gradient may directly correspond to the weight values ​​connecting the activated neurons in the second-to-last layer to the neurons in the output layer. In lower layers, the gradient may depend on the weight values ​​and the computed error gradients from the upper layers. The weights can then be adjusted to minimize the error. This method of weight adjustment is sometimes called "backpropagation" because it involves a "backward path" through the neural network.

[0072] In practice, the error gradient of the weights can be calculated over a small number of examples so that the calculated gradient approximates the true error gradient. This approximation method is sometimes called stochastic gradient descent. Stochastic gradient descent may be repeated until the achievable error rate for the entire system stops decreasing or until the error rate reaches a target level. After training, the DCN may be presented with new images, and a forward passthrough through the network may produce an output 322 that can be considered the DCN's inference or prediction.

[0073] Deep belief networks (DBNs) are probabilistic models with multiple layers of hidden nodes. DBNs can be used to extract hierarchical representations of training datasets. DBNs can be obtained by stacking layers of restricted Boltzmann machines (RBMs). RBMs are a type of artificial neural network that can learn probability distributions over a set of inputs. Because RBMs can learn probability distributions without information about which class each input should belong to, they are frequently used in unsupervised learning. Using a hybrid paradigm of unsupervised and supervised learning, the lower RBM of a DBN can be trained unsupervised and can function as a feature extractor, while the upper RBM can be trained supervised (on a combined distribution of inputs from previous layers and target classes) and can function as a classifier.

[0074] Deep convolutional networks (DCNs) are networks of convolutional networks that are constructed with additional pooling and normalization layers. DCNs have achieved state-of-the-art performance for a wide range of tasks. DCNs can be trained using supervised learning, where both the input and output targets are known in a large number of cases, and the network weights are modified using gradient descent.

[0075] A DCN can be a feedforward network. In addition, as mentioned above, connections from neurons in the first layer of a DCN to groups of neurons in the next higher layer are shared across the neurons in the first layer. The feedforward and shared connections of a DCN can be leveraged for high-speed processing. The computational burden of a DCN can be significantly less than that of a similarly sized neural network, for example, one that includes recursive or feedback connections.

[0076] The processing of each layer of a convolutional network can be considered as a spatially invariant template or basis projection. If the input is initially decomposed into multiple channels, such as the red, green, and blue channels of a color image, the convolutional network trained on that input can be considered three-dimensional, having two spatial dimensions along the image axes and a third dimension that captures color information. The output of the convolutional connections can be considered as forming a feature map in subsequent layers, where each element of the feature map (e.g., feature map 320) receives input from a range of neurons in the previous layer (e.g., feature map 318) and from each of the multiple channels. The values ​​in the feature map can be further processed using rectification, max(0,x), or other nonlinearities. Values ​​from neighboring neurons can further undergo pooling, which corresponds to downsampling and can provide additional local invariance and dimensionality reduction.

[0077] Figure 4 is a block diagram showing an example of a deep convolutional network 450. The deep convolutional network 450 may include several different types of layers based on connectivity and weight sharing. As shown in Figure 4, the deep convolutional network 450 includes convolutional blocks 454A and 454B. Each of the convolutional blocks 454A and 454B may consist of a convolutional layer (CONV) 456, a normalization layer (LNorm) 458, and a maximum pooling layer (MAX POOL) 460.

[0078] The convolutional layer 456 may include one or more convolutional filters that can be applied to the input data 452 to generate a feature map. Although only two convolutional blocks 454A, 454B are shown, this disclosure is not limited in that way, and instead, any number of convolutional blocks (e.g., convolutional blocks 454A, 454B) may be included in the deep convolutional network 450 according to design preferences. A normalization layer 458 may normalize the output of the convolutional filters. For example, the normalization layer 458 may provide whitening or side suppression. A max pooling layer 460 may provide downsampling aggregation across space for local invariance and dimensionality reduction.

[0079] For example, the parallel filter bank of the deep convolutional network may be loaded onto the CPU 212 or GPU 214 of the computing component 210 to achieve high performance and low power consumption. In an alternative embodiment, the parallel filter bank may be loaded onto the DSP 216 or ISP 218 of the computing component 210. In addition, the deep convolutional network 450 may have access to other processing blocks that may reside on the computing component 210, such as sensor processors and navigation modules, which are dedicated to sensors and navigation, respectively.

[0080] The deep convolutional network 450 may also include one or more fully connected layers, such as layer 462A (labeled "FC1") and layer 462B (labeled "FC2"). The deep convolutional network 450 may further include a logistic regression (LR) layer 464. Between each layer 456, 458, 460, 462A, 462B, and 464 of the deep convolutional network 450 are weights (not shown) that will be updated. The output of each layer (e.g., 456, 458, 460, 462A, 462B, 464) can serve as input to one of the subsequent layers (e.g., 456, 458, 460, 462A, 462B, 464) in the deep convolutional network 450 to learn a hierarchical feature representation from the input data 452 (e.g., images, audio, video, sensor data, and / or other input data) supplied in the first convolutional block 454A. The output of the deep convolutional network 450 is a classification score 466 for the input data 452. The classification score 466 can be a set of probabilities, where each probability is the probability of the input data containing the feature from the set of features.

[0081] In some cases, XR systems can enable the authoring of virtual content that may be displayed by the XR system. For example, virtual content may be presented within the XR environment as text, images, holograms, audio tones, voices, music, video, etc. (e.g., visible within the field of view of the XR display), and may include notifications such as reminders, emotions, questions, warnings, advice, commands, and labels.

[0082] Figure 5 shows an exemplary view 500 through a pass-through display of an XR system according to an aspect of the present disclosure. View 500 includes objects in an environment visible through the display, in this case a refrigerator 502 and a dog 504. Objects may include things such as avatars, people, doors, houses, and trees. In some cases, objects may also include conceptual objects such as a definable environment, location, or space, such as a kitchen, a lobby, an intersection of two roads, or a user's view. In some cases, objects may have characteristics. For example, an object may be characterized as being associated with a particular activity or behavior (e.g., moving, being able to be opened) or a condition (e.g., above a certain temperature, unlocked, being of a certain age). In some cases, a view such as view 500 may also include virtual objects such as notifications. In some cases, a notification may include a visible and / or audible message or message instruction presented by the XR system, and the notification may be presented digitally by the XR system as an object including text, images, holograms, audio tones, voices, music, and / or video within the field of view of the XR system. View 500 includes two notifications, a speech bubble 506, and a notification icon 508.

[0083] In some cases, notifications can be anchored to (e.g., associated with, attached to, or linked to) an object. This object could be a virtual object within an XR environment or an object in a real-world environment. For example, a callout 506 could be anchored to a refrigerator 502, which is a real object in a real-world environment. In some examples, notifications can also be anchored to a specific environment or to a conceptual object such as a view 500, as well as a notification icon 508.

[0084] In some examples, notifications can have a lifecycle. For instance, a notification may be created, transition, and complete. Objects that influence the lifecycle of a notification may be called actors. In some examples, the author who creates a notification may also be an actor to the notification. In some cases, a notification may transition from one state (e.g., a stable state) to another. For example, a first actor may author a notification such as callout 506. In some examples, during creation, a notification may be dynamically anchored based on, for example, the context in which the notification is being authored and the actor performing the authoring.

[0085] In some cases, a notification may contain transition information that defines how and under what conditions the notification may transition to another state. For example, bubble 506 may contain transition information indicating that it may transition from a state where bubble 506 is anchored to refrigerator 502 to a state where it is anchored to view 500 of an actor who may or may not be the creator of bubble 506. In addition, the content of bubble 506, such as a message to walk the dog, may be presented at a specific time in another type of media, such as an audio format. In some cases, transitions may be defined dynamically, for example, based on the context of the notification, the actor performing the authoring, future actors for the notification, etc. In some cases, animation may be displayed to indicate that the notification is transitioning from one state to another. For example, when bubble 506 transitions from a state where it is anchored to refrigerator 502 to a state where it is anchored to view 500 of an actor, bubble 506 may appear to shrink and become a glowing spot that flows into the actor's notification area.

[0086] In some cases, notifications may be automatically rejected based on one or more rejection criteria. In some cases, whether the rejection criteria are met may be determined, for example, based on behavior, objects, and / or scene analysis. In some cases, the rejection criteria may be included in the notification, and the rejection criteria may be defined dynamically.

[0087] Figure 6 shows the fields of Notice 600 as described in this disclosure. In some cases, Notice 600 may include a set of fields or elements that define what is presented for the notice and how the notice works. The fields shown for Notice 600 are illustrative, and it should be understood that other examples may include fewer files or additional fields. Notice 600 includes a message field, an author field, target actor and target actor action fields, anchor and anchor policy fields, initial media type and initial media policy fields, one or more transition events, transition action and transition media type fields, and a completion policy field. The message field may include information that should be presented by Notice 600 for a notice that includes audio, visual, and / or text information. For example, the message field may include text that may be displayed in the notice. In other examples, the message field may include objects, links, video / audio clips, etc., that may be accessed and / or presented in the notice.

[0088] The Creator field may contain instructions from the actor who created the Notice 600. In some cases, if the Notice 600 was created automatically, the Creator field may indicate that the Notice 600 was created automatically, or it may indicate the actor to whom the Notice 600 was created automatically. In some examples, the Target Actor field may indicate that the Target Actor is the creator. In some cases, the creator may create a Notice 600 intended to target another actor. In such cases, the Target Actor may indicate that the Notice 600 may be broadcast to a specific other intended actor, multiple other actors, or all actors. In some cases, the Notice 600 may only be visible to the actor specified in the Target Actor field.

[0089] The anchor field may indicate the object (e.g., related) to which notification 600 can be anchored. In some examples, notification 600 may be anchored to an object, and notification 600 may be accessible when the anchored object is accessible (e.g., notification 600 may be visible when the object to which notification 600 is anchored is visible). In some cases, notification 600 may include an anchor policy field. The anchor policy field may contain metadata about displaying notification 600 anchored to an object. For example, the anchor policy may indicate that notification 600 may be displayed after a certain time or in a specific relationship to the anchor object (e.g., one foot above the anchor object). If the target actor is different from the creator, the notification policy may include information about how the notification may be communicated to the other actor. For example, the notification policy may indicate that the notification may be sent to the target actor when the target actor comes within a certain distance of the anchor object or when the anchor object is within the target actor's line of sight.

[0090] In some cases, the initial media type may indicate the media format of the message. For example, the initial media type may indicate that the message is in text format. Other media types may include hyperlinks, audio, video, holographics, etc. The initial media policy may include metadata indicating how the media may be presented. For example, metadata may indicate the font, size, color, etc., for text media types. Similarly, for video, metadata may indicate the video size, resolution, looping, etc.

[0091] In some cases, notification 600 may contain information about one or more transitions. For example, notification 600 may contain a transition event field for a first notification. The transition event field may contain metadata indicating an action or condition that may trigger a transition in the notification. For example, the transition event field may indicate that a first transition may occur when notification 600 is read by a target actor. In some cases, the transition event field may also indicate one or more actions and / or conditions (e.g., events) that can be recognized by contextual analysis of data received by the XR system, such as captured images, sensed motion, or XR content received for presentation. In some cases, contextual analysis may be performed by one or more ML models. In some examples, the transition event field may specify the expected output (e.g., prediction) from one or more ML models to apply and / or one or more ML models that may trigger the transition.

[0092] In Figure 6, notification 600 also includes a transition action field for the first transition. The transition action field may contain metadata indicating what actions the transition may perform. For example, the transition action field may indicate that notification 600 may change the object to which notification 600 is anchored to another object. Other transitions may include changing the message presented by notification 600, changing the media type of the message, changing the aspect of the XR scene (e.g., a virtual environment), or providing feedback to an actor. In some cases, a transition media type field may be included to help indicate when the media type of the message may be changed for the transition. In some cases, instructions for animations and / or effects that are displayed as part of the transition (e.g., played, heard, felt, etc.) may be included in the transaction field. In other cases, instructions for animations and / or effects may be included in a separate field. In some cases, instructions for animations and / or effects may be indicated by specified metadata indicating what actions the transition may perform (e.g., included as part of it). In other cases, instructions for animations and / or effects may be explicitly specified.

[0093] In some cases, notification 600 may also include a completion policy field. The completion policy field may contain metadata about when the notification can be completed and / or dismissed. For example, if the notification can be dismissed manually, the completion policy field may be left blank or indicate that the notification can be dismissed manually. In other cases, the completion policy field may indicate that the notification cannot be dismissed manually. In some cases, the completion policy field may contain metadata indicating actions and / or conditions that can cause the notification to be automatically dismissed, which can be recognized by contextual analysis of the data received by the XR system.

[0094] Figure 7 is a block diagram of a notification engine 700 according to an aspect of the present disclosure. In some cases, the notification engine 700 may be included as part of an XR engine, such as the XR engine 220 in Figure 2, or as an XR application running on an XR system, such as the XR system 200 in Figure 2. The notification engine 700 may receive environment information 702, which may include information about the real and / or virtual environment, such as digital information (e.g., from the image processing engine 224, rendering engine 226, and / or XR engine 220 in Figure 2) about images of the environment adjacent to the XR system 200 and / or about virtual objects that will be rendered within the XR environment. In some examples, the environment information 702 may be input to a context analysis engine 706 of the notification engine 700. The context analysis engine 706 may be communicably coupled to a notification creation engine 708, an event monitoring engine 710, and a notification completion engine 712. The notification engine 700 may also receive input information 704 from, for example, a user interface. In some cases, UI input information 704 may be input to the notification creation engine 708.

[0095] In some embodiments, the notification creation engine 708 may create a notification based on UI input information 704 and / or contextual information from the context analysis engine 706. For example, the context analysis engine 706 may process input environment information 702 to automatically determine whether to create a notification. In some cases, the context analysis engine 706 may use scene analysis and context along with learned environmental conditions to determine whether to create a notification. Learned environmental conditions may be learned based on previous experiences, for example, by one or more ML models trained to recognize environmental conditions under which a user is likely to create a notification. In some cases, these notifications may be personalized and / or trained based on personal preferences. For example, the context analysis engine 706 may analyze received images of the environment in a proximity XR system over a period of time to determine that the user has not walked the dog during that period and send instructions to the notification creation engine to create a notification to walk the dog.

[0096] In some cases, instructions for creating a notification may be received by the notification creation engine 708, for example, based on UI input information 704. The notification creation engine 708 may obtain contextual information from, for example, the context analysis engine 706 and generate a notification template based on the contextual information. In some examples, a user interface, such as a virtual form or a set of prompts, may be provided to the user to collect information for creating a notification. One or more parts of the user interface or prompts may be pre-filled based on the notification template. In some cases, the notification template may include suggested values ​​for creating, transitioning to, and completing the notification. Suggested values ​​may include, for example, a suggested message, a suggested target for the message, a suggested object to anchor the notification, transition information, completion criteria, etc. In some cases, the suggested values ​​may be based on a previously created notification, for example, based on a combination of the notification creation time, location, and current contextual information about the current action that the user of the XR system appears to be performing (for example, a template for creating a reminder to purchase a pantry item based on contextual information about viewing through the pantry). In some cases, suggested values ​​for a notification template may be crowdsourced for similar notifications based on similar actions, similar locations, metadata (e.g., about transition policies, completion criteria, actor types, etc.), and contextual information such as demographics. In some cases, pre-filled values ​​may be adjusted based on previously answered prompts or fields, for example. In some cases, rather than pre-filling a form, pre-filled values ​​may be represented by a notification preview that shows how a notification created based on the pre-filled values ​​will look and / or behave. For example, an XR system may be able to recognize a set of actions that could trigger a transition for a notification.The user interface for previewing notifications may display an animated graphical representation of the specified action and / or condition, as well as the resulting transitions on the user interface for previewing notifications.

[0097] In some cases, the creator of a notification may anchor the notification to an anchor object for another target actor. The target actor may receive an existing notification 714 anchored to the anchor object, for example, if the target actor moves near the anchor object. In such a case, the received existing notification 714 is received by the notification creation engine 708, and a corresponding notification may be created for the target actor.

[0098] In some cases, the notification creation engine 708 may output the created notification to the event monitoring engine 710. The event monitoring engine 710 may set up and / or monitor events corresponding to the trigger events of the received notification. In some examples, the event monitoring engine 710 may configure the context analysis engine 706 to monitor specific trigger events as indicated in the received notification. In some cases, the event monitoring engine 710 may monitor the context analysis engine 706 for indications of specific trigger events as indicated in the received notification.

[0099] In some examples, a created notification may include one or more transitions. To help create a more interactive and useful XR ecosystem, notifications are not static notifications, but can transition from one state to another. Allowing notifications to dynamically adjust their states can make them more relevant and / or useful, as it allows them to respond to behavior by appearing to respond to and interact with actors. In some cases, the creator of a notification may specify one or more transitions in the notification and conditions (e.g., trigger events) that can trigger the transitions (e.g., cause the transitions to occur).

[0100] In some cases, a transition may occur based on the determination that the target actor has read the notification. If a transition may occur based on the target actor having read the notification (e.g., a message), whether the target actor has read the notification may be determined, for example, based on eye tracking and / or gaze detection. In some cases, the XR system may include one or more sensors, such as imaging sensors, that can image the user's eyes to help track the user's eyes (e.g., a pair of eyes). Such eye-tracking or gaze detection systems may be used to help determine where the user is looking.

[0101] In some cases, the determination that a user has read a notification may be based on eye saccadic movements. Eye saccadic movements can be rapid, simultaneous movements of both eyes in a common direction, and saccadic movements can be detected when reading text. In some cases, eye-tracking information may be passed to a context analysis engine 706 as part of environmental information 702, which may detect and / or measure eye saccadic movements. The context analysis engine may compare the measured eye saccadic movements to an expected amount of saccadic movements. In some cases, if the target actor's eyes are directed towards the notification and the measured saccadic movements correspond to an expected amount of saccadic movements, it may be determined that the target actor has read the notification. In some cases, the expected amount of saccadic movements may be dynamically determined based on other contextual factors, such as the length of the notification message, the complexity of the message, the target actor's profile (e.g., estimated reading speed, age, etc.), and whether the user is multitasking or may be fatigued. In some cases, the expected amount of saccadic movements may be determined using one or more machine learning models. If the expected amount of saccade movement is measured, the context analysis engine 706 may indicate to the event monitoring engine 710 that the notification has been read.

[0102] In some cases, a transition may be based on whether an action specified in the notification (e.g., a trigger event) has been performed. For example, the context analysis engine 706 may include one or more machine learning models that can detect specific conditions and / or actions of the target user. For example, the context analysis engine 706 may include a machine learning model that can perform behavioral analysis of the target actor and / or scene analysis of the real environment to determine whether a specified trigger event has occurred. Thus, the context analysis engine may detect (e.g., based on environmental information 702 such as images, gyroscope measurements, and location information) that the actor is walking the dog or has gone outside without the dog. In some cases, the notification may include a transition that is triggered based on conditions, actions, and / or non-actions (or any combination thereof) detectable by the context analysis engine 706. In some cases, the event monitoring engine 710 may initiate one or more machine learning models (e.g., of the context analysis engine 706) to monitor specific conditions and / or actions specified in the notification. In some cases, a transition may be based on a specific quality of performing an action. For example, a transition may occur based on walking the dog for at least a certain amount of time, distance, and / or speed. In some examples, a transition may be triggered based on a non-action (e.g., failure to perform an action). For instance, a notification to walk the dog may be triggered to transition if the event (e.g., walking the dog) is not detected by a certain time.

[0103] In some cases, the state of a notification changes during a transition. A change in the state of a notification can change how the notification is presented. In some cases, the message of a notification may change during a transition. For example, if a transition is triggered after the dog has been walked, the message of the notification may change from "Please walk the dog" to "Thank you for walking the dog." In some cases, the text properties of the message (e.g., media policy) may change. For example, the font, color, size, etc., of the message may change. In some cases, the media type of a notification may change. For example, for a transition triggered based on not walking the dog by a certain time, an audio component may be presented along with, or instead of, a text message prompting the dog to be walked. In some cases, changes to the message may be used to emphasize or de-emphasize the notification, or simply to keep the notification interesting.

[0104] In some cases, notifications may transition back and forth between specific states. For example, after walking a dog, a notification might transition to a different message for a certain period of time, and then to the original message to be displayed at a specific time the following day. Another example is a notification to display a packing list when it is detected that a target actor is packing a suitcase; this notification might transition to displaying the packing list when the suitcase is packed, and not displaying the list when the suitcase is not packed. In some cases, certain transitions may be automatically triggered based on context. For example, certain transitions may be automatically set for all notifications to meet specific requirements and / or policies, such as changing font color / size based on time, light level, and maintaining a certain level of contrast. In some cases, the requirements and / or policies may be set by the target actor.

[0105] In some cases, the anchor object associated with a notification may change during a transition. For example, a notification to walk the dog may be triggered to transition from being anchored to an object (e.g., a refrigerator, a dog) to being anchored to the target actor's view if the event (e.g., walking the dog) is not detected by a certain time. Similarly, after the dog has walked, the notification may transition from being anchored to a view to being anchored to another object. In some cases where a notification is anchored to an object in the environment, the notification may be called object-locked or world-locked. When a notification is anchored to a view, the notification may be called view-locked or head-locked. In some cases, a head-locked notification may appear within a fixed area of ​​the XR device (e.g., the notification area). As another example, a notification may move from one anchor object to another as a result of a transition. Thus, a notification anchored to the refrigerator may be anchored to the dog if the dog has not walked for a certain time.

[0106] In some embodiments, how a notification is anchored to an object can change during a transition. For example, the location where a notification appears relative to an anchor object can change, or the notification can transition from being stationary relative to the anchor object to moving relative to the anchor object. As a more specific example, a notification containing a message to walk the dog may be anchored to the dog so that the notification appears on the dog. If the dog has not been walked for a certain period of time, the notification may transition so that the message appears as a speech bubble coming out of the dog's mouth, while it is still anchored to the dog.

[0107] In some cases, the XR scene surrounding (e.g., adjacent to) a notification (and / or anchor object) may be modified during the transition. In some cases, it may be useful to change how a notification is presented by modifying the scene in which the notification is presented (e.g., adjacent to the notification), for example, to highlight the notification. For example, in the case of a notification to walk the dog, if the dog has not been walked by a certain time, the portion of the scene visible in the view through the XR device may be blurred, except for the notification, the anchor object (e.g., the dog), and / or other objects associated with the notification, such as the door. Other modifications to the scene may include highlighting portions of the scene, adding warning messages to portions of the scene, or covering portions of the scene.

[0108] In some embodiments, the notification completion engine 712 may reject a notification and / or mark it as completed. In some cases, a notification may be manually rejected by an actor, for example, and the notification completion engine 712 may receive input information 704 from a user interface, for example, indicating that the actor is attempting to reject the notification. The notification completion engine 712 may reject the notification accordingly. In some cases, the notification completion engine 712 may automatically reject / mark it as completed. In some examples, a notification may include a completion policy that indicates actions and / or conditions under which the notification may be considered completed or automatically rejected. The notification creation engine 708 and / or the event monitoring engine 710 may provide the notification completion engine 712 with information regarding the completion policy and / or the notification. In a similar manner to that described above with respect to the event monitoring engine, the notification creation engine 708 may set up and / or monitor events corresponding to actions and / or conditions under which the notification may be considered completed or automatically rejected. In some examples, the notification completion engine 712 may be configured to monitor the context analysis engine 706 for such events. In some cases, the notification completion engine 712 may monitor the context analysis engine 706 for instructions for specific events, such as those indicated by completion policies. The context analysis engine 706 may perform behavioral analysis of the target actor and / or scene analysis of the real environment to help determine whether an event in the completion policy has been met. In some cases, the completion policy may include quality requirements for a specific action. For example, a completion policy may only be met if the dog has walked for a certain length of time, distance, speed, etc. Based on the information from the context analysis engine 706, the context analysis engine 706 may determine to automatically mark the notification as rejected / completed if it determines that the completion policy has been met.

[0109] In some cases, if a transition action / condition is detected and / or if the notification is dismissed / marked as complete, feedback on whether the notification was viewed may be provided to the actor and / or creator. In some cases, the feedback may be provided as a notification or presented through an alternative mechanism.

[0110] Figure 8 is a flowchart illustrating a process 800 for displaying content according to an aspect of this disclosure. Process 800 may be executed by a computing device (or apparatus) or component of a computing device (e.g., a chipset, a codec, etc.), such as the host processor 152 in Figure 1, the computing component 210 in Figure 2, and / or the processor 1110 in Figure 11. The computing device may be a mobile device (e.g., a cell phone, the mobile device 1050 in Figures 10A and 10B), a network-connected wearable device such as a wristwatch, an extended reality (XR) device such as a virtual reality (VR) device or an augmented reality (AR) device (e.g., the HMD 910 in Figures 9A and 9B, the mobile device 1050 in Figures 10A and 10B), a vehicle or a component or system of a vehicle, or other type of computing device. For example, the computing device may be an XR device including an XR display. The operation of process 800 can be performed on one or more processors (for example, the host processor 152 in Figure 1, the computing component 210 in Figure 2, and / or the processor 1110 in Figure 11) and implemented as an operating software component.

[0111] In block 802, a computing device (or its components) may receive content for display. The content is associated with a first object and a first transition. The first transition indicates a change to apply to the content based on a trigger condition. For example, a notification, such as the callout 506 in Figure 5, may be received from a creator, and the notification includes an anchor and information about one or more transitions, such as one or more transition events, transition actions, and transition media types, as shown in Figure 6. In some cases, the content includes the notification. In some cases, the trigger condition is based on the quality of performing the action. For example, a transition may occur based on walking a dog for at least a certain time, distance, and / or speed. In some cases, the first transition includes associating the content with a second object, and the content is output for display in relation to the second object. For example, as mentioned above with respect to Figure 5, a notification anchored to the refrigerator may become anchored to the dog if the dog is not being walked. In some cases, the second object includes a view for display. For example, as mentioned above with respect to Figure 5, the notification to walk the dog could be a transition from being anchored to an object (e.g., a refrigerator, a dog, etc.) to being anchored to the view of the target actor.

[0112] In block 804, the computing device (or its components) may output content for display related to the first object. For example, a notification such as the callout 506 in Figure 5 may be output as shown in the XR environment.

[0113] In block 806, a computing device (or its components) may determine that a trigger condition has been met. For example, as described with respect to Figure 7, an event monitoring engine may set up and / or monitor events corresponding to trigger events of received notifications, and the event monitoring engine may determine whether an event corresponding to a trigger event has occurred (e.g., based on a transition event). In some cases, the trigger condition includes reading content for display. In some cases, to determine that a trigger condition has been met, a computing device (or its components) may track the movement of a pair of eyes and determine that the content has been read based on the tracked eye movements. For example, a transition may be triggered by reading a notification (e.g., via a transition event), and eye tracking may be used to determine whether the notification has been read. In some cases, a computing device (or its components) may determine that the content has been read by measuring the amount of saccadic movement of the eyes and comparing the measured amount of saccadic movement of the eyes to an expected amount of saccadic movement. In some cases, the expected amount of saccadic movement may be determined using one or more machine learning models. In some cases, the expected amount of saccadic movement may be determined based on the length of the content.

[0114] In some cases, the trigger condition is the absence of an action being performed. In some cases where the trigger condition is the absence of an action being performed, a computing device (or its components) may determine that the trigger condition has been met by receiving images of the environment adjacent to the extended reality device. Based on the received images, the computing device (or its components) may further determine that the action has not been performed. For example, a notification to walk the dog may be triggered to transition if the event (e.g., walking the dog) is not detected by a certain time. Similarly, in some cases, the trigger condition is the performance of an action. In some cases where the trigger condition is the performance of an action, a computing device (or its components) may determine that the trigger condition has been met by receiving images of the environment adjacent to the extended reality device. Based on the received images, the computing device (or its components) may further determine that the action has been performed. In some cases, to determine that an action has been performed, a computing device (or its components) may apply a machine learning model to detect the performance of the action.

[0115] In block 808, the computing device (or its components) may modify the content for presentation based on a first transition in response to a determination that a trigger condition has been met. For example, changes in how content is displayed, such as the notification to walk the dog as described with respect to Figure 5, may be changed based on the transition action. In some cases, to modify the content, the computing device (or its components) may modify the appearance of the content. For example, in some cases, to modify the appearance of the content, the computing device (or its components) may modify the text of the displayed content. As mentioned above, after the dog has been walked, the notification message may change from "Please walk the dog" to "Thank you for walking the dog." As another example, in some cases, to modify the appearance of the content, the computing device (or its components) may modify the media type of the content. As mentioned above, an audio component may be presented along with, or instead of, the text message to walk the dog. In another example, in some cases, to modify the appearance of the content, the computing device (or its components) may modify an extended reality scene adjacent to the content. As explained above, in the case of a notification to walk the dog, if the dog has not been walked by the specified time, the portion of the scene visible in the view through the XR device may be blurred.

[0116] In block 810, the computing device (or its components) may output modified content for display. In some cases, outputting modified content may include displaying the modified content and / or displaying an animation to indicate that the content is transitioning from one state to another (to the modified content). As mentioned above, in some cases the computing device may be an XR device including an XR display. In such cases, the computing device may be configured to output modified content for display on the XR display.

[0117] Figure 9A is a perspective view 900 showing a head-mounted display (HMD) 910 in several examples. The HMD 910 could be, for example, an augmented reality (AR) headset, a virtual reality (VR) headset, a mixed reality (MR) headset, an extended reality (XR) headset, or any combination thereof. The HMD 910 could be an example of an XR system 200, a SLAM system, or a combination thereof. The HMD 910 includes a first camera 930A and a second camera 930B along the front portion of the HMD 910. In some examples, the HMD 910 may have only a single camera. In some examples, the HMD 910 may include one or more additional cameras in addition to the first camera 930A and the second camera 930B. In some examples, the HMD 910 may include one or more additional sensors in addition to the first camera 930A and the second camera 930B.

[0118] Figure 9B is a perspective view 930 showing the head-mounted display (HMD) 910 of Figure 9A being worn by user 920 in several examples. User 920 wears the HMD 910 on his head, over his eyes. The HMD 910 can capture images using a first camera 930A and a second camera 930B. In some examples, the HMD 910 displays one or more display images based on images captured by the first camera 930A and the second camera 930B toward user 920's eyes. The display images may provide a stereoscopic view of the environment, and may have information overlaid and / or other modifications. For example, the HMD 910 may display a first display image based on an image captured by the first camera 930A to user 920's right eye. The HMD 910 may display a second display image based on an image captured by the second camera 930B to user 920's left eye. For example, the HMD910 may provide overlay information within the overlaid display image on top of the images captured by the first camera 930A and the second camera 930B.

[0119] The HMD910 may not include wheels, propellers, or any other means of transport of its own. Instead, the HMD910 relies on the user 920's movements to move the HMD910 within the environment. In some cases, for example, if the HMD910 is a VR headset, the environment may be entirely or partially virtual. If the environment is at least partially virtual, movement through the virtual environment may also be virtual. For example, movement through a virtual environment may be controlled by an input device 208. The movement actuator may include any such input device 208. Movement through a virtual environment may not require wheels, propellers, legs, or any other form of transport. Even if the environment is virtual, the virtual environment may not be mapped and / or may be generated by devices other than the HMD910, such as a remote server or console associated with a video game or video game platform, so SLAM techniques may still be valuable.

[0120] Figure 10A is a perspective view 1000 showing the front 1055 of a mobile device 1050 performing the features described herein, including, for example, feature tracking and / or visual simultaneous location identification and mapping (VSLAM) using one or more front cameras 1030A-B, by some example. The mobile device 1050 may be, for example, a mobile phone, satellite phone, portable gaming console, music player, health tracking device, wearable device, wireless communication device, laptop, mobile device, any other type of computing device or computing system 1100 described herein, or a combination thereof. The front 1055 of the mobile device 1050 includes a display screen 1045. The front 1055 of the mobile device 1050 includes a first camera 1030A and a second camera 1030B. The first camera 1030A and the second camera 1030B are shown within the bezel around the display screen 1045 on the front 1055 of the mobile device 1050. In some examples, the first camera 1030A and the second camera 1030B can be positioned in a notch or cutout from the display screen 1045 on the front 1055 of the mobile device 1050. In some examples, the first camera 1030A and the second camera 1030B can be under-display cameras positioned between the display screen 1045 and the rest of the mobile device 1050, so that light passes through the portion of the display screen 1045 before reaching the first camera 1030A and the second camera 1030B. In the perspective view 1000, the first camera 1030A and the second camera 1030B are front cameras. The first camera 1030A and the second camera 1030B are oriented perpendicular to the plane of the front 1055 of the mobile device 1050. In some examples, the front 1055 of the mobile device 1050 may have only a single camera. In some examples, the mobile device 1050 may include one or more additional cameras in addition to the first camera 1030A and the second camera 1030B.In some examples, the mobile device 1050 may include one or more additional sensors in addition to the first camera 1030A and the second camera 1030B.

[0121] Figure 10B is a perspective view 1010 showing the back 1065 of the mobile device 1050. The mobile device 1050 includes a third camera 1030C and a fourth camera 1030D on the back 1065 of the mobile device 1050. In perspective view 1010, the third camera 1030C and the fourth camera 1030D are located on the back. The third camera 1030C and the fourth camera 1030D are oriented perpendicular to the plane of the back 1065 of the mobile device 1050. The back 1065 of the mobile device 1050 does not have a display screen 1045 as shown in perspective view 1010, but in some examples, the back 1065 of the mobile device 1050 may have a second display screen. If the back 1065 of the mobile device 1050 has a display screen 1045, any arrangement of the third camera 1030C and the fourth camera 1030D relative to the display screen 1045 may be used, as described with respect to the first camera 1030A and the second camera 1030B on the front 1055 of the mobile device 1050. In some examples, the back 1065 of the mobile device 1050 may have only a single camera. In some examples, the mobile device 1050 may include one or more additional cameras in addition to the first camera 1030A, the second camera 1030B, the third camera 1030C, and the fourth camera 1030D. In some examples, the mobile device 1050 may include one or more additional sensors in addition to the first camera 1030A, the second camera 1030B, the third camera 1030C, and the fourth camera 1030D.

[0122] Similar to the HMD 910, the mobile device 1050 does not include wheels, propellers, or any other means of transport of its own. Instead, the mobile device 1050 relies on the movement of the user holding or wearing the mobile device 1050 to move the mobile device 1050 within the environment. In some cases, for example, if the mobile device 1050 is used for AR, VR, MR, or XR, the environment may be entirely or partially virtual. In some cases, the mobile device 1050 may be slotted into a head-mounted device (HMD) (e.g., into the HMD cradle) so that the mobile device 1050 functions as the display of the HMD, and the display screen 1045 of the mobile device 1050 functions as the display of the HMD. If the environment is at least partially virtual, movement through the virtual environment may also be virtual. For example, movement through the virtual environment may be controlled by one or more joysticks, buttons, video game controllers, mice, keyboards, trackpads, and / or other input devices coupled to the mobile device 1050 in a wired or wireless manner.

[0123] Figure 11 shows an example of a system implementing a particular aspect of the present technology. In particular, Figure 11 shows an example of a computing system 1100, which can be, for example, an internal computing system, a remote computing system, a camera, or any computing device that constitutes any component that communicates with each other using connection 1105. Connection 1105 can be a physical connection using a bus, or a direct connection to a processor 1110, such as in a chipset architecture. Connection 1105 can also be a virtual connection, a networked connection, or a logical connection.

[0124] In some examples, the computing system 1100 is a distributed system in which the functions described in this disclosure may be distributed across a data center, multiple data centers, a peer network, etc. In some examples, one or more of the system components described represent many components, each performing some or all of the functions described. In some cases, the components may be physical or virtual devices.

[0125] An exemplary system 1100 includes at least one processing unit (CPU or processor) 1110 and a connection 1105, the connection 1105 connecting various system components to the processor 1110, including system memory 1115 such as read-only memory (ROM) 1120 and random access memory (RAM) 1125. The computing system 1100 may include a high-speed memory cache 1112 that is directly connected to the processor 1110, connected in close proximity to the processor 1110, or integrated as part of the processor 1110.

[0126] The processor 1110 may include any general-purpose processor, a dedicated processor in which hardware or software services such as services 1132, 1134, and 1136 stored in a memory device 1130, and software instructions are incorporated into the actual processor design, configured to control the processor 1110. The processor 1110 may be a fully self-contained computing system including multiple cores or processors, buses, memory controllers, caches, etc. A multicore processor may be symmetric or asymmetric.

[0127] To enable interaction with the user, the computing system 1100 includes an input device 1145 which can represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, speech, a camera, an accelerometer, and a gyroscope. The computing system 1100 may also include an output device 1135 which may be one or more of several output mechanisms. In some cases, a multimodal system may allow the user to provide multiple types of input / output to communicate with the computing system 1100. The computing system 1100 may include a communication interface 1140 which can control and manage user input and system output overall.Communication interfaces include audio jacks / plugs, microphone jacks / plugs, universal serial bus (USB) ports / plugs, Apple® Lightning® ports / plugs, Ethernet ports / plugs, fiber optic ports / plugs, proprietary wired ports / plugs, Bluetooth® wireless signal transmission, Bluetooth® Low Energy (BLE) wireless signal transmission, IBEACON® wireless signal transmission, radio-frequency identification (RFID) wireless signal transmission, near-field communications (NFC) wireless signal transmission, dedicated short-range communication (DSRC) wireless signal transmission, 802.11 Wi-Fi wireless signal transmission, wireless local area network (WLAN) signal transmission, visible light communication (VLC), and Worldwide Interoperability for Microwave Access. It is possible or facilitates the reception and / or transmission of wired or wireless communications using wired and / or wireless transceivers, including those using Access (WiMAX), infrared (IR) wireless signaling, Public Switched Telephone Network (PSTN) signaling, Integrated Services Digital Network (ISDN) signaling, 3G / 4G / 5G / LTE cellular data network wireless signaling, ad hoc network signaling, radio signaling, microwave signaling, infrared signaling, visible light signaling, ultraviolet light signaling, wireless signaling along the electromagnetic spectrum, or any combination thereof.The communication interface 1140 may also include one or more GNSS receivers or GNSS transceivers used to determine the location of the computing system 1100 based on the reception of one or more signals from one or more satellites associated with one or more Global Navigation Satellite Systems (GNSS) systems. GNSS systems include, but are not limited to, the US-based Global Positioning System (GPS), the Russia-based Global Navigation Satellite System (GLONASS), the China-based Beidou Navigation Satellite System (BDS), and the European-based Galileo GNSS. There are no restrictions on operation on any particular hardware configuration; therefore, the fundamental features described herein can be easily superseded as improved hardware or firmware configurations are developed.

[0128] The storage device 1130 can be a non-volatile and / or non-temporary and / or computer-readable memory device, such as a magnetic cassette, flash memory card, solid-state memory device, digital multipurpose disk, cartridge, floppy disk, flexible disk, hard disk, magnetic tape, magnetic strip / stripe, any other magnetic storage medium, flash memory, memory stick memory, any other solid-state memory, compact disc read-only memory (CD-ROM) optical disc, rewritable compact disc (CD) optical disc, digital video disk (DVD) optical disc, Blu-ray disc (BDD) optical disc, holographic optical disc, another optical medium, secure digital (SD) card, micro secure digital (microSD) card, memory stick® card, smart card chip, EMV chip, subscriber identity module (SIM) card, mini / micro / nano / pico SIM card, or another integrated circuit (integrated) Circuits, ICs, chips / cards, random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash EPROM, cache memory (L1 / L2 / L3 / L4 / L5 / L#), resistive random-access memory (RRAM / ReRAM), phase change memory (PCM), spin transfer memory.This can be a hard disk or other type of computer-readable medium capable of storing computer-accessible data, such as RAM, STT-RAM, another memory chip or cartridge, and / or a combination thereof.

[0129] The storage device 1130 may contain software services, servers, and other services, and when the code defining such software is executed by the processor 1110, it causes the system to perform functions. In some examples, a hardware service that performs a particular function may include software components stored on a computer-readable medium in relation to the necessary hardware components such as the processor 1110, connection 1105, and output device 1135 in order to perform the function.

[0130] As used herein, the term “computer-readable medium” includes, but is not limited to, portable or nonportable storage devices, optical devices, and various other media capable of storing, storing, or transporting instructions and / or data. Computer-readable medium may include non-transient media capable of storing data and that do not contain carrier waves and / or transient electronic signals propagating wirelessly or via wired connections. Examples of non-transient media may include, but are not limited to, optical storage media such as magnetic disks or magnetic tapes, compact discs (CDs) or digital multipurpose discs (DVDs), flash memory, memory, or memory devices. Computer-readable medium may store code and / or machine-executable instructions that may represent procedures, functions, subprograms, programs, routines, subroutines, modules, software packages, classes, or any combination of instructions, data structures, or program statements. Code segments may be coupled to other code segments or hardware circuits by passing and / or receiving information, data, arguments, parameters, or memory content. Information, arguments, parameters, data, etc., may be passed, forwarded, or transmitted using any preferred means, including memory sharing, message passing, token passing, network transmission, etc.

[0131] In some examples, computer-readable storage devices, media, and memory may include cable or wireless signals, such as bitstreams. However, non-transient computer-readable storage media, where mentioned, explicitly exclude media such as energy, carrier signals, electromagnetic waves, and signals themselves.

[0132] Specific details are given in the above description to give a complete understanding of the examples provided herein. However, it will be understood by those skilled in the art that the examples can be practiced without these specific details. For the sake of clarity, in some examples the technique may be presented as including individual functional blocks, which include devices, device components, steps or routines in the way they are embodied in software, or combinations of hardware and software. Additional components other than those shown in the figures and / or described herein may be used. For example, circuits, systems, networks, processes, and other components may be shown as components in the form of block diagrams so as not to obscure the examples with unnecessary details. In other examples, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary details so as not to obscure the examples.

[0133] Individual examples may be described above as processes or methods represented as flowcharts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. While flowcharts can describe operations as sequential processes, many of these operations can be performed in parallel or simultaneously. Furthermore, the order of operations may be rearranged. A process terminates when its operations are complete, but it may have additional steps not shown in the diagram. A process can correspond to a method, function, procedure, subroutine, subprogram, etc. If a process corresponds to a function, its termination may correspond to the function returning to a calling function or main function.

[0134] The processes and methods described above can be implemented using computer-executable instructions stored on or available from computer-readable media. Such instructions may include, for example, instructions and data that cause a general-purpose computer, a dedicated computer, or a processing device to perform a particular function or set of functions, or to configure a general-purpose computer, a dedicated computer, or a processing device to perform a particular function or set of functions. The portion of the computer resources used may be accessible over a network. Computer-executable instructions may be, for example, binaries, or intermediate format instructions such as assembly language, firmware, or source code. Examples of computer-readable media that may be used to store instructions, information used, and / or information created in the methods described in the examples include magnetic or optical disks, flash memory, USB devices with non-volatile memory, and networked storage devices.

[0135] Devices implementing processes and methods in accordance with these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take on any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) that perform the required tasks may be stored in computer-readable or machine-readable media. A processor(s) may perform the required tasks. Typical examples of form factors include laptops, smartphones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rack-mount devices, and standalone devices. The functionalities described herein may also be embodied in peripherals or add-in cards. Such functionalities may also, as a further example, be implemented between different chips on a circuit board or between different processes running within a single device.

[0136] Instructions, a medium for transmitting such instructions, computing resources for executing those instructions, and other structures supporting such computing resources are exemplary means of providing the functionality described in this disclosure.

[0137] While the above description illustrates aspects of this application in relation to specific examples, it will be recognized by those skilled in the art that this application is not limited thereto. Therefore, although examples for illustrating this application are described in detail herein, it should be understood that the concept of the present invention can be embodied and adopted in various ways, and that, unless limited by the prior art, the appended claims are intended to be interpreted as including such variations. The various features and aspects of this application described above may be used individually or in combination. Furthermore, the examples may be used in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of this specification. Therefore, this specification and the drawings should be considered illustrative and not limiting. For illustrative purposes, the methods are described in a particular order. It should be understood that in alternative examples, the methods may be performed in an order different from that described.

[0138] Those skilled in the art will understand that the symbols or terms less than ("<") and greater than (">") used herein may be replaced with the symbols less than or equal to ("≦") and greater than or equal to ("≧"), respectively, without departing from the scope of this description.

[0139] Where a component is described as "configured to perform a particular operation," such configuration can be achieved, for example, by designing an electronic circuit or other hardware to perform that operation, by programming a programmable electronic circuit (e.g., a microprocessor or other suitable electronic circuit) to perform that operation, or by any combination thereof.

[0140] The phrase "connected to" refers to any component that is physically connected to another component, either directly or indirectly, and / or communicates with another component, either directly or indirectly (for example, connected to another component via a wired or wireless connection and / or other preferred communication interface).

[0141] The wording of a claim or other statement that includes "at least one of" a set and / or "one or more" of a set indicates that one member of that set, or multiple members of that set (in any combination), satisfy the scope of the claim. For example, the wording of a claim that includes "at least one of A and B" means A, B, or A and B. In another example, the wording of a claim that includes "at least one of A, B, and C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The wording "at least one of" a set and / or "one or more" of a set does not limit the set to items listed within it. For example, the wording of a claim that includes "at least one of A and B" may mean A, B, or A and B, and may also include items not listed in the set A and B.

[0142] The various exemplary logic blocks, modules, circuits, and algorithmic steps described in relation to the examples disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly demonstrate this hardware- and software compatibility, various exemplary components, blocks, modules, circuits, and steps have been outlined above in terms of their functionality. Whether such functionality is implemented as hardware or executed as software depends on the specific application and the design constraints imposed on the overall system. A person skilled in the art may implement the described functionality in various ways for each specific application, but such a decision on implementation should not be construed as a cause for departure from the scope of this application.

[0143] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as general-purpose computers, wireless communication device handsets, or integrated circuit devices with multiple applications, including applications in wireless communication device handsets and other devices. Any feature described as a module or component may be implemented as a whole in an integrated logic device, or discretely as separate but interoperable logic devices. When implemented in software, the techniques may be implemented at least in part by a computer-readable data storage medium comprising program code that, when executed, performs one or more of the methods described above. The computer-readable data storage medium may also form part of a computer program product, which may include packaging materials. Computer-readable media may include random access memory (RAM), such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, and other memory or data storage media. These techniques may be additionally or alternatively implemented by computer-readable communication media that carry or communicate program code in the form of instructions or data structures, such as propagating signals or waves, which can be accessed, read, and / or executed by a computer.

[0144] The program code may be executed by a processor, which may include one or more processors such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such processors may be configured to perform any of the techniques described herein. A general-purpose processor may be a microprocessor, but alternatively, a processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Thus, as used herein, the term “processor” may refer to any of the structures described herein, any combination thereof, or any other structure or device suitable for implementing the techniques described herein. In addition, in some embodiments, the functionality described herein may be provided within a dedicated software module or hardware module configured for encoding and decoding, or incorporated within a composite video encoder-decoder (CODEC).

[0145] Claim language or other wording that includes phrases such as "at least one processor configured to do so," or "at least one processor configured to do so," indicates that one or more processors (in any combination) can perform the associated operations (one or more). For example, claim language that includes "at least one processor configured to do X, Y, and Z" means that a single processor can perform operations X, Y, and Z, or that each of several processors may task with a particular subset of operations X, Y, and Z, so that several processors perform X, Y, and Z together, or that a group of several processors may work together to perform operations X, Y, and Z. In another example, claim language that includes "at least one processor configured to do X, Y, and Z" may mean that any single processor may perform only at least a subset of operations X, Y, and Z.

[0146] The embodiments for explaining this disclosure include the following:

[0147] Embodiment 1. An extended reality device for displaying content, comprising at least one memory and at least one processor coupled to the at least one memory, wherein the at least one processor is configured to receive content for display, the content being associated with a first object and a first transition, the first transition indicating a change to be applied to the content based on a trigger condition, output content for display associated with the first object, determine that the trigger condition has been met, modify the content for presentation based on the first transition in response to the determination that the trigger condition has been met, and output the modified content for display.

[0148] Embodiment 2. The extended reality device according to Embodiment 1, wherein the content includes a notification.

[0149] Embodiment 3. The Extended Reality Apparatus according to Embodiment 2, wherein at least one processor is configured to track the movement of a pair of eyes and, based on the tracked eye movements, determine that content has been read.

[0150] Embodiment 4. The Extended Reality Apparatus according to Embodiment 3, wherein at least one processor is configured to measure the amount of eye saccadic movement and compare the measured amount of eye saccadic movement with an expected amount of saccadic movement in order to determine that content has been read.

[0151] Embodiment 5. The extended reality device according to Embodiment 4, wherein the expected amount of saccadic motion can be determined based on the length of the content.

[0152] Embodiment 6. An extended reality device according to any one of Embodiments 1 to 5, wherein the trigger condition includes reading content for display.

[0153] Embodiment 7. An extended reality device according to any one of Embodiments 1 to 6, wherein the trigger condition includes the absence of an action being performed, and in order to determine that the trigger condition has been met, at least one processor is configured to receive an image of the environment adjacent to the extended reality device and, based on the received image, determine that the action was not performed.

[0154] Embodiment 8. An extended reality device according to any one of Embodiments 1 to 7, wherein the trigger condition includes the execution of an action, and in order to determine that the trigger condition has been met, at least one processor receives an image of the environment adjacent to the extended reality device and determines, based on the received image, that an action has been performed.

[0155] Embodiment 9. The extended reality device according to Embodiment 8, wherein at least one processor is configured to apply a machine learning model to detect the execution of an action in order to determine that an action is being performed.

[0156] Embodiment 10. An extended reality device according to any one of Embodiments 1 to 9, wherein the trigger condition is based on the quality of the action performed.

[0157] Embodiment 11. The Extended Reality Apparatus according to Embodiment 10, wherein a first transition includes associating content with a second object, and the content is output for display in relation to the second object.

[0158] Embodiment 12. The extended reality apparatus according to Embodiment 11, wherein the second object includes a view for display.

[0159] Embodiment 13. An extended reality device according to any one of embodiments 1 to 12, wherein at least one processor is configured to modify the appearance of the content in order to modify the content.

[0160] Embodiment 14. The Extended Reality Apparatus according to Embodiment 13, wherein at least one processor is configured to modify the text of the content to be displayed in order to modify the appearance of the content.

[0161] Embodiment 15. The extended reality apparatus according to Embodiment 13, wherein at least one processor is configured to modify the media type of the content in order to modify the appearance of the content.

[0162] Embodiment 16. The Extended Reality Apparatus according to Embodiment 13, wherein at least one processor is configured to modify an Extended Reality scene adjacent to the content in order to modify the appearance of the content.

[0163] Embodiment 17. An extended reality device according to any one of embodiments 1 to 16, wherein the extended reality device includes an extended reality display, and the extended reality device is configured to output content modified for display on the extended reality display.

[0164] Embodiment 18. A method for displaying Extended Reality Content, comprising: receiving Content for Display, the Content being associated with a first object and a first transition, the first transition indicating a change to be applied to the Content based on a trigger condition; outputting Content for Display associated with a first object; determining that a trigger condition has been met; modifying the Content for Presentation based on the first transition in response to the determination that a trigger condition has been met; and outputting the modified Content for Display.

[0165] Apparatus 19. The method of Apparatus 18, wherein the content includes a notice.

[0166] Embodiment 20. The method of Embodiment 19, wherein determining that a trigger condition has been met includes tracking the movement of a pair of eyes and determining, based on the tracked eye movement, that the content has been read.

[0167] Embodiment 21. The method of Embodiment 20, wherein determining that content has been read includes measuring the amount of saccadic movement of the eyes and comparing the measured amount of saccadic movement of the eyes with an expected amount of saccadic movement.

[0168] Embodiment 22. The method according to Embodiment 21, wherein the expected amount of saccadic movement can be determined based on the length of the content.

[0169] Embodiment 23. The method of Embodiment 18, wherein the trigger condition includes reading content for display.

[0170] Embodiment 24. The method according to any one of embodiments 18 to 23, wherein the trigger condition includes the absence of an action being performed, and determining that the trigger condition has been met includes receiving an image of the environment adjacent to the extended reality device, and determining, based on the received image, that the action was not performed.

[0171] Embodiment 25. The method according to any one of Embodiments 18 to 24, wherein the trigger condition includes the execution of an action, and determining that the trigger condition has been met includes receiving an image of the environment adjacent to the extended reality device, and determining, based on the received image, that the action has been executed.

[0172] Embodiment 26. The method of Embodiment 25, wherein determining that an action is being performed includes applying a machine learning model to detect the execution of an action.

[0173] Embodiment 27. The method according to any one of Embodiments 18 to 26, wherein the trigger condition is based on the quality of the action performed.

[0174] Embodiment 28. The method according to any one of Embodiments 18 to 27, wherein the first transition includes associating content with a second object, and the content is output for display in relation to the second object.

[0175] Embodiment 29. The method according to any one of Embodiments 18 to 28, wherein modifying the content includes modifying the appearance of the content.

[0176] Embodiment 30. A non-temporary computer-readable medium storing instructions, wherein when an instruction is executed by at least one processor, the medium causes at least one processor to receive content for display, the content being associated with a first object and a first transition, the first transition indicating a change to be applied to the content based on a trigger condition, to output content for display associated with the first object, to determine that the trigger condition has been met, and in response to the determination that the trigger condition has been met, to modify the content for presentation based on the first transition, and to output the modified content for display.

[0177] Embodiment 31. The method of Embodiment 28, wherein the second object includes a view for display.

[0178] Embodiment 32. The method according to any one of Embodiments 29 to 31, wherein modifying the appearance of the content includes modifying the text of the content as it will be displayed.

[0179] Embodiment 33. The method according to any one of Embodiments 29 to 32, wherein modifying the appearance of the content includes modifying the media type of the content.

[0180] Embodiment 34. The method according to any one of Embodiments 29 to 33, wherein modifying the appearance of the content includes modifying an extended reality scene adjacent to the content.

[0181] Embodiment 35. The method according to any one of embodiments 18 to 29 and 31 to 34, further comprising outputting content modified for display on a display on an Extended Reality Display.

[0182] Embodiment 36. A non-temporary computer-readable medium of Embodiment 30, which, when executed by at least one processor, stores instructions causing at least one processor to perform one or more operations according to Embodiments 18 to 29 and 31 to 35.

[0183] Embodiment 37. An apparatus for displaying extended reality content, comprising one or more means for performing operations according to any of Embodiments 18 to 29 and 31 to 35.

Claims

1. An extended reality device for displaying content, At least one memory, At least one processor coupled to the at least one memory, Receiving content for display, wherein the content is associated with a first object and a first transition, and the first transition indicates a change to be applied to the content based on a trigger condition. Outputting the content for display related to the first object, Determining that the aforementioned trigger condition has been met, In response to the determination that the trigger condition is met, the content is modified for presentation based on the first transition, Outputting the modified content for display A processor configured to perform the following: An extended reality device equipped with [the following features].

2. The Extended Reality Apparatus according to claim 1, wherein the content includes a notification.

3. In order to determine that the trigger condition has been met, at least one processor, Tracking the movement of a pair of eyes, Based on the tracked eye movements, it is determined that the content has been read. The extended reality device according to claim 2, configured as described above.

4. In order to determine that the content has been read, at least one processor, The amount of saccadic movement of the eye is measured, The amount of saccade movement measured by the eye is compared with the amount of saccade movement expected. The extended reality device according to claim 3, configured as described above.

5. The extended reality device according to claim 4, wherein the expected amount of saccadic motion can be determined based on the length of the content.

6. The extended reality device according to claim 1, wherein the trigger condition includes reading the content for display.

7. The trigger condition includes the absence of an action being performed, and in order to determine that the trigger condition has been met, the at least one processor, The extended reality device receives images of the environment adjacent to it. Based on the received image, it is determined that the action was not performed. The extended reality device according to claim 1, configured as described above.

8. The trigger condition includes the execution of an action, and in order to determine that the trigger condition has been met, the at least one processor, The extended reality device receives images of the environment adjacent to it. Based on the received image, it is determined that the action has been performed. The extended reality device according to claim 1, configured as described above.

9. The extended reality device according to claim 8, wherein the at least one processor is configured to apply a machine learning model to detect the execution of the action in order to determine that the action is being performed.

10. The extended reality apparatus according to claim 1, wherein the trigger condition is based on the quality of the action performed.

11. The Extended Reality Apparatus according to claim 1, wherein the first transition includes associating the content with a second object, and the content is output for display in relation to the second object.

12. The extended reality apparatus according to claim 11, wherein the second object includes a view for display.

13. The extended reality apparatus according to claim 1, wherein the at least one processor is configured to modify the appearance of the content in order to modify the content.

14. In order to modify the appearance of the content, the at least one processor, The text of the content that will be displayed, The media type of the aforementioned content, or Extended reality scenes adjacent to the aforementioned content The extended reality device according to claim 13, configured to modify at least one of the following.

15. The extended reality apparatus according to claim 13, wherein the at least one processor is further configured to output instructions for an effect to be displayed.

16. The extended reality device according to claim 1, wherein the extended reality device includes an extended reality display, and the extended reality device is configured to output the modified content for display on the extended reality display.

17. A method for displaying extended reality content, A step of receiving content for display, wherein the content is associated with a first object and a first transition, and the first transition indicates a change to be applied to the content based on a trigger condition. A step of outputting the content for display related to the first object, The steps include determining that the trigger condition has been met, In response to the determination that the trigger condition has been met, the step of modifying the content for presentation based on the first transition, The steps include outputting the modified content for display and Methods that include...

18. The method according to claim 17, wherein the content includes a notice.

19. The step of determining that the trigger condition has been met is: Steps to track the movement of a pair of eyes, A step of determining that the content has been read based on the tracked movement of the eye. The method according to claim 18, including the method described in claim 18.

20. The step of determining that the aforementioned content has been read is: The steps include measuring the amount of saccadic movement of the eye, A step of comparing the measured amount of saccadic movement of the eye with the expected amount of saccadic movement. The method according to claim 19, including the method described in claim 19.

21. The method according to claim 20, wherein the expected amount of saccadic movement can be determined based on the length of the content.

22. The method according to claim 17, wherein the trigger condition includes the step of reading the content for display.

23. The trigger condition includes the absence of an action being performed, and the step of determining that the trigger condition has been met is: The steps include receiving images of the environment adjacent to the extended reality device, The steps include determining that the action was not performed based on the received image, and The method according to claim 17, including the method described in claim 17.

24. The trigger condition includes the execution of an action, and the step of determining that the trigger condition has been met is: The steps include receiving images of the environment adjacent to the extended reality device, The steps include determining that the action has been performed based on the received image, and The method according to claim 17, including the method described in claim 17.

25. The method according to claim 24, wherein the step of determining that the action has been performed includes the step of applying a machine learning model to detect the performance of the action.

26. The method according to claim 17, wherein the trigger condition is based on the quality of the action performed.

27. The method according to claim 17, wherein the first transition includes the step of associating the content with a second object, and the content is output for display in relation to the second object.

28. The method according to claim 17, wherein the step of modifying the content includes the step of modifying the appearance of the content.

29. The method according to claim 17, further comprising the step of outputting an instruction for an effect that will be displayed.

30. A non-temporary computer-readable storage medium storing instructions, wherein when an instruction is executed by at least one processor, the at least one processor receives Receiving content for display, wherein the content is associated with a first object and a first transition, and the first transition indicates a change to be applied to the content based on a trigger condition. Outputting the content for display related to the first object, Determining that the aforementioned trigger condition has been met, In response to the determination that the trigger condition is met, the content is modified for presentation based on the first transition, Outputting the modified content for display A non-temporary computer-readable storage medium that enables the following.

Images