Dynamic Content Presentation for Extended Reality Systems

Abstract

Systems and techniques for extended reality (XR) operation are described. The XR system displays virtual content using a display according to a display setting. The display setting may, for example, identify a position, orientation, and / or size of the displayed virtual content. When the virtual content is displayed by the display, for example, using a see-through or pass-through display, an environment may be viewable using the display. The imaging system may determine a degree of perception of the virtual content displayed by the user using the display based on one or more attributes of one or both eyes of the user of the imaging system. The attributes may, for example, identify eye position, eye movement, pupil dilation, saccade, fixation, blinking, and / or squinting. The XR system may determine modifications to the display setting based on the degree of perception of the virtual content by the user.

Description

[Technical field]

[0001]

[0001] The present application relates to image processing, and more particularly to a system and method for determining a user's level of perception of virtual content and modifying display settings for displaying the virtual content to the user based on the determined user's level of perception of the virtual content. [Background technology]

[0002]

[0002] An extended reality (XR) device is a device that displays an environment to a user, for example through a head-mounted display (HMD) or other device. The environment is at least partially different from the real-world environment in which the user is located. The user can generally interactively change their view of the environment, for example by tilting or moving the HMD or other device. Virtual reality (VR) and augmented reality (AR) are examples of XR.

[0003]

[0003] In some cases, an XR system may include an optical "see-through" display that allows a user to see the user's real-world environment based on light from the real-world environment passing through the display. In some cases, an XR system may include a digital "pass-through" display that allows a user to see a view of the user's real-world environment, or a view of a virtual environment based on the user's real-world environment, based on a view of the environment captured by one or more cameras and displayed on the display. The optical "see-through" XR system or the digital "pass-through" XR system may be worn by a user while the user is engaged in an activity in his or her real-world environment.

[0004]

[0004] An XR system can overlay virtual content on top of a user's view of the environment. The virtual content can provide useful information to a user of the XR system. However, there are situations in which the virtual content may interfere with or distract a user from the user's activities in the user's real-world environment. On the other hand, there are situations in which a user may not notice virtual content, such as an alert, that could have been useful to the user. Summary of the Invention

[0005]

[0005] In some examples, systems and techniques for extended reality (XR) content management are described. In some examples, an XR system causes virtual content to be displayed using a display according to a display setting associated with the virtual content. The display setting may, for example, identify a position, orientation, and / or size of the virtual content displayed on the display, and / or a position, orientation, and / or size of the virtual content relative to a portion of an environment viewable through the display. When the virtual content is displayed by the display, for example, through a see-through or pass-through display, the environment may be viewable through the display. The imaging system may determine a perception level for the virtual content displayed by the user using the display based on one or more position attributes of one or both eyes of the user of the imaging system. The imaging system may determine the position attribute based on one or more sensors focused on the user, such as one or more cameras facing one or both eyes of the user. The position attribute may identify, for example, eye position, eye movement, pupil dilation, saccades, fixations, blinks, squinting, optokinetic reflexes or responses, vestibulo-ocular reflexes or responses, accommodation reflexes or responses, or combinations thereof. In some examples, the imaging system's determination of the level of perception of the virtual content may include determining a level of understanding of the virtual content by the user. The level of understanding of the virtual content by the user may be based on the level of perception of the virtual content by the user, the complexity of the virtual content, the uniqueness of the virtual content, historical data associated with the user, and / or contextual data. The imaging system may determine modifications to display settings corresponding to the virtual content based on the level of perception of the virtual content by the user and / or the level of understanding of the virtual content by the user. Modifications to the display settings may, for example, hide, remove, reduce, reduce salience, reduce priority, or terminate the display of the virtual content.Modifications to the display settings may, for example, emphasize, magnify, move, re-orient, increase salience, and / or increase priority of the virtual content.

[0006]

[0006] In one example, an apparatus for image processing is provided. The apparatus includes a memory and one or more processors (e.g., implemented in a circuit) coupled to the memory. The one or more processors are configured and capable of: causing virtual content to be displayed using a display according to display settings associated with the virtual content; determining a perception level for the virtual content displayed by a user using the display based on one or more perception-related attributes of the user when the virtual content is displayed by the display, where the environment is viewable using the display; determining a modification to the display settings corresponding to the virtual content based on the perception level of the user for the virtual content.

[0007] In another example, a method of image processing is provided that includes causing virtual content to be displayed using a display in accordance with display settings associated with the virtual content, where an environment is viewable using the display when the virtual content is displayed by the display, determining a level of perception by a user to the virtual content displayed using the display based on one or more perception-related attributes of the user, and determining modifications to the display settings corresponding to the virtual content based on the level of perception by the user to the virtual content.

[0008]

[0008] In another example, a non-transitory computer-readable medium is provided having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to display virtual content using a display in accordance with display settings associated with the virtual content, determine a perception level for the virtual content displayed by a user using the display based on one or more perception-related attributes of the user when the virtual content is displayed by the display, and determine modifications to the display settings corresponding to the virtual content based on the user's perception level of the virtual content.

[0009] In another example, an apparatus for image processing is provided that includes means for displaying virtual content using a display in accordance with display settings associated with the virtual content, where an environment is viewable using the display when the virtual content is displayed by the display, means for determining a perception level for the virtual content displayed by a user using the display based on one or more perception-related attributes of the user, and means for determining modifications to the display settings corresponding to the virtual content based on the perception level of the virtual content by the user.

[0010] In some aspects, the one or more perception-related attributes of the user are associated with one or more eyes of the user. In some aspects, the one or more perception-related attributes of the user include at least one of: one or more attributes of one or more eyes of the user, one or more attributes of one or more facial expressions of the user, and one or more gestures of the user.

[0011] In some aspects, the environment is viewable using the display based at least in part on light from the environment passing through at least a portion of the display. In some aspects, the environment is viewable using the display based at least in part on causing a view of the environment to be displayed by the display.

[0012] In some aspects, determining the user's level of perception of the virtual content includes using one or more perception-related attributes of the user as input to one or more trained machine learning systems. In some aspects, the methods, apparatus, and computer-readable media described above further include receiving feedback through a user interface corresponding to the user's level of perception of the virtual content, and updating the one or more trained machine learning systems based on the feedback.

[0013] In some aspects, the methods, devices, and computer-readable media described above further include receiving sensor data captured by one or more sensors indicative of one or more eyes of the user, and determining one or more perception-related attributes of the user based on the sensor data. In some aspects, the methods, devices, and computer-readable media described above further comprise one or more sensors.

[0014]

[0014] In some aspects, the methods, devices, and computer-readable media described above further include determining one or more perception-related attributes of the user based on sensor data captured by one or more image sensors, the sensor data including one or more images of one or more eyes of the user.

[0015] In some aspects, the methods, devices, and computer readable media described above further include determining a level of understanding of the virtual content by the user based on the user's level of perception of the virtual content, and determining modifications to the display settings based on the level of perception includes determining modifications to the display settings based on the level of understanding. In some aspects, the methods, devices, and computer readable media described above further include receiving historical information associated with the user, and determining a level of understanding of the virtual content by the user is based on the historical information regarding the user.

[0016]

[0016] In some aspects, the methods, devices, and computer readable media described above further include determining a characteristic of the virtual content, and determining the user's level of perception of the virtual content is based on the characteristic of the virtual content. In some aspects, the methods, devices, and computer readable media described above further include determining a complexity level of the virtual content, and determining the user's level of perception of the virtual content is based on the complexity level of the virtual content. In some aspects, the methods, devices, and computer readable media described above further include determining a uniqueness level of the virtual content, and determining the user's level of perception of the virtual content is based on the uniqueness level of the virtual content. In some aspects, the methods, devices, and computer readable media described above further include determining a uniqueness level of the virtual content with respect to the environment, and determining the user's level of perception of the virtual content is based on the uniqueness level of the virtual content with respect to the environment.

[0017] In some aspects, the modification to the display settings corresponding to the virtual content includes causing the display to stop displaying at least a portion of the virtual content. In some aspects, the modification to the display settings corresponding to the virtual content includes causing the display to display at least a portion of the virtual content more prominently than before the modification. In some aspects, the modification to the display settings corresponding to the virtual content includes modification to one or more characteristics of the virtual content, the one or more characteristics including at least one of a position, an orientation, a depth, a size, a color, a font size, a font color, a font, a language, and a layout.

[0018] In some aspects, determining a level of perception of the virtual content by the user includes determining that the user has perceived the virtual content. In some aspects, determining a level of perception of the virtual content by the user includes determining that the user does not perceive the virtual content. In some aspects, determining a level of perception of the virtual content by the user includes determining that the user has perceived the virtual content to a first perception level of the plurality of perception levels. In some aspects, determining a level of perception of the virtual content by the user includes determining that the user has perceived the virtual content to a second perception level of the plurality of perception levels.

[0019] In some aspects, modifications to the display settings are based on the likelihood that the virtual content will be re-viewed by the user within a threshold time.

[0020]

[0020] In some aspects, determining the user's level of perception of the virtual content includes determining a confidence level corresponding to the user's level of perception of the virtual content, and modifications to the display settings are based on the confidence level.

[0021] In some aspects, the one or more perception-related attributes of the user include one or more eye positions of one or more eyes of the user relative to the virtual content. In some aspects, the one or more perception-related attributes of the user include one or more characteristics of one or more saccades by one or more eyes of the user, the one or more characteristics including at least one of frequency, duration, timing, saccade velocity, saccade amplitude, eye position, and eye movement. In some aspects, the one or more perception-related attributes of the user include one or more characteristics of one or more fixations by one or more eyes of the user, the one or more characteristics including at least one of frequency, duration, timing, eye position, and eye movement. In some aspects, the one or more perception-related attributes of the user include one or more characteristics of one or more pupil dilations by one or more eyes of the user, the one or more characteristics including at least one of frequency, duration, timing, pupil dilation level, eye position, and eye movement. In some aspects, the one or more perception-related attributes of the user include one or more characteristics of one or more blinks by one or more eyelids of the user, the one or more characteristics including at least one of frequency, duration, timing, blink rate, eye position, and eye movement. In some aspects, the one or more perception-related attributes of the user include one or more characteristics of one or more squints by one or more eyelids of the user, the one or more characteristics including at least one of frequency, duration, timing, squint level, eye position, and eye movement.

[0022]

[0022] In some aspects, the methods, apparatus, and computer-readable media described above further include determining a degree of comprehension of the string by the user based on a perceptual level of the virtual content and a length of the string, where the virtual content includes the string.

[0023] In some aspects, the methods, apparatus, and computer-readable media described above further comprise a display.

[0024]

[0024] In some aspects, the methods, devices, and computer-readable media described above further include determining at least one of the user's one or more perception-related attributes based at least in part on sensor data including a representation of one or more eyes of the user, wherein the one or more sensors are configured to capture the sensor data.

[0025] In some aspects, determining a level of perception of the virtual content by the user includes determining a level of understanding of the virtual content by the user based on one or more perception-related attributes of the user. In some aspects, determining a level of understanding of the virtual content by the user is based on one or more perception-related attributes of the user and at least one of one or more characteristics of the virtual content, the context data, and a user profile of the user. In some aspects, the user profile includes historical data associated with the user. In some aspects, the context data includes one or more reactions by the user to the virtual content. In some aspects, the context data includes a location of the XR system.

[0026] In some aspects, the device is, is part of, and / or includes a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a head mounted display (HMD) device, a wireless communication device, a mobile device (e.g., a mobile phone and / or mobile handset and / or a so-called "smartphone" or other mobile device), a camera, a personal computer, a laptop computer, a server computer, a vehicle or a computing device or component of a vehicle, another device, or a combination thereof. In some aspects, the device includes a camera or multiple cameras for capturing one or more images. In some aspects, the device further includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the devices described above may include one or more sensors (e.g., one or more inertial measurement units (IMUs), such as one or more gyroscopes, one or more gyrometers, one or more accelerometers, any combination thereof, and / or other sensors.

[0027]

[0027] This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used independently to determine the scope of the claimed subject matter, which should be understood by reference to the entire specification of this patent, any or all drawings, and appropriate portions of each claim.

[0028]

[0028] The above, together with other features and embodiments, will become more apparent with reference to the following specification, claims, and accompanying drawings. [Brief description of the drawings]

[0029]

[0029] Exemplary embodiments of the present application are described in detail below with reference to the following drawings. [Figure 1]

[0030] 1 is a block diagram illustrating an example architecture of an image capture and processing system, in accordance with some examples. [Diagram 2]

[0031] FIG. 1 is a block diagram illustrating an example architecture of an extended reality (XR) system that implements a process for determining a level of perception and / or understanding of virtual content displayed using a display by a user viewing the display, according to some examples. [Figure 3A]

[0032] FIG. 1 is a perspective view illustrating a head mounted display (HMD) used as an extended reality (XR) system, according to some examples. [Figure 3B]

[0033] FIG. 3B is a perspective view illustrating the head mounted display (HMD) of FIG. 3A being worn by a user, according to some examples. [Figure 4A]

[0034] FIG. 1 is a perspective view illustrating the front of a mobile handset that includes a forward-facing camera and can be used as an extended reality (XR) system, according to some examples. [Figure 4B]

[0035] FIG. 1 is a perspective view illustrating the rear of a mobile handset that includes a rear-facing camera and can be used as an extended reality (XR) system, according to some examples. [Figure 5A]

[0036] 1A-1C are conceptual diagrams illustrating images of a Boston environment before and after overlay of virtual content displayed according to display settings, in accordance with some examples. [Figure 5B]

[0037] 1A-1C are conceptual diagrams illustrating images of a Boston environment with virtual content overlaid before and after modifications to display settings, in accordance with some examples. [Figure 6]

[0038] 1A-1C are conceptual diagrams illustrating an image containing a book before and after overlay of virtual content displayed according to display settings, in accordance with some examples. [Figure 7]

[0039] 1A-1C are conceptual diagrams illustrating images of a street environment before and after overlay of virtual content displayed according to display settings, in accordance with some examples. [Figure 8]

[0040] FIG. 1 is a block diagram illustrating a process for determining perception and comprehension levels for virtual content based on one or more trained machine learning (ML) models, according to some examples. [Figure 9]

[0041] FIG. 1 is a block diagram illustrating an example of a neural network that may be used by a trained machine learning system to analyze users viewing extended reality content, according to some examples. [Figure 10]

[0042] FIG. 1 is a flow diagram illustrating a process for extended reality (XR) display operations, according to some examples. [Figure 11]

[0043] FIG. 1 illustrates an example of a computing system for implementing certain aspects described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0030]

[0044] Specific aspects and embodiments of the present disclosure are provided below. As will be apparent to those skilled in the art, some of these aspects and embodiments may be applied independently, and some of them may be applied in combination. In the following description, for the purpose of explanation, specific details are set forth to provide a thorough understanding of the embodiments of the present application. However, it will be apparent that various embodiments may be practiced without these specific details. The figures and descriptions are not intended to be limiting.

[0031]

[0045] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of exemplary embodiments provides those skilled in the art with an enabling description for implementing the exemplary embodiments. It should be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the present application as set forth in the appended claims.

[0032]

[0046] A camera is a device that uses an image sensor to receive light and capture image frames, such as still images or video frames. The terms "image," "image frame," and "frame" are used interchangeably herein. A camera may be configured with various image capture and image processing settings. Different settings result in images with different appearances. Some camera settings, such as ISO, exposure time, aperture size, f / stop, shutter speed, focus, and gain, are determined and applied before or during the capture of one or more image frames. For example, settings or parameters may be applied to an image sensor to capture one or more image frames. Other camera settings, such as contrast, brightness, saturation, sharpness, levels, curves, or color changes, may constitute post-processing of one or more image frames. For example, settings or parameters may be applied to a processor (e.g., an image signal processor or ISP) to process one or more image frames captured by the image sensor.

[0033]

[0047] An extended reality (XR) system or device can provide virtual content to a user and / or combine a real-world view of a physical environment (scene) with a virtual environment (including the virtual content). The XR system facilitates user interaction with such a combined XR environment. The real-world view can include real-world objects (also called physical objects), such as people, vehicles, buildings, tables, chairs, and / or other real-world or physical objects. The XR system or device can facilitate interaction with different types of XR environments (e.g., a user can use the XR system or device to interact with an XR environment). The XR system can include a virtual reality (VR) system that facilitates interaction with an augmented reality (AR) environment, an MR system that facilitates interaction with a mixed reality (MR) environment, and / or other XR systems. Examples of XR systems or devices include head-mounted displays (HMDs), smart glasses, among others. In some cases, the XR device can track parts of the user (e.g., the user's hands and / or fingertips) to allow the user to interact with items of virtual content.

[0034]

[0048] Described herein are systems and techniques for optimizing content understanding and real-world engagement for extended reality (XR) systems, such as augmented reality (AR), virtual reality (VR), and / or mixed reality (MR) systems. XR systems can include, for example, HMDs, AR glasses, heads-up displays in vehicles, mobile handsets, and other types of devices and systems.

[0035]

[0049] In some cases, an XR system may include an optical "see-through" or digital "pass-through" display (e.g., a see-through or pass-through AR HMD or AR glasses), allowing the XR system to display XR content (e.g., AR content) directly on the real-world view without displaying video content. For example, a user may view a physical object through a display (e.g., glasses or lenses), and the AR system may display AR content on the display, providing the user with an enhanced visual perception of one or more real-world objects. In one example, the display of an optical see-through AR system may include a lens or glass in front of each eye (or a single lens or glass across both eyes). The see-through display may allow a user to view a real-world or physical object directly, and may display an enhanced image (e.g., projected or otherwise displayed) of that object or additional AR content. This may allow the user's visual perception of the real world to be augmented.

[0036]

[0050] Optical see-through or digital pass-through XR systems can be worn while the user is engaged in the real world (as opposed to VR, where the user is immersed in virtual content and the real world is completely occluded). Unlike smartphones, PCs, and other computing devices, head-mounted XR devices (e.g., smart glasses, HMDs, etc.) are worn on the face and thus mediate the user's visual and auditory sensory channels. For this reason, presenting content on a head-mounted XR device may interfere with or distract the user from their ability to effectively interact with and perceive their surroundings.

[0037]

[0051] XR systems and techniques for operating the XR system are described herein. In some examples, an imaging system, such as an XR system, causes virtual content to be displayed using a display according to a display setting associated with the virtual content. The display setting may, for example, identify a position, orientation, size, color, and / or layout of the virtual content displayed on the display and / or the position, orientation, size, color, and / or layout of the virtual content relative to a portion of an environment viewable through the display. The environment may be viewable through the display when the virtual content is displayed by the display, for example, through a see-through or pass-through display. The imaging system may determine a level of perception for the virtual content displayed by the user using the display through the user's eye or eyes based on one or more attributes of the user's eye or eyes of the imaging system. The imaging system may determine the attributes based on one or more sensors focused on the user, such as one or more cameras facing the user's eye or eyes. The eye position attributes may identify, for example, eye position, eye movement, pupil dilation, saccade, fixation, blinking, squinting, optokinetic reflex or response, vestibulo-ocular reflex or response, accommodation reflex or response, other eye and / or eyelid related attributes described herein, or combinations thereof. The perception level of the virtual content may identify, for example, whether the user perceived the content, and in some cases, may further identify how well the user perceived the content. In some examples, the determination of the perception level of the virtual content by the imaging system may include determining a level of grasping of the virtual content by the user. The level of grasping of the virtual content by the user may be based on a level of perception of the virtual content by the user, one or more characteristics of the virtual content, a user profile of the user, contextual data, or combinations thereof.The one or more characteristics of the virtual content may include the complexity of the virtual content, the uniqueness of the virtual content, the distinctiveness of the virtual content relative to the environment viewable through the display, and / or the like. The user's user profile may include historical data associated with the user and / or the user's current capabilities. The imaging system may determine modifications to the display settings corresponding to the virtual content based on the user's level of perception of the virtual content and / or level of understanding of the virtual content. Modifications to the display settings may, for example, hide, erase, reduce, reduce salience, reduce priority, or terminate the display of the virtual content. Modifications to the display settings may, for example, highlight, enlarge, move, re-orient, increase salience, and / or increase priority of the virtual content.

[0038]

[0052] The XR systems and techniques described herein generally provide numerous technical advantages and benefits over conventional XR and display technologies. For example, the XR systems and techniques described herein provide customization of XR content presentation to a user, including displaying virtual content to the user based on detection of the user's actions, perceptions, and / or grasp. For example, the display of virtual content to the user is customized based on an analysis of the virtual content (e.g., the complexity of the virtual content, the uniqueness of the virtual content, and the uniqueness of the virtual content to the environment), an analysis of the user himself (e.g., based on historical data such as education and occupation and previous actions), and / or an analysis of the context, based on an analysis of the user's level of perception and / or grasp of the virtual content. The XR systems and techniques described herein optimize the use of limited display real estate by prioritizing certain virtual content over views of other virtual content and / or the environment. For example, the XR systems and techniques described herein optimize and highlight important virtual content, such as content that warns the user of impending danger, and minimize or hide less important virtual content, such as content that the user has already seen and / or closed. The XR systems and techniques described herein improve the safety of XR technology by reducing distraction from virtual content when a user needs to focus on critical and potentially dangerous tasks, such as driving, cooking, or surgery, and instead highlighting virtual content that helps the user perform the critical task. The XR systems and techniques described herein improve XR system efficiency by reducing the amount of virtual content displayed and / or the duration for which virtual content is displayed, compared to systems that do not intelligently hide or close virtual content based on the user already perceiving and / or grasping the virtual content.These efficiency gains include reduced bandwidth usage or data to and from the display, reduced power usage by the display, reduced heat generated by the display and / or processor and / or associated components, reduced heat dissipation required for the display and / or processor and / or associated components, or combinations thereof.

[0039]

[0053] Various aspects of the application are described with respect to the figures. FIG. 1 is a block diagram illustrating the architecture of an image capture and processing system 100. The image capture and processing system 100 includes various components used to capture and process images of one or more scenes (e.g., images of a scene 110). The image capture and processing system 100 can capture standalone images (or photographs) and / or can capture videos that include multiple images (or video frames) in a particular order. A lens 115 of the system 100 faces the scene 110 and accepts light from the scene 110. The lens 115 bends the light toward the image sensor 130. The light received by the lens 115 passes through an aperture controlled by one or more control mechanisms 120 and is received by the image sensor 130. In some examples, the scene 110 is a scene in an environment, such as the environment facing the environment-facing sensor 210 of FIG. 2. In some examples, the scene 110 is a scene of at least a portion of a user, such as the user facing the user-facing sensor 205 of FIG. 2. For example, scene 110 may be a scene of one or both of a user's eyes and / or at least a portion of a user's face.

[0040]

[0054] The one or more controls 120 may control exposure, focus, and / or zoom based on information from image sensor 130 and / or based on information from image processor 150. The one or more controls 120 may include multiple mechanisms and components. For example, the control 120 may include one or more exposure controls 125A, one or more focus controls 125B, and / or one or more zoom controls 125C. The one or more controls 120 may include additional controls beyond those shown, such as controls that control analog gain, flash, HDR, depth of field, and / or other image capture properties.

[0041]

[0055] The focus control mechanism 125B of the control mechanism 120 can obtain the focus setting. In some examples, the focus control mechanism 125B stores the focus setting in a memory register. Based on the focus setting, the focus control mechanism 125B can adjust the position of the lens 115 relative to the position of the image sensor 130. For example, based on the focus setting, the focus control mechanism 125B can move the lens 115 closer to or farther from the image sensor 130 by actuating a motor or servo, thereby adjusting the focus. In some cases, additional lenses, such as one or more microlenses above each photodiode of the image sensor 130, may be included in the system 100, each of which bends light received from the lens 115 toward a corresponding photodiode before the light reaches the photodiode. The focus setting may be determined via contrast detection autofocus (CDAF), phase detection autofocus (PDAF), or some combination thereof. The focus settings may be determined using the control mechanism 120, the image sensor 130, and / or the image processor 150. The focus settings may be referred to as image capture settings and / or image processing settings.

[0042]

[0056] The exposure control 125A of the control mechanism 120 can obtain an exposure setting. In some cases, the exposure control 125A stores the exposure setting in a memory register. Based on this exposure setting, the exposure control 125A can control the size of the aperture (e.g., aperture size or f / stop), the duration the aperture is open (e.g., exposure time or shutter speed), the sensitivity of the image sensor 130 (e.g., ISO speed or film speed), the analog gain applied by the image sensor 130, or any combination thereof. The exposure setting may be referred to as an image capture setting and / or an image processing setting.

[0043]

[0057] The zoom control 125C of the control mechanism 120 can obtain the zoom setting. In some examples, the zoom control 125C stores the zoom setting in a memory register. Based on the zoom setting, the zoom control 125C can control the focal length of an assembly of lens elements (lens assembly) including the lens 115 and one or more additional lenses. For example, the zoom control 125C can control the focal length of the lens assembly by actuating one or more motors or servos to move one or more of the lenses relative to each other. The zoom setting may be referred to as an image capture setting and / or an image processing setting. In some examples, the lens assembly may include a parfocal zoom lens or a variable focus zoom lens. In some examples, the lens assembly may include a focusing lens (which may be the lens 115 in some cases) that first accepts light from the scene 110, and then the light passes through an afocal zoom system between the focusing lens (e.g., the lens 115) and the image sensor 130 before the light reaches the image sensor 130. In some cases, an afocal zoom system may include two positive (e.g., converging, convex) lenses of equal or similar focal lengths (e.g., within a threshold difference) with a negative (e.g., diverging, concave) lens between them. In some cases, the zoom control 125C moves one or more of the lenses in the afocal zoom system, such as one or both of the negative and positive lenses.

[0044]

[0058] The image sensor 130 includes one or more arrays of photodiodes or other light-sensitive elements. Each photodiode measures an amount of light that ultimately corresponds to a particular pixel in the image produced by the image sensor 130. In some cases, different photodiodes may be covered by different color filters, and thus may measure light that matches the color of the filter covering the photodiode. For example, a Bayer color filter includes a red color filter, a blue color filter, and a green color filter, and each pixel of the image is generated based on red light data from at least one photodiode covered by a red color filter, blue light data from at least one photodiode covered by a blue color filter, and green light data from at least one photodiode covered by a green color filter. Other types of color filters may use yellow, magenta, and / or cyan (also called "emerald") color filters instead of or in addition to red, blue, and / or green filters. Some image sensors may be completely devoid of color filters and instead use different photodiodes (possibly stacked vertically) across the entire pixel array. Different photodiodes across the pixel array can have different spectral sensitivity curves and therefore respond to different wavelengths of light. Monochrome image sensors may also lack color filters and therefore no color depth.

[0045]

[0059] In some cases, image sensor 130 may alternatively or additionally include an opaque and / or reflective mask that blocks light from reaching some photodiodes or portions of some photodiodes at some times and / or from some angles, which may be used for phase detection autofocus (PDAF). Image sensor 130 may also include an analog gain amplifier for amplifying an analog signal output by the photodiode and / or an analog-to-digital converter (ADC) for converting an analog signal output from the photodiode (and / or amplified by the analog gain amplifier) ​​to a digital signal. In some cases, instead or in addition, some components or functions discussed with respect to one or more of control mechanisms 120 may be included within image sensor 130. The image sensor 130 may be a charge-coupled device (CCD) sensor, an electron-multiplying CCD (EMCCD) sensor, an active-pixel sensor (APS), a complimentary metal-oxide semiconductor (CMOS), an N-type metal-oxide-semiconductor (NMOS), a hybrid CCD / CMOS sensor (e.g., sCMOS), or some other combination thereof.

[0046]

[0060] Image processor 150 may include one or more processors, such as one or more image signal processors (ISP) (including ISP 154), one or more host processors (including host processor 152), and / or one or more of any other types of processors 1110 discussed with respect to computing system 1100. Host processor 152 may be a digital signal processor (DSP) and / or other types of processors. In some implementations, image processor 150 is a single integrated circuit or chip (e.g., referred to as a system-on-chip or SoC) that includes host processor 152 and ISP 154. In some cases, the chip may include one or more input / output ports (e.g., input / output (I / O) ports 156), central processing units (CPUs), graphics processing units (GPUs), broadband modems (e.g., 3G, 4G or LTE, 5G, etc.), memory, connectivity components (e.g., Bluetooth, Global Positioning System (GPS), etc.), any combination thereof, and / or other components.The I / O ports 156 may include any suitable input / output ports or interfaces according to one or more protocols or specifications, such as an Inter-Integrated Circuit 2 (I2C) interface, an Inter-Integrated Circuit 3 (I3C) interface, a Serial Peripheral Interface (SPI) interface, a serial General Purpose Input / Output (GPIO) interface, a Mobile Industry Processor Interface (MIPI) (e.g., a MIPI CSI-2 physical (PHY) layer port or interface, etc.), an Advanced High-performance Bus (AHB) bus, any combination thereof, and / or other input / output ports. In one illustrative example, the host processor 152 may communicate with the image sensor 130 using an I2C port and the ISP 154 may communicate with the image sensor 130 using a MIPI port.

[0047]

[0061] Image processor 150 may perform several tasks such as demosaicing, color space conversion, image frame downsampling, pixel interpolation, automatic exposure (AE) control, automatic gain control (AGC), CDAF, PDAF, automatic white balance, merging image frames to form HDR images, image recognition, object recognition, feature recognition, accepting input, managing output, managing memory, or some combination thereof. Image processor 150 may store image frames and / or processed images in random access memory (RAM) 140 and / or 1120, read-only memory (ROM) 145 and / or 1125, a cache, a memory unit, another storage device, or some combination thereof.

[0048]

[0062] Various input / output (I / O) devices 160 may be connected to image processor 150. I / O devices 160 may include a display screen, a keyboard, a keypad, a touch screen, a track pad, a touch-sensitive surface, a printer, any other output device 1135, any other input device 1145, or any combination thereof. In some cases, captions may be entered into image processing device 105B through a physical keyboard or keypad of I / O device 160 or through a virtual keyboard or keypad of a touch screen of I / O device 160. I / O 160 may include one or more ports, jacks, or other connectors that enable a wired connection between system 100 and one or more peripheral devices, through which system 100 may receive data from and / or send data to one or more peripheral devices. I / O 160 may include one or more wireless transceivers that enable a wireless connection between system 100 and one or more peripheral devices, through which system 100 may receive data from and / or transmit data to one or more peripheral devices. The peripheral devices may include any of the types of I / O devices 160 previously described, and may themselves be considered I / O devices 160 when coupled to a port, jack, wireless transceiver, or other wired and / or wireless connector.

[0049]

[0063] In some cases, image capture and processing system 100 may be a single device. In some cases, image capture and processing system 100 may be two or more separate devices including image capture device 105A (e.g., a camera) and image processing device 105B (e.g., a computing device coupled to a camera). In some implementations, image capture device 105A and image processing device 105B may be coupled, for example, via one or more wires, cables, or other electrical connectors, and / or wirelessly via one or more wireless transceivers. In some implementations, image capture device 105A and image processing device 105B may be separate from one another.

[0050]

[0064] As shown in Figure 1, a vertical dashed line divides the image capture and processing system 100 of Figure 1 into two portions, which respectively represent image capture device 105A and image processing device 105B. Image capture device 105A includes lens 115, control mechanism 120, and image sensor 130. Image processing device 105B includes image processor 150 (including ISP 154 and host processor 152), RAM 140, ROM 145, and I / O 160. In some cases, some components shown in image capture device 105A, such as ISP 154 and / or host processor 152, may be included within image capture device 105A.

[0051]

[0065] The image capture and processing system 100 may include an electronic device, such as a mobile or fixed telephone handset (e.g., a smartphone, a mobile phone, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video gaming console, a video streaming device, an Internet Protocol (IP) camera, or any other suitable electronic device. In some examples, the image capture and processing system 100 may include one or more wireless transceivers for wireless communication, such as cellular network communication, 802.11 wi-fi communication, wireless local area network (WLAN) communication, or any combination thereof. In some implementations, the image capture device 105A and the image processing device 105B may be different devices. For example, the image capture device 105A may include a camera device, and the image processing device 105B may include a computing device, such as a mobile handset, a desktop computer, or other computing device.

[0052]

[0066] Although image capture and processing system 100 is shown as including several components, one skilled in the art will appreciate that image capture and processing system 100 may include many more components than those shown in FIG. 1. The components of image capture and processing system 100 may include software, hardware, or one or more combinations of software and hardware. For example, in some implementations, the components of image capture and processing system 100 may include and / or be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits), and / or may include and / or be implemented using computer software, firmware, or any combination thereof, to perform various operations described herein. The software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of an electronic device implementing image capture and processing system 100.

[0053]

[0067] 2 is a block diagram illustrating an example architecture of an extended reality (XR) system 200 that implements a process for determining a level of perception and / or understanding of virtual content displayed using a display 225 by a user viewing the display 225. In some examples, the XR system 200 includes at least one image capture and processing system 100, an image capture device 105A, an image processing device 105B, or a combination thereof. In some examples, the XR system 200 includes at least one computing system 1100.

[0054]

[0068] The XR system 200 includes one or more user-facing sensors 205. The user-facing sensors 205 capture sensor data that measures and / or tracks information about aspects of the user's body and / or behavior by the user. In some examples, the user-facing sensors 205 include one or more cameras facing at least a portion of the user. The one or more cameras can include one or more image sensors that capture images of at least a portion of the user. For example, the user-facing sensors 205 can include one or more cameras focused on one or both eyes (and / or one or both eyelids) of the user, and the image sensors of the cameras capture images of the user's eye or eyes. The one or more cameras may also be referred to as eye capturing sensors. In some implementations, the one or more cameras can capture a series of images over time, which in some examples can be sequenced together in a temporal order, e.g., into a video. These series of images may depict or otherwise indicate, for example, the user's eye movement, pupil dilation, blinking (using the eyelids), squinting (using the eyelids), saccades, fixations, eye moisture levels, optokinetic reflexes or responses, vestibulo-ocular reflexes or responses, accommodation reflexes or responses, other attributes related to the eyes and / or eyelids described herein, or combinations thereof. In Figure 2, one or more user-facing sensors 205 are shown as cameras facing the user's eyes and capturing images of the user's eyes.The user-facing sensors 205 may include one or more sensors that track information about the user's body and / or behavior, such as cameras, meiji sensors, microphones, heart rate monitors, oximeters, biometric sensors, positioning receivers, Global Navigation Satellite System (GNSS) receivers, inertial measurement units (IMUs), accelerometers, gyroscopes, gyrometers, barometers, thermometers, altimeters, depth sensors, light detection and ranging (LIDAR) sensors, radio detection and ranging (RADAR) sensors, sound detection and ranging (SODAR) sensors, sound navigation and ranging (SONAR) sensors, time of flight (ToF) sensors, structured light sensors, other sensors discussed herein, or combinations thereof. In some examples, the one or more user-facing sensors 205 include at least one image capture and processing system 100, image capture device 105A, image processing device 105B, or a combination thereof. In some examples, the one or more user-facing sensors 205 include or are themselves an input device 1145 of the computing system 1100.

[0055]

[0069] The XR system 200 includes a virtual content generator 215 that generates virtual content. The virtual content can include two-dimensional (2D) shapes, three-dimensional (3D) shapes, 2D objects, 3D objects, 2D models, 3D models, 2D animations, 3D animations, 2D images, 3D images, textures, parts of other images, alphanumeric characters, alphanumeric strings, or combinations thereof. In FIG. 2, the virtual content generated by the virtual content generator 215 is shown as a tetrahedron. Examples of virtual content including alphanumeric strings include virtual content 525, virtual content 530, virtual content 535, virtual content 630, and virtual content 730. In some examples, the virtual content generator 215 includes a software element, such as a set of instructions corresponding to a program executed on a processor, such as the processor 1110, image processor 150, host processor 152, ISP 154, or a combination thereof, of the computing system 1100. In some examples, the virtual content generator 215 includes one or more hardware elements. For example, the virtual content generator 215 may include a processor, such as the processor 1110 of the computing system 1100, the image processor 150, the host processor 152, the ISP 154, or a combination thereof. In some examples, the virtual content generator 215 includes a combination of one or more software elements and one or more hardware elements.

[0056]

[0070] The XR system 200 includes a display 225 that displays virtual content at least partially overlaid on a view of the environment. The view of the environment may include a view of a real-world environment around the XR system 200. The view of the environment may include a view of a virtual environment and / or a mixed environment that is at least partially based on the real-world environment and is at least partially virtual. In some examples, the display 225 can include an output device 1135. In some examples, the output device 1135 can include the display 225.

[0057]

[0071] In some examples, the display 225 of the XR system 200 is an optical "see-through" display that allows light from the real-world environment (scene) around the XR system 200 to pass across (e.g., through) the display 225 and reach one or both of the user's eyes. For example, the display 225 may be at least partially transparent, semi-transparent, light-transmitting, or a combination thereof. In one illustrative example, the display 225 includes a transparent, semi-transparent, and / or light-transmitting lens and a projector. The projector projects virtual content onto the lens. The lens may be, for example, a lens of a pair of glasses, a lens of a goggle, a contact lens, a lens of a head-mounted display (HMD) device, or a combination thereof. Light from the real-world environment passes through the lens and reaches one or both of the user's eyes. Because the projector projects the virtual content onto the lens, the virtual content appears to be overlaid on top of a view of the user's environment from the perspective of one or both of the user's eyes. The positioning of the virtual content projected by the projector onto the lens can be identified and / or indicated by a display setting (e.g., first display setting 250, second display setting 255). The compositor 220 can determine and / or modify the display setting.

[0058]

[0072] In some examples, the display 225 of the XR system 200 includes a projector that does not have a lens as discussed above with respect to an optical see-through display. Instead, the display 225 can use its projector to project virtual content onto one or both of the user's eyes. In some examples, the projector of the display 225 can project virtual content onto one or both retinas of one or both of the user's eyes. Such a display 225 can be referred to as an optical see-through display, a virtual retinal display (VRD), a retinal scan display (RSD), or a retinal projector (RP). Light from the real-world environment (scene) still reaches the user's eye or eyes. Because the projector projects virtual content into the user's eye or eyes, the virtual content appears to be overlaid on the user's view of the environment from the perspective of one or both of the user's eyes. The positioning of the virtual content projected by the projector to one or both eyes of the user can be identified and / or indicated by a display setting (e.g., first display setting 250, second display setting 255). The compositor 220 can determine and / or modify the display setting.

[0059]

[0073] In some examples, the display 225 of the XR system 200 is a digital "pass-through" display that allows the user to see a view of the environment by displaying the view of the environment on the display 225. The view of the environment displayed on the digital pass-through display can be, for example, a view of the real-world environment around the XR system 200 based on (image) sensor data captured by one or more environmental facing sensors 210 of the XR system 200. The view of the environment displayed on the pass-through display can be a view of a virtual or mixed environment that is separate from but based on the real-world environment. For example, the virtual or mixed environment can include virtual objects and / or backgrounds that can be mapped to an area and / or volume of space having dimensions based on the dimensions of the area and / or volume of space in the real-world environment in which the user and the XR system 200 are present. The XR system 200 can determine the dimensions of the area and / or volume of the space in the real-world environment in which the user and the XR system 200 are present. In some implementations, the XR system may include one or more environmental facing sensors 210 in the XR system 200 that capture images of the environment (e.g., the surroundings of the XR system) and / or depth data of the environment. This can ensure that the user does not accidentally go down stairs, bump into a wall or obstacle, or otherwise have a negative and / or potentially dangerous interaction with the real-world environment while exploring the virtual or mixed environment displayed on the display 225.

[0060]

[0074] The XR system 200 can use the compositor 220 to overlay the virtual content generated by the virtual content generator 215 over at least a portion of the environment displayed on the display 225 in examples where the display 225 is a digital pass-through display. In some examples, the compositor 220 can overlay the virtual content completely over the environment displayed on the display 225 such that the virtual content appears to be completely in front of the remainder of the environment displayed on the display 225 from the perspective of one or both eyes of a user viewing the display 225. In some examples, the compositor 220 can overlay at least a portion of the virtual content over portions of the environment displayed on the display 225 such that the virtual content appears to be in front of some portions of the environment displayed on the display 225 but behind other portions of the environment displayed on the display 225 from the perspective of one or both eyes of a user viewing the display 225. Thus, the compositor 220 can provide simulated depth to the virtual content and overlay portions of the environment displayed on the display 225 over portions of the virtual content. An example of this simulated depth is shown in FIG. 5B, where the head of the Red Auerbach statue is partially overlaid on top of a portion of the virtual content 530 according to display settings 555.

[0061]

[0075] The XR system 200 can use the compositor 220, in examples where the display 225 is an optical see-through display, to prevent portions of the real-world environment from being overlaid by the virtual content generated by the virtual content generator 215. In some examples, the compositor 220 can only partially overlay the virtual content onto the real-world environment on the display such that the virtual content appears to be behind at least a portion of the real-world environment from the perspective of one or both eyes of a user viewing the display 225. In some examples, the compositor 220 can only partially overlay the virtual content onto the real-world environment on the display such that the virtual content appears to be behind at least a portion of the real-world environment and in front of other portions of the real-world environment from the perspective of one or both eyes of a user viewing the display 225. Thus, the compositor 220 can provide simulated depth to the virtual content and prevent portions of the real-world environment from being overlaid by the virtual content. The positioning of the virtual content relative to the environment can be identified and / or indicated by a display configuration (e.g., the first display configuration 250, the second display configuration 255). The compositor 220 can determine and / or modify the display settings.

[0062]

[0076] The one or more environmental facing sensors 210 of the XR system 200 are one or more sensors pointed, oriented, and / or focused away from the user and / or at a portion of the real-world environment. For example, the one or more environmental facing sensors 210 can be pointed, oriented, and / or facing in the direction in which the user and / or the front of the XR system 200 faces. The environmental facing sensors 210 capture sensor data that measures and / or tracks information about the real-world environment. In some examples, the environmental facing sensors 210 include one or more cameras facing at least a portion of the real-world environment. The one or more cameras can include one or more image sensors that capture images of at least a portion of the real-world environment. For example, the environmental facing sensors 210 can include one or more cameras focused on the real-world environment (e.g., the surroundings of the XR system 200), and the image sensors of the cameras capture images of the real-world environment (e.g., the surroundings). Such a camera can capture a series of images over time, which in some examples can be sequenced together in a time order, e.g., into a video. These series of images can depict or otherwise show, for example, a floor, a ground, a wall, a ceiling, the sky, water, plants, other people other than the user, parts of the user's body (e.g., an arm or leg), a structure, a vehicle, an animal, a device, other objects, or combinations thereof. In FIG. 2, the one or more environmental facing sensors 210 are shown as cameras facing a house (an example of a structure). In some examples, the one or more environmental facing sensors 210 include at least one image capture and processing system 100, an image capture device 105A, an image processing device 105B, or a combination thereof. In some examples, the one or more environmental facing sensors 210 include at least one input device 1145 of the computing system 1100, or are themselves an input device 1145 of the computing system 1100.

[0063]

[0077] The environmental facing sensors 210 may include cameras, image sensors, positioning receivers, Global Navigation Satellite System (GNSS) receivers, Inertial Measurement Units (IMUs), accelerometers, gyroscopes, gyrometers, barometers, thermometers, altimeters, depth sensors, Light Detection and Ranging (LIDAR) sensors, Radio Detection and Ranging (RADAR) sensors, Sound Detection and Ranging (SODAR) sensors, Acoustic Navigation and Ranging (SONAR) sensors, Time of Flight (ToF) sensors, structured light sensors, other sensors discussed herein, or combinations thereof.

[0064]

[0078] In some examples, the XR system 200 may also include one or more additional sensors 208, such as cameras, image sensors, positioning receivers, Global Navigation Satellite System (GNSS) receivers, Inertial Measurement Units (IMUs), accelerometers, gyroscopes, gyrometers, barometers, thermometers, altimeters, depth sensors, Light Detection and Ranging (LIDAR) sensors, Radio Detection and Ranging (RADAR) sensors, Sound Detection and Ranging (SODAR) sensors, Acoustic Navigation and Ranging (SONAR) sensors, Time of Flight (ToF) sensors, structured light sensors, other sensors discussed herein, or combinations thereof, etc. In some implementations, the additional sensors 208 may supplement or refine sensor readings from the user-facing sensors 205 and / or the environment-facing sensors 210. For example, inertial measurement units (IMUs), accelerometers, gyroscopes, or other sensors may be used by the attribution engine 230 to refine a determination of the user's perception of the virtual content (e.g., by detecting a head shake or nod by the user). In another example, depth sensors, light detection and ranging (LIDAR) sensors, radio detection and ranging (RADAR) sensors, sound detection and ranging (SODAR) sensors, acoustic navigation and ranging (SONAR) sensors, time-of-flight (ToF) sensors may be used by the synthesizer 220 to identify portions of the real-world environment (e.g., identifiable objects) that should not be overlaid by the display when creating the simulated depth of the virtual content.

[0065]

[0079] The XR system 200 includes a compositor 220. The compositor 220 composes, synthesizes, and / or combines a view of the virtual content in the environment that the user views through the display 225. The compositor 220 of the XR system 200 can determine a first set of display settings for the display 225 (e.g., first display setting 250). The compositor 220 of the XR system 200 can modify the first set of display settings for the display 225 to generate a second set of display settings for the display 225 (e.g., second display setting 255). In an XR system 200 where the display 225 is a digital "pass-through" display, the compositor 220 can generate an image that synthesizes, synthesizes, and / or combines a view of the environment with the virtual content generated by the virtual content generator 215 (e.g., based on sensor data from the environment facing sensor 210). The display settings generated by the compositor 220 can indicate the position, orientation, depth, size, color, font size, font color, text language, layout, and / or other properties of the virtual content and / or of particular elements or portions of the virtual content. In an XR system 200 where the display 225 is an optical "see-through" display, the compositor 220 can generate display settings that indicate the position, orientation, depth, size, color, font size, font color, text language, and / or other properties of the virtual content displayed by the display 225 (e.g., projected onto a lens by a projector of the display 225) and / or of particular elements or portions of the virtual content. In an XR system 200 where the display 225 is a virtual retinal display (VRD), the compositor 220 can generate display settings that indicate the position, orientation, depth, size, color, font size, font color, text language, and / or other properties of the virtual content and / or particular elements or portions of the virtual content displayed by the display 225 (e.g., projected by a projector of the display 225 to one or both of the user's eyes).In FIG. 2, the compositor 220 (on the left side of the XR system 200) is shown as adding virtual content (represented by tetrahedrons) to a view of the environment (represented by the house). In FIG. 2, the display 225 (on the left side of the XR system 200) is shown as a display that displays and / or provides both a view of the virtual content (represented by tetrahedrons) and a view of the environment (represented by the house). In some examples, the compositor 220 includes a software element, such as a set of instructions corresponding to a program, executed on a processor, such as the processor 1110, the image processor 150, the host processor 152, the ISP 154, or a combination thereof, of the computing system 1100. In some examples, the compositor 220 includes one or more hardware elements. For example, the compositor 220 may include a processor, such as the processor 1110, the image processor 150, the host processor 152, the ISP 154, or a combination thereof, of the computing system 1100. In some examples, the synthesizer 220 includes a combination of one or more software elements and one or more hardware elements.

[0066]

[0080] The XR system 200 includes an attribute engine 230 that determines one or more perception-related attributes based on sensor data from the user-facing sensor 205. The user's perception-related attributes can include one or more attributes of the user's eye or eyes, one or more attributes of the user's facial expression, one or more gestures of the user, or a combination of the above. For example, the user's attributes can include a position of the user's eye or eyes at a particular time, a movement of the user's eye or eyes, a position of the user's saccade of one eye or eyes, a movement of the user's saccade of one eye or eyes, a number of the user's saccades of one eye or eyes, a frequency of the user's saccade of one eye or eyes, a duration of the user's saccade of one eye or eyes, a position of the user's fixation of one eye or eyes, a movement of the user's fixation of one eye or eyes, a number of the user's fixations of one eye or eyes, fixation frequency of one or both eyes of the user, fixation duration of one or both eyes of the user, blink position of one or both eyes and / or eyelids of the user, blink movement of one or both eyes and / or eyelids of the user, blink count of one or both eyes and / or eyelids of the user, blink frequency of one or both eyes and / or eyelids of the user, blink duration of one or both eyes and / or eyelids of the user, blink position of one or both eyes and / or eyelids of the user and / or eyelid squinting position, eye and / or eyelid squinting movement of one or both of the user's eyes and / or eyelids, number of times the user's eyes and / or eyelids squint, frequency of the user's eyes and / or eyelids squinting, duration of the user's eyes and / or eyelids squinting, degree or level of the user's eyes and / or eyelids squinting, gaze position of one or both of the user's eyes, the number of gazes of both eyes, the frequency of gazes of one or both eyes of the user, the duration of gaze of one or both eyes of the user, the degree or level of gaze of one or both eyes of the user, the eye position of pupil dilation of one or both eyes of the user, the eye movement of pupil dilation of one or both eyes of the user, the number of pupil dilations of one or both eyes of the user, the frequency of pupil dilation of one or both eyes of the user, the duration of pupil dilation of one or both eyes of the user, the degree or level of pupil dilation of one or both eyes of the user, the lens shape of one or both eyes of the user (e.g.,a change in the lens shape of one or both of the user's eyes (e.g., associated with accommodation using the ciliary muscle), ... degree or level of head tilt, eye position of the user's one or both eyes while the user tilts his / her head, eye movement of the user's one or both eyes while the user tilts his / her head, number of times the user shakes his / her head, frequency of the user's head shaking, duration of the user's head shaking, degree or level of the user's head shaking, number of times the user shakes his / her head, frequency of the user's head shaking, duration of the user's head shaking, degree or level of the user's head shaking, eye position of the user's one or both eyes while the user shakes his / her head, eye movement of the user's one or both eyes while the user shakes his / her head, nodding of the user's head number of user's head nods, frequency of user's head nods, duration of user's head nods, degree or level of user's head nods, number of user's head nods, frequency of user's head nods, duration of user's head nods, degree or level of user's head nods, eye position of one or both of the user's eyes during the user's head nods, eye movement of one or both of the user's eyes during the user's head nods, number of user's smiles, frequency of user's smiles, duration of user's smiles, degree or level of user's smiles, number of user's smiles, frequency of user's smiles, duration of user's smiles , degree or level of a smile by the user, eye position of the user's eye or eyes while the user is smiling, eye movement of the user's eye or eyes while the user is smiling, number of times the user laughs, frequency of the user laughs, duration of the user laughs, degree or level of a laughter by the user, number of times the user laughs, frequency of the user laughs, duration of the user laughs, degree or level of a laughter by the user, eye position of the user's eye or eyes while the user is laughing, eye movement of the user's eye or eyes while the user is laughing, number of times the user squints,frequency of a user squinting, duration of a user squinting, degree or level of a user squinting, number of times a user squints, frequency of a user squinting, duration of a user squinting, degree or level of a user squinting, eye position of one or both eyes of the user while the user squints, eye movement of one or both eyes of the user while the user squints, number of times a user cries, frequency of a user cried, duration of a user cried, degree or level of a user cried, number of times a user cried, frequency of a user cried, duration of a user cried, degree or level of a user cried, eye position of one or both eyes of the user while the user cries, eye movement of one or both eyes of the user while the user cries, moisture level of one or both eyes of the user, dryness level of one or both eyes of the user, optokinetic reflex or response of one or both eyes of the user. eye position, eye movement of the user's monocular or binocular optokinetic reflex or response, number of the user's monocular or binocular optokinetic reflex or response, frequency of the user's monocular or binocular optokinetic reflex or response, duration of the user's monocular or binocular optokinetic reflex or response, eye position of the user's monocular or binocular vestibular reflex or response, eye movement of the user's monocular or binocular vestibular reflex or response, number of the user's monocular or binocular vestibular reflex or response, vestibular eye of the user's monocular or binocular The information may include the frequency of the reflex or response, the duration of the vestibulo-ocular reflex or response of one or both of the user's eyes, other attributes related to the eye and / or eyelid described herein, the eye position of the accommodation reflex or response of one or both of the user's eyes, the eye movement of the accommodation reflex or response of one or both of the user's eyes, the number of accommodation reflexes or responses of one or both of the user's eyes, the frequency of the accommodation reflex or response of one or both of the user's eyes, the duration of the accommodation reflex or response of one or both of the user's eyes, or any combination thereof.

[0067]

[0081] In FIG. 2 , the attribute engine 230 is shown as identifying the movement (represented by the dotted arrow) of the gaze direction (represented by the solid black arrow) of the user's eyes. In some examples, the attribute engine 230 includes a software element, such as a set of instructions corresponding to a program executing on a processor, such as the processor 1110, the image processor 150, the host processor 152, the ISP 154, or a combination thereof, of the computing system 1100. In some examples, the attribute engine 230 includes one or more hardware elements. For example, the attribute engine 230 may include a processor, such as the processor 1110, the image processor 150, the host processor 152, the ISP 154, or a combination thereof, of the computing system 1100. In some examples, the attribute engine 230 includes a combination of one or more software elements and one or more hardware elements. In some examples, the attribute engine 230 includes and / or executes one or more artificial intelligence (AI) algorithms and / or one or more machine learning (ML) systems. The one or more AI algorithms and / or ML systems can receive sensor data from the user-facing sensors 205 as input and can output attributes. Examples of the one or more ML systems of the attribute engine 230 include the first trained ML model 825 and / or the neural network 900.

[0068]

[0082] The XR system 200 includes a perception engine 235 that determines and / or estimates a level of perception of the virtual content by the user (e.g., using one or both of the user's eyes). The perception engine 235 determines and / or estimates the user's level of perception of the virtual content based on perception-related attributes identified by the attribute engine 230 and / or based directly on sensor data from the user focus sensor 205. The perception engine 235 determines and / or estimates the user's level of perception of the virtual content based on a display setting (e.g., display setting 250) for the virtual content as generated by the compositor 220. The perception engine 235 can use the display setting (e.g., display setting 250) to identify where the virtual content is to be displayed along the display 225. The perception engine 235 can use the perception-related attributes identified by the attribute engine 230 to determine where the user's eye or eyes are looking, how the user's eye or eyes are moving, the extent to which the user's eye or eyes are squinting while looking, the extent to which the user's eye or eyes are blinking, the extent to which the user's eye or eyes are fixating, the extent to which the user's eye or eyes are making a saccade, the extent to which the user is smiling, the extent to which the user is laughing, the extent to which the user is frowning, the extent to which the user is crying, or a combination thereof. The perception engine 235 can determine whether the user looked at and / or near the virtual content, how long the user looked at and / or near the virtual content, how closely the user's gaze is aligned with the location of the virtual content, etc. The perception engine 235 may interpret, for example, squinting and / or fixation as an indication that the user is focusing on the virtual content and / or another content displayed on the display 225. The perception engine 235 may interpret, for example, blinks and / or saccades as an indication that the user is potentially missing (e.g., not perceiving) virtual content and / or other content displayed on the display 225.The level of perception of the virtual content by the user output by the perception engine 235 may be a value indicating no perception of the virtual content by the user, for example, if the user closes his / her eyes and / or looks very far away from the location of the virtual content while the virtual content is displayed on the display 225. The level of perception of the virtual content by the user output by the perception engine 235 may be a value indicating very high perception of the virtual content by the user, for example, if the user gazes at the location where the virtual content is displayed for a long time. The level of perception of the virtual content by the user output by the perception engine 235 may be a value between a value indicating a higher level of perception than the above example of no perception and a value indicating a lower level of perception than the above example of high perception. The level of perception of the virtual content determined and output by the perception engine 235 may be referred to as a degree of perception of the virtual content and / or a metric of perception of the virtual content.

[0069]

[0083] In FIG. 2, the perception engine 235 is shown as identifying whether the gaze direction of the user's eyes (represented by the solid black arrow) matches the location of the virtual content (represented by the tetrahedron). In some examples, the perception engine 235 includes a software element, such as a set of instructions corresponding to a program, executing on a processor, such as the processor 1110, the image processor 150, the host processor 152, the ISP 154, or a combination thereof, of the computing system 1100. In some examples, the perception engine 235 includes one or more hardware elements. For example, the perception engine 235 can include a processor, such as the processor 1110, the image processor 150, the host processor 152, the ISP 154, or a combination thereof, of the computing system 1100. In some examples, the perception engine 235 includes a combination of one or more software elements and one or more hardware elements. In some examples, the perception engine 235 includes and / or executes one or more AI algorithms and / or one or more ML systems. The one or more AI algorithms and / or one or more ML systems can receive as input the one or more perception-related attributes generated by the attribute engine 230 (and / or the sensor data captured by the user-facing sensors 205) and can output a level of perception of the virtual content by the user. Examples of the one or more ML systems of the perception engine 235 include a second trained ML model 835 and / or a neural network 900.

[0070]

[0084] The XR system 200 includes a comprehension engine 240 that determines and / or estimates the user's level of comprehension and / or understanding of the virtual content. The comprehension engine 240 determines and / or estimates the user's level of comprehension and / or understanding of the virtual content based on the user's level of perception of the virtual content determined by the perception engine 235. In some examples, the comprehension engine 240 determines and / or estimates the user's level of comprehension and / or understanding of the virtual content based on an analysis of the virtual content by the virtual content evaluation engine 245. In some examples, the comprehension engine 240 determines and / or estimates the user's level of comprehension and / or understanding of the virtual content based on historical data associated with the user.

[0071]

[0085] In some examples, the XR system 200 includes a virtual content rating engine 245. The virtual content rating engine 245 generates an analysis of the virtual content. In some examples, the virtual content rating engine 245 generates an analysis of the virtual content based on the complexity of the virtual content, the uniqueness of the virtual content, or a combination thereof. In some examples, the virtual content rating engine determines a metric based on the uniqueness. The uniqueness may be a measure of how often (e.g., how many times and / or how frequently) the virtual content has been displayed (e.g., previously) on the display 225. If the virtual content includes a message or object that is displayed very frequently on the display 225, the virtual content may be determined to have low uniqueness. If the virtual content includes a message or object that has never been displayed on the display 225 before, the virtual content rating engine 245 may determine that the virtual content has high uniqueness. If the virtual content includes a message or object that has been displayed infrequently (e.g., once or twice previously) on the display 225, the virtual content rating engine 245 may determine that the virtual content has a medium degree of uniqueness. The uniqueness may be within a range of possible values ​​(e.g., low, medium, high, and one or more values ​​between any two of these). In some examples, the uniqueness may be referred to as a uniqueness score, a uniqueness level, a uniqueness metric, or a degree of uniqueness.

[0072]

[0086] In some examples, the virtual content includes an alphanumeric string (e.g., a string of text). The virtual content rating engine 245 can evaluate the complexity of the virtual content based at least in part on the length of the string, with longer strings having a higher complexity than shorter strings. The virtual content rating engine 245 can generate a complexity metric for the virtual content based at least in part on the number of steps included in a set of instructions in the virtual content. For example, the virtual content rating engine 245 can provide a higher complexity metric for virtual content that includes instructions with more steps and a lower complexity metric for virtual content that includes instructions with fewer steps. The virtual content rating engine 245 can analyze the complexity of the virtual content based at least in part on the complexity of the words included in the string, as determined by the virtual content rating engine 245. For example, the virtual content rating engine 245 can assign a higher complexity to highly technical terms (e.g., "semiconductor," "exhaust gasket," "anti-siphon valve") than to more common everyday words (e.g., "today," "stop," "continue"). The virtual content rating engine 245 may assign or provide a higher complexity to a string that includes more complex words than a string that includes fewer complex words and / or more non-complex and / or less complex words (e.g., common and / or simple words). The virtual content rating engine 245 may analyze the complexity of the virtual content based at least in part on the complexity of any equations or formulas included in the string as determined by the virtual content rating engine 245. For example, the virtual content rating engine 245 may assign a higher complexity to an equation or formula that includes more variables and / or corresponds to a higher level of mathematical difficulty than an equation or formula that includes fewer variables and / or corresponds to a lower level of mathematical difficulty.The virtual content rating engine 245 may assign a higher complexity to strings that contain more complex equations or expressions than strings that contain fewer complex equations or expressions and / or more equations or expressions that are less complex. Complexity may be within a range of possible values ​​(e.g., low, medium, high, and one or more values ​​between any two of these). In some examples, complexity may be referred to as a complexity score, a complexity level, a complexity metric, or a complexity degree.

[0073]

[0087] The virtual content rating engine 245 can generate a complexity metric for the virtual content based at least in part on an estimated amount of prior knowledge required to understand or grasp the subject matter described in the virtual content. For example, the virtual content rating engine 245 can assign a higher complexity to virtual content that includes subject matter associated with a high level of prior knowledge and a lower complexity to virtual content that includes subject matter requiring a lower level of prior knowledge. The high and low levels of prior knowledge can be associated, for example, with courses in which the subject matter may typically be taught and / or learned. For example, a subject matter requiring a lower level of prior knowledge may be a subject matter typically taught and / or learned in third grade, while a subject matter requiring a higher level of prior knowledge may be a subject matter typically taught and / or learned in graduate school (e.g., a particular graduate course that has another graduate course and a number of undergraduate courses as prerequisites). For example, virtual content that describes differential equations or Banach spaces requires a higher level of prior knowledge than virtual content that describes Pythagoras' theorem. The virtual content rating engine 245 can determine the subject matter of the virtual content based on words contained in the virtual content. The virtual content assessment engine 245 can compare the subject matter to a look-up table that identifies a corresponding level of prior knowledge associated with the subject matter.

[0074]

[0088] The virtual content rating engine 245 may generate a complexity metric for the virtual content based at least in part on the complexity of one or more shapes, objects, images, and / or textures of the virtual content. For example, the virtual content rating engine 245 may assign a higher complexity to virtual content that includes more complex polygons and / or polyhedrons than to virtual content that includes less complex polygons and / or polyhedrons. In FIG. 2, the virtual content rating engine 245 is shown as identifying complexities ranging from low complexity (represented by triangles), to medium complexity (represented by tetrahedrons), to high complexity (represented by stellated dodecahedrons).

[0075]

[0089] In some cases, to rate the virtual content, the virtual content rating engine 245 can determine a complexity metric and a uniqueness metric for the virtual content. The virtual content rating engine 245 can convert the complexity metric and the uniqueness metric into a single metric for the virtual content. The single metric can be a fusion metric based on the complexity metric and the uniqueness metric, and can be a product, sum, or average of the complexity metric and the uniqueness metric. In one illustrative example, the virtual content can include a notification that reads, "Warning: Hot Stove!", in which case the virtual content rating engine 245 can assign a complexity metric with a value of 1 and a uniqueness metric with a value of 2. The virtual content rating engine 245 can determine a single metric with a value of 1×2=2. In another illustrative example, the virtual content can include a notification that reads, "Add salt and stir the roux until the mixture is bubbly and foamy," in which case the virtual content rating engine 245 can determine that the complexity metric has a value of 3 and that the uniqueness metric has a value of 5. The virtual content rating engine 245 can determine a single metric with a value of 3×5=15.

[0076]

[0090] The historical data associated with a user may include the user's level of education, such as a high school, a high school graduate, a college, a college graduate, a graduate school, a post-graduate school, a graduate school graduate, etc. The historical data associated with a user may include the user's area of ​​education, such as a major, a minor, a field of study, a course of study, a school or program (e.g., engineering, life sciences), etc. The historical data associated with a user may include the user's job, career, occupation, and / or profession. The historical data associated with a user may include the user's transaction history, the user's browsing history, the user's game play history, the user's virtual content viewing history, the user's location history, the user's residence history, etc. The historical data associated with a user may include the user's age, time of day (which may indicate, for example, whether the user is tired), the user's understanding or comprehension of previously presented virtual content, the user's feedback (e.g., via the feedback engine 260) for previously presented virtual content, the user's successful completion of actions requested or suggested by previously presented virtual content, etc.

[0077]

[0091] The comprehension engine 240 determines and / or estimates the user's level of comprehension and / or understanding of the virtual content based on the user's level of perception of the virtual content determined by the perception engine 235, an analysis of the virtual content by the virtual content rating engine 245, historical data associated with the user, or a combination thereof. For example, if the perception engine 235 indicates that the user viewed the virtual content for 10 seconds, the virtual content received a low uniqueness and low complexity rating from the virtual content rating engine 245, and the virtual content relates to a subject about which the user is highly familiar based on the user's historical data, the comprehension engine 240 may determine and / or estimate that the user has comprehended and / or understood the virtual content to a high level of comprehension and / or understanding. On the other hand, if the perception engine 235 indicates that the user viewed the virtual content for 5 seconds, the virtual content received a high uniqueness and high complexity rating from the virtual content rating engine 245, and the virtual content is not related to a subject with which the user is familiar based on the user's historical data, the comprehension engine 240 may determine and / or infer that the user has comprehended and / or understood the virtual content to a low comprehension and / or understanding level. The comprehension and / or understanding level of the virtual content determined and output by the comprehension engine 240 may be referred to as a degree of comprehension and / or understanding of the virtual content and / or a metric of the comprehension and / or understanding of the virtual content.

[0078]

[0092] Different combinations of perception levels, ratings from the virtual content rating engine 245, and historical data associated with the user may cause the comprehension engine 240 to determine and / or estimate different levels of comprehension and / or understanding of the virtual content by the user. For example, if the virtual content relates to a subject with which the user is highly familiar based on the user's historical data, the comprehension engine 240 may determine and / or estimate that the user has comprehended and / or understood the virtual content to a reasonably high level of comprehension and / or understanding, even if the virtual content received a high uniqueness and high complexity rating from the virtual content rating engine 245 and / or the perception engine 235 indicates that the user viewed the virtual content for a relatively short period of time. If the perception engine 235 indicates that the user viewed the virtual content for an extended period of time, the comprehension engine 240 may determine and / or estimate that the user has comprehended and / or understood the virtual content to a reasonably high level of comprehension and / or understanding, even if the virtual content received a high uniqueness and high complexity rating from the virtual content rating engine 245 and / or the virtual content is not related to a subject with which the user is familiar based on the user's historical data. If the perception engine 235 indicates that the virtual content evaluation engine 245 has evaluated the virtual content as having low uniqueness and / or low complexity, the comprehension engine 240 may determine and / or infer that the user has grasped and / or understood the virtual content to a reasonably high level of comprehension and / or understanding, even if the perception engine 235 indicates that the user has viewed the virtual content relatively briefly and / or the virtual content does not relate to a subject with which the user is familiar based on the user's historical data.

[0079]

[0093] In some examples, the comprehension engine 240 may determine and / or estimate a user's level of comprehension and / or understanding of the virtual content based on the perception-related attributes determined by the attribute engine 230. For example, the comprehension engine 240 may increase its determination and / or estimation of the level of comprehension and / or understanding of the virtual content if the perception-related attributes determined by the attribute engine 230 indicate that the user appears to be reacting to the virtual content, for example, by smiling, laughing, frowning, crying, nodding, shaking his head, tilting his head, or saying something related to the virtual content. In some examples, the comprehension engine 240 may decrease its determination and / or estimation of the level of comprehension and / or understanding of the virtual content if the perception-related attributes determined by the attribute engine 230 indicate that the user appears to be reacting negatively to the virtual content, for example, by frowning, crying, shaking his head (e.g., in a "no" motion), or verbally expressing negative emotions, as these may indicate that the user is upset or frustrated due to the user's inability to understand the virtual content at a high level.

[0080]

[0094] In some examples, the comprehension engine 240 can determine and / or estimate the user's level of comprehension and / or understanding of the virtual content based on the context data. The context data can include, for example, a reaction by the user to the virtual content. In some examples, the comprehension engine 240 can increase its determination and / or estimate of the user's level of comprehension and / or understanding of the virtual content based on an identification that the user performed an action that the virtual content requests or suggests the user to perform. For example, if the virtual content requests the user to pick up an object and the XR system 200 determines (e.g., based on sensor data from the user-facing sensors 205 and / or the environment-facing sensors 210) that the user picked up the object, the comprehension engine 240 can determine and / or estimate that the user has comprehended and / or understood the virtual content to a high level of comprehension and / or understanding. In some examples, the comprehension engine 240 can decrease its determination and / or estimate of the user's level of comprehension and / or understanding of the virtual content based on an identification that the user performed an action opposite to what the virtual content requests or suggests the user to perform. For example, if the virtual content provides driving instructions requesting the user to turn onto a particular street and the XR system 200 determines (e.g., based on sensor data from user-facing sensors 205 and / or environmental-facing sensors 210) that the user has not turned onto that street, the comprehension engine 240 may determine and / or infer that the user has grasped and / or understood the virtual content to a low comprehension and / or understanding level.

[0081]

[0095] The context data may include, for example, the location of the user and / or other objects detected in the environment, the location of the XR system 200, the state of the XR system 200 (e.g., low battery or high battery), the time of day, user input received through the user interface of the XR system 200, previous virtual content displayed by the XR system, the resolution of the display 225, the speed at which the user and / or the XR system 200 is moving, whether the environment around the XR system 200 is static or dynamic, environmental obstacle detection, environmental noise level, whether a second person is speaking to the user, or any combination thereof. In some examples, the comprehension engine 240 may provide a higher determination and / or estimation of the user's level of comprehension and / or understanding of the virtual content when the user is walking rather than driving because the user may be more distracted by the virtual content while driving than while walking. The XR system 200 may delay the display of the virtual content until the user stops driving in order to improve safety in some cases. In some examples, the comprehension engine 240 may provide a higher determination and / or estimation of the user's level of comprehension and / or understanding of the virtual content when the environmental noise level around the user and / or XR system 200 is lower than when the environmental noise level is higher, because a lower environmental noise level suggests that the user is less distracted by the virtual content than a higher environmental noise level. In some examples, the comprehension engine 240 may provide a lower determination and / or estimation of the user's level of comprehension and / or understanding of the virtual content when a second person is speaking to the user than when there is no second person speaking to the user, because a second person speaking to the user distracts the user from the virtual content. In some examples, the comprehension engine 240 may provide a higher determination and / or estimation of the user's level of comprehension and / or understanding of the virtual content when the user and / or XR system 200 is moving at a slower speed than a faster speed.Because a user may be more distracted by the virtual content when traveling at a faster speed than while traveling at a slower speed. The XR system 200 may delay the display of the virtual content until the user slows down to a slower speed to improve safety when the user is driving or otherwise operating a vehicle. In some examples, the comprehension engine 240 may provide a higher determination and / or estimation of the user's level of comprehension and / or understanding of the virtual content during the afternoon than during the early morning or late night because the user is more likely to be less fatigued during the afternoon than during the early morning or late night.

[0082]

[0096] The user's level of understanding or appreciation of the virtual content output by the comprehension engine 240 may be a value indicative of no understanding or appreciation of the virtual content by the user, for example, if the user closes his / her eyes and / or looks very far away from the location of the virtual content while the virtual content is displayed on the display 225. The user's level of understanding or appreciation of the virtual content output by the comprehension engine 240 may be a value indicative of a very high perception of the virtual content by the user, for example, if the user gazes at the location where the virtual content is displayed for a long period of time, the virtual content has low complexity, the virtual content has low uniqueness, and the historical data associated with the user indicates that the user is very familiar with the subject matter to which the virtual data pertains. The user's level of appreciation of the virtual content output by the comprehension engine 240 may be a value between a value indicative of a higher level of appreciation than the no appreciation or appreciation example above and a value indicative of a lower level of appreciation than the high appreciation or appreciation example above.

[0083]

[0097] In FIG. 2, the comprehension engine 240 is shown as identifying whether the user comprehends or understands the virtual content, represented by a dotted arrow from the virtual content (represented by a tetrahedron) to the user's mind. In some examples, the comprehension engine 240 includes a software element, such as a set of instructions corresponding to a program, executing on a processor, such as the processor 1110, the image processor 150, the host processor 152, the ISP 154, or a combination thereof, of the computing system 1100. In some examples, the comprehension engine 240 includes one or more hardware elements. For example, the comprehension engine 240 can include a processor, such as the processor 1110, the image processor 150, the host processor 152, the ISP 154, or a combination thereof, of the computing system 1100. In some examples, the comprehension engine 240 includes a combination of one or more software elements and one or more hardware elements. In some examples, the comprehension engine 240 includes and / or executes one or more AI algorithms and / or one or more ML systems. The one or more AI algorithms and / or one or more ML systems can receive as input the one or more perception-related attributes generated by the attribute engine 230 (and / or the sensor data captured by the user focus sensor 205) and can output a perception level for the virtual content. Examples of the one or more ML systems of the perception engine 240 include a third trained ML model 865 and / or a neural network 900.

[0084]

[0098] Both the compositor 220 and the display 225 are shown twice in the XR system 200 of FIG. 2, once on the left side of the XR system 200 and once on the right side of the XR system 200. It should be understood that these two instances of the compositor 220 and the display 225 may represent the same compositor 220 and / or the same display 225 at different times. For example, the compositor 220 and the display 225 shown on the left side of the XR system 200 represent the compositor 220 generating a first display setting 250 and the display 225 displaying virtual content overlaid on a view of the environment based on the first display setting 250. The compositor 220 and the display 225 shown on the right side of the XR system 200 represent the compositor 220 generating a second display setting 255 by at least partially modifying the first display setting 250 and the display 225 displaying virtual content overlaid on a view of the environment based on the second display setting 255.

[0085]

[0099] The compositor 220 may perform modifications of the first display configuration 250 to generate the second display configuration 255 based on one or more factors. These factors may include, for example, the user's level of perception of the virtual content as determined by the perception engine 235, the user's level of understanding and / or comprehension of the virtual content as determined by the comprehension engine 240, the user's perception-related attributes as determined by the attribute engine 230, historical data about the user, contextual data, sensor data from the user-facing sensors 205, sensor data from the environmental facing sensors 210, or combinations thereof. The modifications of the first display configuration 250 by the compositor 220 to generate the second display configuration 255 may change aspects of the virtual content. For example, the modifications may change the position, orientation, depth, size, color, font size, font color, text language, and / or other properties of the virtual content and / or of particular elements or portions of the virtual content. In some examples, the modifications may remove, erase, hide, and / or terminate the display of the virtual content on the display 225. In some cases, the modifications may add additional virtual content for display on display 225. In one illustrative example, if perception engine 235 indicates that the user has perceived the virtual content to a high level and / or comprehension engine 240 indicates that the user has comprehended the virtual content to a high level, compositor 220 may scale back the virtual content displayed on display 225 or even terminate display of the virtual content on display 225.On the other hand, if the perception engine 235 indicates that the user has perceived the virtual content to a low level, and / or the perception-related attributes from the attribute engine 230 indicate that the user has perceived the virtual content to a low level, but the perception-related attributes from the attribute engine 230 indicate that the user is attempting to perceive and / or comprehend the virtual content, the compositor 220 may increase the size displayed on the display 225, the virtual content, and / or increase the font size of alphanumeric strings in the virtual content, and may reduce and / or terminate the display of other virtual content on the display 225 to reduce user distraction.

[0086]

[0100] In Fig. 2, the left compositor 220 of the XR system 200 is shown as adding virtual content (represented by tetrahedrons) to a view of an environment (represented by a house) according to a first display configuration 250 in which the virtual content (represented by tetrahedrons) is small. In Fig. 2, the left display 225 of the XR system 200 is shown as a display that displays and / or provides a view of both the virtual content (represented by tetrahedrons) and the view of the environment (represented by a house) according to a first display configuration 250 in which the virtual content (represented by tetrahedrons) is small. In Fig. 2, the right compositor 220 of the XR system 200 is shown as adding virtual content (represented by tetrahedrons) to a view of an environment (represented by a house) according to a second display configuration 255 in which the virtual content (represented by tetrahedrons) is large. In FIG. 2 , the right display 225 of the XR system 200 is shown as a display that displays and / or provides a view of both the virtual content (represented by the tetrahedrons) and a view of the environment (represented by the house) according to a second display configuration 255, in which the virtual content (represented by the tetrahedrons) is large.

[0087]

[0101] In some examples, the XR system 200 includes a feedback engine 260. The feedback engine 260 can detect feedback received from the user interface. The feedback can be feedback regarding the virtual content, the modification by the compositor 220 from the first display configuration 250 to the second display configuration 255, and / or a determination by the XR system 200 on which the modification by the compositor 220 from the first display configuration 250 to the second display configuration 255 is based. The determination by the XR system 200 can include, for example, a level of perception of the virtual content by the user as determined by the perception engine 235, a level of understanding and / or perception of the virtual content by the user as determined by the comprehension engine 240, perception-related attributes determined by the attribute engine 230, historical data about the user, contextual data, sensor data from the user facing sensors 205, sensor data from the environment facing sensors 210, or a combination thereof. The feedback received by the feedback engine 260 can be positive feedback or negative feedback. For example, if the virtual content requests or suggests that the user perform a particular action (e.g., turn right) and the user performs the action, the feedback engine 260 can interpret this performance of the action by the user as positive feedback. Positive feedback can also be based on perception-related attributes such as the user smiling, laughing, nodding, making a positive statement (e.g., "yes," "I see," "got it," "next"), or otherwise reacting positively to the virtual content. On the other hand, if the virtual content requests or suggests that the user perform a particular action (e.g., turn right on XYZ Street) and the user does not perform the action or performs a different action (e.g., the user turns left on XYZ Street), the feedback engine 260 can interpret the non-performance of the action by the user or the performance of the different action by the user as negative feedback.Negative feedback can also be based on perception-related attributes, such as the user grimacing, crying, shaking their head (e.g., in a "no" motion), making negative statements (e.g., "no," "wrong," "not good," "not this"), or otherwise reacting negatively to the virtual content.

[0088]

[0102] In some examples, the feedback engine 260 provides feedback to one or more ML systems of the XR system 200 to update the one or more ML systems of the XR system 200. The feedback engine 260 can provide as training data to one or more ML systems of the XR system 200 the feedback, the virtual content that triggered the feedback, the display settings that triggered the feedback, modifications to the display settings that triggered the feedback, the user's level of perception of the virtual content as determined by the perception engine 235, the user's level of understanding and / or appreciation of the virtual content as determined by the comprehension engine 240, perception-related attributes determined by the attribute engine 230 corresponding to the feedback, historical data about the user corresponding to the feedback, contextual data corresponding to the feedback, sensor data from the user facing sensors 205 corresponding to the feedback, sensor data from the environment facing sensors 210 corresponding to the feedback, or combinations thereof. For example, the feedback engine 260 may provide such training data to one or more ML systems of the attribute engine 230 (e.g., the first trained ML model 825), to one or more ML systems of the perception engine 235 (e.g., the second trained ML model 835), to one or more ML systems of the understanding engine 240 (e.g., the third trained ML model 865), or to any combination thereof.

[0089]

[0103] In some examples, feedback engine 260 includes a software element, such as a set of instructions corresponding to a program executing on a processor, such as processor 1110, image processor 150, host processor 152, ISP 154, or a combination thereof, of computing system 1100. In some examples, feedback engine 260 includes one or more hardware elements. For example, feedback engine 260 may include a processor, such as processor 1110, image processor 150, host processor 152, ISP 154, or a combination thereof, of computing system 1100. In some examples, feedback engine 260 includes a combination of one or more software elements and one or more hardware elements.

[0090]

[0104] Over time, the system can learn and optimize the amount of time a message should remain in place for a user. The system determines the likelihood that a message will be read / seen at a particular time, for example, by developing a confidence level for the message. If a high likelihood is determined, there may be no need to use an eye tracking camera.

[0091]

[0105] FIG. 3A is a perspective view 300 showing a head mounted display (HMD) 310 used as an extended reality (XR) system 200. The HMD 310 may be, for example, an augmented reality (AR) headset, a virtual reality (VR) headset, a mixed reality (MR) headset, an extended reality (XR) headset, or any combination thereof. The HMD 310 may be an example of the XR system 200. The HMD 310 includes a first camera 330A and a second camera 330B along the front of the HMD 310. The first camera 330A and the second camera 330B may be examples of the environment facing sensor 210 of the XR system 200. The HMD 310 includes a third camera 330C and a fourth camera 330D that face the user's eyes when the user's eyes face the display 340. The third camera 330C and the fourth camera 330D may be examples of the user-facing sensor 205 of the XR system 200. In some examples, the HMD 310 may have only a single camera with a single image sensor. In some examples, the MHD 310 may include one or more additional cameras in addition to the first camera 330A, the second camera 330B, the third camera 330C, and the fourth camera 330D. In some examples, the MHD 310 may include one or more additional sensors in addition to the first camera 330A, the second camera 330B, the third camera 330C, and the fourth camera 330D, and the one or more additional sensors may also include other types of user-facing sensor 205 and / or environment-facing sensor 210 of the XR system 200. In some examples, the first camera 330A, the second camera 330B, the third camera 330C, and / or the fourth camera 330D may be examples of the image capture and processing system 100, the image capture device 105A, the image processing device 105B, or a combination thereof.

[0092]

[0106] The HMD 310 may include one or more displays 340 visible to the user 320 wearing the HMD 310 on the user's head. The one or more displays 340 of the HMD 310 may be examples of the one or more displays 225 of the XR system 200. In some examples, the HMD 310 may include one display 340 and two viewfinders. The two viewfinders may include a left viewfinder for the left eye of the user 320 and a right viewfinder for the right eye of the user 320. The left viewfinder may be oriented so that the left eye of the user 320 sees the left side of the display. The right viewfinder may be oriented so that the left eye of the user 320 sees the right side of the display. In some examples, the HMD 310 may include two displays 340, including a left display that displays content to the left eye of the user 320 and a right display that displays content to the right eye of the user 320. The display(s) 340 of the HMD 310 may be a digital "pass-through" display or an optical "see-through" display.

[0093]

[0107] The HMD 310 may include one or more earpieces 335 that may function as speakers and / or headphones to output audio to one or more ears of a user of the HMD 310. Although one earpiece 335 is shown in FIGS. 3A and 3B, it should be understood that the HMD 310 may include two earpieces, one for each ear (left and right) of the user. In some examples, the HMD 310 may also include one or more microphones (not shown). The one or more microphones may be examples of the user-facing sensors 205 and / or the environment-facing sensors 210 of the XR system 200. In some examples, the audio output by the HMD 310 to the user through the one or more earpieces 335 may include or be based on audio recorded using the one or more microphones.

[0094]

[0108] FIG. 3B is a perspective view 350 showing the head mounted display (HMD) of FIG. 3A being worn by a user 320. The user 320 wears the HMD 310 on the user's 320 head over the user's 320 eyes. The HMD 310 can capture images using a first camera 330A and a second camera 330B. In some examples, the HMD 310 displays one or more output images to the user's 320 eyes using a display 340. In some examples, the output image can include virtual content generated by the virtual content generator 215, composited using the compositor 220, and / or displayed by the display 225 according to a display setting (e.g., first display setting 250, second display setting 255). The output image can be based on images captured by the first camera 330A and the second camera 330B, for example, with the virtual content overlaid. The output image may provide a stereoscopic view of the environment, possibly with virtual content overlaid and / or other modifications. For example, the HMD 310 may display a first display image based on an image captured by the first camera 330A to the right eye of the user 320. The HMD 310 may display a second display image based on an image captured by the second camera 330B to the left eye of the user 320. For example, the HMD 310 may provide overlaid virtual content in the display image overlaid on the images captured by the first camera 330A and the second camera 330B. The third camera 330C and the fourth camera 330D may capture images of the eye before, during, and / or after the user views the display image displayed by the display 340. In this manner, sensor data from the third camera 330C and / or the fourth camera 330D may capture the reaction of the user's eye (and / or other parts of the user) to the virtual content. Earpieces 335 of the HMD 310 are shown in the ears of a user 320 .The HMD 310 may output audio to the user 320 through earpiece 335 and / or through another earpiece (not shown) of the HMD 310 in the other ear (not shown) of the user 320.

[0095]

[0109] 4A is a perspective view 400 showing the front of a mobile handset 410 that includes a forward-facing camera and can be used as an extended reality (XR) system 200. The mobile handset 410 may be an example of an XR system 200. The mobile handset 410 may be, for example, a mobile phone, a satellite phone, a portable game console, a music player, a health tracking device, a wearable device, a wireless communication device, a laptop, a mobile device, any other type of computing device or computing system described herein, or a combination thereof.

[0096]

[0110] The front surface 420 of the mobile handset 410 includes a display 440. The front surface 420 of the mobile handset 410 includes a first camera 430A and a second camera 430B. The first camera 430A and the second camera 430B may be examples of the user-facing sensors 205 of the XR system 200. The first camera 430A and the second camera 430B may face the user, including the user's eyes, while content (e.g., virtual content overlaid on top of an environment) is displayed on the display 440. The display 440 may be an example of the display 225 of the XR system 200.

[0097]

[0111] The first camera 430A and the second camera 430B are shown within a bezel around the display 440 on the front face 420 of the mobile handset 410. In some examples, the first camera 430A and the second camera 430B can be located in a notch or cutout cut out of the display 440 on the front face 420 of the mobile handset 410. In some examples, the first camera 430A and the second camera 430B can be under-display cameras located between the display 440 and the remainder of the mobile handset 410, so that light passes through a portion of the display 440 before reaching the first camera 430A and the second camera 430B. The first camera 430A and the second camera 430B in the perspective view 400 are forward-facing cameras. The first camera 430A and the second camera 430B face in a direction perpendicular to the plane of the front face 420 of the mobile handset 410. The first camera 430A and the second camera 430B may be two of one or more cameras of the mobile handset 410. The first camera 430A and the second camera 430B may be the sensors 405A and 405B, respectively. In some examples, the front face 420 of the mobile handset 410 may have only a single camera.

[0098]

[0112] In some examples, the front surface 420 of the mobile handset 410 may include one or more additional cameras in addition to the first camera 430A and the second camera 430B. The one or more additional cameras may also be examples of the user-facing sensors 205 of the XR system 200. In some examples, the front surface 420 of the mobile handset 410 may include one or more additional sensors in addition to the first camera 430A and the second camera 430B. The one or more additional sensors may also be examples of the user-facing sensors 205 of the XR system 200. In some cases, the front surface 420 of the mobile handset 410 includes two or more displays 440. The one or more displays 440 of the front surface 420 of the mobile handset 410 may be examples of the displays 225 of the XR system 200. For example, the one or more displays 440 may include one or more touch screen displays.

[0099]

[0113] The mobile handset 410 may include one or more speakers 435A and / or other audio output devices (e.g., earphones or headphones or connectors thereto) that can output audio to one or more ears of a user of the mobile handset 410. While one speaker 435A is shown in FIG. 4A, it should be understood that the mobile handset 410 can include more than one speaker and / or other audio devices. In some examples, the mobile handset 410 can also include one or more microphones (not shown). The one or more microphones may be examples of user-facing sensors 205 and / or environmental facing sensors 210 of the XR system 200. In some examples, the mobile handset 410 can include one or more microphones along and / or adjacent to the front surface 420 of the mobile handset 410, which are examples of user-facing sensors 205 of the XR system 200. In some examples, audio output by the mobile handset 410 to the user through one or more speakers 435A and / or other audio output devices may include or be based on audio recorded using one or more microphones.

[0100]

[0114] 4B is a perspective view 450 showing a rear view 460 of a mobile handset that includes a rear-facing camera and can be used as an extended reality (XR) system 200. The mobile handset 410 includes a third camera 430C and a fourth camera 430D on the rear view 460 of the mobile handset 410. The third camera 430C and the fourth camera 430D in the perspective view 450 are rear-facing. The third camera 430C and the fourth camera 430D may be examples of the environmental facing sensor 210 of the XR device 200 of FIG. 2. The third camera 430C and the fourth camera 430D face in a direction perpendicular to the plane of the rear view 460 of the mobile handset 410.

[0101]

[0115] The third camera 430C and the fourth camera 430D may be two of the one or more cameras of the mobile handset 410. In some examples, the back surface 460 of the mobile handset 410 may have only a single camera. In some examples, the back surface 460 of the mobile handset 410 may include one or more additional cameras in addition to the third camera 430C and the fourth camera 430D. The one or more additional cameras may also be examples of the environmental facing sensors 210 of the XR system 200. In some examples, the back surface 460 of the mobile handset 410 may include one or more additional sensors in addition to the third camera 430C and the fourth camera 430D. The one or more additional sensors may also be examples of the environmental facing sensors 210 of the XR system 200. In some examples, the first camera 430A, the second camera 430B, the third camera 430C, and / or the fourth camera 430D may be examples of the image capture and processing system 100, the image capture device 105A, the image processing device 105B, or a combination thereof.

[0102]

[0116] The mobile handset 410 may include one or more speakers 435B and / or other audio output devices (e.g., earphones or headphones or connectors thereto) that can output audio to one or more ears of a user of the mobile handset 410. While one speaker 435B is shown in FIG. 4B, it should be understood that the mobile handset 410 can include more than one speaker and / or other audio device. In some examples, the mobile handset 410 can also include one or more microphones (not shown). The one or more microphones may be examples of the user-facing sensors 205 and / or the environmental facing sensors 210 of the XR system 200. In some examples, the mobile handset 410 may include one or more microphones along and / or adjacent to the back surface 460 of the mobile handset 410 that are examples of the environmental facing sensors 210 of the XR system 200. In some examples, audio output by the mobile handset 410 to the user through one or more speakers 435B and / or other audio output devices may include or be based on audio recorded using one or more microphones.

[0103]

[0117] The mobile handset 410 may use the display 440 on the front face 420 as a pass-through display. For example, the display 440 may display an output image. The output image may be based on an image captured by the third camera 430C and / or the fourth camera 430D, e.g., with virtual content overlaid. The first camera 430A and / or the second camera 430B may capture images of the user's eye (and / or other parts of the user) before, during, and / or after the output image including the virtual content is displayed on the display 440. In this manner, sensor data from the first camera 430A and / or the second camera 430B may capture a reaction of the user's eye (and / or other parts of the user) to the virtual content.

[0104]

[0118] FIG 5A is a conceptual diagram 500 illustrating an image 505 of a Boston environment 510 before and after an overlay of virtual content displayed in accordance with display settings 520. On the left side of FIG 5A, the image 505 of the Boston environment 510 is shown without an overlay of virtual content (before overlay). On the right side of FIG 5A, the image 505 of the Boston environment 510 is shown with an overlay of virtual content (after overlay), including virtual content 525, virtual content 530, and virtual content 535. The virtual content is overlaid on top of the image 505 of the environment 510 in accordance with display settings 520.

[0105]

[0119] The image 505 of the environment 510 may be an example of an image captured by the environment facing sensor 210 of the XR system 200. For example, the image 505 of the environment 510 may be an example of an image captured by the first camera 330A and / or the second camera 330B of the HMD 310. Similarly, the image 505 of the environment 510 may be an example of an image captured by the third camera 430C and / or the fourth camera 430D of the mobile handset 410. The image 505 of the Boston environment 510 includes a view of a statue of Boston Celtics coach Red Auerbach. The image 505 of the Boston environment 510 includes a view of a memorial shoe of Boston Celtics player Larry Bird. The image 505 of the Boston environment 510 includes a view of an entrance to a sports apparel store.

[0106]

[0120] Virtual content 525, virtual content 530, and virtual content 535 are examples of virtual content generated by virtual content generator 215 of XR system 200. Display configuration 520 is an example of a first display configuration 250 generated by compositor 220 of XR system 200. Virtual content 525 includes text that reads "LANDAM: Boston Celtics player Larry Bird's shoe memorial." The display configuration causes virtual content 525 to be overlaid on image 505 of environment 510 near the bottom of image 505 where Larry Bird's shoe memorial is depicted in image 505. Virtual content 530 includes text that reads "LANDAM: Boston Celtics coach Red Auerbach's statue." The display configuration causes virtual content 530 to be overlaid on image 505 of environment 510 near the center of image 505 where Red Auerbach's statue is depicted in image 505. The virtual content 535 includes text that reads "Store: Sports Apparel Store." Depending on the display settings, the virtual content 530 is overlaid on top of the image 505 of the environment 510 near the right side of the image 505 where the entrance to the sports apparel store is depicted.

[0107]

[0121] FIG. 5B is a conceptual diagram 550 showing an image 505 of the Boston environment 510 with virtual content overlay before and after a modification to the display setting 520. On the left side of FIG. 5B, the image 505 of the Boston environment 510 is shown with an overlay of virtual content according to the display setting 520, similar to the right side of FIG. 5A. On the right side of FIG. 5B, the image 505 of the Boston environment 510 is shown with an overlay of virtual content according to the display setting 555. The display setting 555 is different from the display setting 520. The display setting 555 may be an example of a second display setting 255 generated by the compositor 220 of the XR system 200. The modification to the display setting 520, i.e., from the display setting 520 to the display setting 555, may be an example of a modification from the first display setting 250 to the second display setting 255 by the compositor 220 of the XR system 200.

[0108]

[0122] In accordance with the second display configuration 555, the virtual content 535 (describing the sports apparel store) is now hidden, erased, and / or terminated from view. In some examples, this modification to erase the virtual content 535 in the second display configuration 555 may be in response to an indication from the XR system 200 (e.g., from the perception engine 235 and / or the comprehension engine 240) that the user has already perceived and / or understood the virtual content 535 to a high level. In some examples, this modification to erase the virtual content 535 in the second display configuration 555 may be in response to negative feedback to the virtual content 535 received via the feedback engine 260, such as a request to filter out virtual content related to the store, an intentional avoidance of the virtual content 535, detection of a negative verbal expression (e.g., "no") from the user while the user is viewing the virtual content 535, or a combination thereof. In some examples, this modification to remove the virtual content 535 in the second display setting 555 may be in response to positive feedback on the virtual content 525 and / or the virtual content 530 received via the feedback engine 260, such as a request received from a user (e.g., via a user interface of the feedback engine 260) to focus on virtual content associated with a landmark or a request to highly focus on the virtual content 525 and / or the virtual content 530.

[0109]

[0123] According to the second display configuration 555, the virtual content 525 (describing the Larry Bird shoe memorial) is now larger and the text is in a larger font than according to the first display configuration 520. According to the second display configuration 555, the virtual content 525 is highlighted and / or displayed more prominently and / or with a higher priority than according to the first display configuration 520. The text of the virtual content 525 is now also longer, so that it now reads "Landmark: A shoe memorial to Larry Bird, who was a player for the Boston Celtics from 1979-1992." In some examples, this modification of highlighting the virtual content 525 in the second display configuration 555 may be in response to an indication from the XR system 200 (e.g., from the perception engine 235 and / or the comprehension engine 240) that the user has perceived the virtual content 525 to a sufficient level of user perception (e.g., above a threshold) but has not yet understood the virtual content 525 to a sufficient level of comprehension (e.g., above a threshold). In some examples, this modification to highlight the virtual content 525 in the second display configuration 555 may be in response to an indication from the XR system 200 (e.g., from the perception engine 235 and / or the grasp engine 240 and / or the perception-related attributes engine 230) that the user has fixated the virtual content 525 and / or experienced a saccade to the virtual content 525 and / or experienced pupil dilation while viewing the virtual content 525. In some examples, this modification to highlight the virtual content 525 in the second display configuration 555 may be in response to positive feedback to the virtual content 525 received via the feedback engine 260, such as a strong focus of the user's gaze on the virtual content and / or a related depiction of a memorial to Larry Bird's shoes, detection of a positive verbal expression from the user while the user is viewing the virtual content 525 (e.g., "tell me more about this"), or a combination thereof. In some examples, this modification to highlight virtual content 525 in second display setting 555 may be in response to negative feedback on virtual content 535 and / or virtual content 530.

[0110]

[0124] According to the second display configuration 555, the virtual content 530 (describing the Red Auerbach statue) is shifted slightly to the right and shifted back (in depth) to appear behind a portion of the head of the Red Auerbach statue. In some examples, this modification of moving the virtual content 530 in the second display configuration 555 may be in response to an indication from the XR system 200 (e.g., from the perception engine 235 and / or the perception engine 240) that the user has not yet perceived and / or grasped the virtual content 530 to a sufficient level (e.g., above a threshold). In some examples, this modification of moving the virtual content 530 in the second display configuration 555 may be in response to positive feedback received from the user (e.g., via a user interface of the feedback engine 260) for the virtual content 530 and / or the virtual content 525 received via the feedback engine 260, such as a request to focus on the virtual content associated with a landmark.

[0111]

[0125] FIG. 6 is a conceptual diagram 600 showing an image 605 of an environment 610 including a book before and after an overlay of virtual content 630 displayed according to display settings 620. On the left side of FIG. 6, the image 605 of the environment 610 including a book is shown without an overlay of virtual content 630 (before overlay). For clarity, the only word shown in the book's text in the image 605 is "superconducting." On the left side of FIG. 6, the image 605 of the environment 610 including a book is shown with an overlay of virtual content 630 (after overlay). The virtual content 630 is overlaid on top of the image 605 of the environment 610 according to display settings 620.

[0112]

[0126] The image 605 of the environment 610 may be an example of an image captured by the environmental facing sensor 210 of the XR system 200. For example, the image 605 of the environment 610 may be an example of an image captured by the first camera 330A and / or the second camera 330B of the HMD 310. Similarly, the image 605 of the environment 610 may be an example of an image captured by the third camera 430C and / or the fourth camera 430D of the mobile handset 410.

[0113]

[0127] Virtual content 630 is an example of virtual content generated by virtual content generator 215 of XR system 200. Display configuration 620 is an example of a display configuration (e.g., first display configuration 250, second display configuration 255) generated by compositor 220 of XR system 200. Virtual content 630 includes a highlighting of the word "superconductivity" in a book with additional text overlaid above the book stating, "Definition: Superconductivity is a set of physical properties observed in certain materials in which electrical resistance vanishes and magnetic flux fields emanate from the material. Any material that exhibits these properties is a superconductor."

[0114]

[0128] In some examples, the display of virtual content 630 overlaid on the image 605 of the environment 610 including the book according to the display settings 620 may be in response to an indication from the XR system 200 (e.g., from the perception engine 235 and / or the grasp engine 240 and / or the attribute engine 230) that the user is squinting and / or tilting his / her head while fixating on the word "superconducting" in the book. In some examples, the XR system 200 provides definitions for other words as well. In some examples, the XR system 200 provides translations of words from one language to another as well.

[0115]

[0129] Figure 7 is a conceptual diagram 700 showing an image 705 of a street environment 710 before and after the overlay of virtual content 730 displayed according to display settings 720. On the left side of Figure 7, the image 705 of the street environment 710 is shown without the overlay of virtual content 730 (before overlay). On the left side of Figure 7, the image 705 of the street environment 710 is shown with the overlay of virtual content 730 (after overlay). The virtual content 730 is overlaid on top of the image 705 of the environment 710 according to display settings 720.

[0116]

[0130] The image 705 of the environment 710 may be an example of an image captured by the environmental facing sensor 210 of the XR system 200. For example, the image 705 of the environment 710 may be an example of an image captured by the first camera 330A and / or the second camera 330B of the HMD 310. Similarly, the image 705 of the environment 710 may be an example of an image captured by the third camera 430C and / or the fourth camera 430D of the mobile handset 410.

[0117]

[0131] Virtual content 730 is an example of virtual content generated by the virtual content generator 215 of the XR system 200. Display setting 720 is an example of a display setting (e.g., first display setting 250, second display setting 255) generated by the compositor 220 of the XR system 200. An image 705 of a street environment 710 depicts a car driving towards the camera and thus towards the user of the XR system 200. The virtual content 730 includes a warning with a warning icon and text that reads "WARNING: Car entering, move away!" The font size of the warning and the text is large.

[0118]

[0132] In some examples, the display of the virtual content 730 overlaid on the image 705 of the street environment 710 according to the display configuration 720 may be in response to the XR system 200 detecting a car from the sensor data of the environment facing sensor 210. The XR system 200 may include an object detection engine. The object detection engine may include a feature detection algorithm, a feature extraction algorithm, a feature recognition algorithm, a feature tracking algorithm, an object detection algorithm, an object recognition algorithm, an object tracking algorithm, a face detection algorithm, a face recognition algorithm, a face tracking algorithm, a person detection algorithm, a person recognition algorithm, a person tracking algorithm, a vehicle detection algorithm, a vehicle recognition algorithm, a vehicle tracking algorithm, a classifier, or a combination thereof. The object detection engine may include one or more AI algorithms and / or ML systems. The object detection engine may include, for example, a neural network 900. In some examples, the display of virtual content 730 overlaid on the image 705 of the street environment 710 in accordance with the display settings 720 may be in response to an indication from the XR system 200 (e.g., from the perception engine 235 and / or the grasping engine 240 and / or the attribution engine 230) that the user did not perceive the car detected by the XR system 200 to a sufficient level (e.g., above a threshold level).

[0119]

[0133] 8 is a block diagram 800 illustrating a process for determining a perception level and a comprehension level for virtual content based on one or more trained machine learning (ML) models. The process of FIG. 8 is performed by an imaging system, such as the XR system 200 of FIG.

[0120]

[0134] The process begins with the capture of sensor data 805 by one or more sensors 810 facing one or both eyes 815 of a user 820. The sensor 810 may be an example of a user-facing sensor 205 of the XR system 200. The imaging system provides the sensor data 805 as an input to a first trained ML model 825. The first trained ML model 825 may be part of the imaging system. The first trained ML model 825 may be part of the attribute engine 230 of the XR system 200 in some examples. In response to receiving the sensor data 805 as an input, the first trained ML model 825 outputs perception-related attribute data 830 corresponding to the eye 815 of the user 820. The perception-related attribute data 830 may identify, for example, the user's eye movement, pupil dilation, blinking, squinting, saccades, fixations, eye moisture levels, etc.

[0121]

[0135] The imaging system provides the perception-related attribute data 830 as an input to the second trained ML model 835. The imaging system also provides the second trained ML model 835 with a display setting 837 for displaying the virtual content 840 via a display 845 as an input. The display 845 is an example of the display 225 of the XR system 200. The display setting 837 is an example of the first display setting 250 and / or the second display setting 255 of the XR system 200. The imaging system can also provide the context information 890 as an input to the second trained ML model 835. The context information can include, for example, the location of the imaging system, ambient sounds, ambient lighting, an activity detected as being performed by the user, etc.

[0122]

[0136] The second trained ML model 835 may be part of the imaging system. The second trained ML model 835 may be part of the perception engine 235 of the XR system 200 in some examples. In response to receiving the perception-related attribute data 830 and / or the display settings 837 and / or the context information 890 as input, the second trained ML model 835 outputs a perception level 850 of the virtual content 840 by the user 820 using the eye 815. The virtual content 840 may be an example of a virtual content generated by the virtual content generator 215 of the XR system 200 and displayed by the display 225 of the XR system 200 according to the display settings (e.g., the first display setting 250, the second display setting 255) determined by the compositor 220 of the XR system 200. The level of perception 850 of the virtual content 840 by the user 820 can be based, for example, on the extent to which the gaze of the user's 820's eye 815 is at or near the location on the display 845 where the virtual content 840 is displayed according to the display settings 837 (e.g., based on time and / or distance).

[0123]

[0137] The imaging system provides as an input to the third trained ML model 865 a perception level 850 of the virtual content 840 by the user 820. In some examples, the imaging system may also provide as an input to the third trained ML model 865 historical data 855 associated with the user 820. In some examples, the imaging system may also provide as an input to the third trained ML model 865 contextual information 890. The historical data 855 associated with the user 820 may identify, for example, the education level of the user 820, the occupation of the user 820, information regarding the historical behavior of the user 820, and / or any other historical information associated with the user. In some examples, the imaging system may also provide as an input to the third trained ML model 865 one or more characteristics of the virtual content. For example, the one or more characteristics of the virtual content may include a rating of the virtual content 840 generated by a virtual content rating engine (e.g., the virtual content rating engine 245 of the XR system 200, etc.). The rating may include, for example, one or more metrics related to the complexity and / or uniqueness of the virtual content 840. In some examples, the virtual content rating engine may generate a complexity metric and a uniqueness metric and convert these two metrics into a single combined metric that reflects both the complexity and the uniqueness. In some examples, the virtual content rating engine averages a value of the uniqueness metric and a value of the complexity metric to generate the combined metric. In some examples, the virtual content rating engine multiplies a value of the uniqueness metric and a value of the complexity metric to generate the combined metric. In some examples, the virtual content rating engine adds a value of the uniqueness metric and a value of the complexity metric to generate the combined metric.

[0124]

[0138] The third trained ML model 865 may be part of the imaging system. The third trained ML model 865 may be part of the comprehension engine 240 of the XR system 200 in some examples. In response to receiving the perception level 850, the historical data 855, the rating 860, and / or the context information 890 as inputs, the third trained ML model 865 outputs a comprehension level 870 of the virtual content 840 by the user 820. The comprehension level 870 may be referred to as a degree of comprehension level 870 and / or a comprehension level metric 870. The comprehension level 870 may be based, for example, on the degree to which the perception level 850 meets or exceeds an appropriate level that considers the complexity and / or uniqueness of the virtual content 840 (e.g., based on the rating 860), based on the user 820's own background (e.g., based on the historical data 855), based on the context information 890, or a combination thereof.

[0125]

[0139] The imaging system uses one or more of the grasp level 870 for the virtual content 840, the perception level 850 for the virtual content 840, the perception-related attribute data 830, and / or the context information 890 as a basis for generating a modification 875 to the display setting 837. The modification 875 to the display setting 837 may be an example of a modification from the first display setting 250 to the second display setting 255 by the compositor 220 of the XR system 200. The imaging system may receive feedback 880, for example, via a user interface of a feedback engine (e.g., feedback engine 260). The imaging system may perform additional training 885 of the first trained ML model 825, the second trained ML model 835, and / or the third trained ML model 865 based on the feedback 880. The imaging system can use feedback 880, which is positive, during training 885 to strengthen weights in the first trained ML model 825, the second trained ML model 835, and / or the third trained ML model 865. The imaging system can use feedback 880, which is negative, during training 885 to modify, remove, or add weights in the first trained ML model 825, the second trained ML model 835, and / or the third trained ML model 865. Although the first trained ML model 825, the second trained ML model 835, and the third trained ML model 865 are shown as separate ML models in FIG. 8, it should be understood that any two (or all three) of these ML models can be implemented in a single ML model. Further, in some examples, any of the trained ML models shown in FIG. 8 (e.g., the first trained ML model 825, the second trained ML model 835, or the third trained ML model 865) may be split into two or more auxiliary trained ML models.For example, one auxiliary trained ML model can receive the inputs shown in Figure 8 associated with the trained ML model in question and can generate the intermediate data. Another auxiliary trained ML model can receive the intermediate data as the last one of its inputs and can generate the output shown in Figure 8 associated with the trained ML model in question.

[0126]

[0140] The perception level 850 and / or comprehension level 870 of the user 820 may include factors related to the cognitive abilities and state of the user. Examples of cognitive-based factors may include, among others, previous experience and / or skill level for a particular task, such as a particular XR task (e.g., reading a notification), a particular real-world task (e.g., cooking in a kitchen), an augmented particular real-world task (e.g., walking in a city and receiving navigation instructions), information about the user's education level, information about the user's knowledge level in a particular field, information about the user's occupation, information about the user's cognitive impairment, or a combination thereof. Examples of cognitive-based factors may alternatively or additionally include attention applied to the augmented content, such as saccades (a type of eye movement used to move the fovea from one point to another) and fixation time, gaze duration, repeated glances, interactions with the content (e.g., scrolling, responding, closing, etc.), among others. Examples of cognitive-based factors may alternatively or additionally include, among others, the user's mental state, such as wakefulness, fatigue, attention divided into two or more activities.

[0127]

[0141] The context of use of the XR system may include aspects of the real-world environment during use of the XR system. Examples of such aspects may include, among others, the noise level in the environment (e.g., ambient sounds, additional people talking in the environment, etc.), the lighting in the environment (e.g., ambient lighting), the activity of the user, the location where the XR system is being used, past history of interacting with content, the time of day, whether the environment is static or dynamic (e.g., if the user is on a moving object such as a vehicle, train, elevator, escalator, etc.), and obstacles in the environment.

[0128]

[0142] Information associated with the content being output by the XR system may include characteristics of the virtual content and / or characteristics of the physical content (e.g., augmented objects). Examples of such characteristics may include the semantic complexity of the message or other virtual content, the severity and importance of the message or other virtual content, the length of the message or other virtual content, the relevance of the message or other virtual content to the task being performed (e.g., reading a book), the uniqueness of the virtual content to the environment, whether the message or other virtual content is contextually expected (e.g., whether receiving the message content may be appropriate or relevant to a situation, time, place, activity, etc.).

[0129]

[0143] In some aspects, the XR system can monitor other external factors, such as user input, display duration of previously presented virtual content, device state (e.g., a power state such as low power), AR device screen resolution, speed of movement of the XR system (and thus the user), and / or other external factors.

[0130]

[0144] The present systems and techniques can be used to determine an initial presentation of content in an XR system and / or to assess a user's perception of virtual content and determine potential outcomes. For example, to determine an initial presentation of content, the XR system can monitor the user's cognitive state, the context of use (e.g., aspects of the real-world environment), and the task being performed. In some cases, the XR system can monitor the content, including physical objects on which the augmented content may be displayed. Based on the cognitive state, the context of use, and the task (and possibly the content), the XR system can determine that an augmentation would be beneficial. The XR system can then provide (e.g., output) an augmentation that can be tailored to the cognitive state, the context, and the task (and possibly the content). In one illustrative example, the task can include a user reading a library book, the cognition can include that the user is sleepy and may not be absorbing the material well (e.g., based on an XR system monitoring saccades and other eye characteristics), the context can include that the room in which the user is located is bright based on being in a library and near a window during the day (e.g., based on input from an ambient light sensor and a Global Navigation Satellite System (GNSS) receiver), and the content can include that the user is reading a challenging book on physics and that the user has never taken a physics course before. While reading the book, the AR device can determine that the user is staring at a word and squinting their eyes. The AR device can then determine that a world definition or translation would be useful to display as a pop-up for the word.

[0131]

[0145] As described above, the present systems and techniques can be used to assess user perception of virtual content and determine potential outcomes. For example, the XR system can present virtual content (e.g., a warning notification such as "low battery", AR content such as an arrow next to a switch with a label "do not press this button", AR content with information associated with a point or place of interest, etc.). If the user completes a task associated with the virtual content or manually closes the virtual content, the XR system can erase the virtual content. If not, the XR system can perform an analysis to determine whether the user perceived the virtual content.

[0132]

[0146] To assess the user's perception of the virtual content, the present systems and techniques can determine the level of perception of the virtual content or the degree of the user's perception of the virtual content. In some examples, to determine whether the user has perceived the virtual content and / or the level or degree of perception of the virtual content, the XR system can evaluate the content, perform an eye analysis of the user (e.g., saccade analysis), and / or perform a secondary analysis of the user (e.g., by checking pupils, squinting, and head movement). In some cases, the XR system can use machine learning (ML)-based pattern recognition (e.g., using one or more neural networks) to determine whether the user has perceived the virtual content. If it is determined that the user has perceived the virtual content (e.g., looked in the direction of the virtual content for more than a threshold time), the XR system can close or erase the virtual content (e.g., erase the content from the display). If the XR system determines that the user is not perceiving the virtual content, the XR system can determine whether to keep the content in place, whether to increase the salience of the content (e.g., based on the context), and / or whether to perform other actions (e.g., based on the context of ambient light, noise, user activity, etc.). For example, if the context indicates that the ambient light is bright, the XR system can increase the display brightness. In another example, if the context indicates that the surrounding environment is noisy, the XR system can increase the volume of the AR content (e.g., when audible virtual content is output).

[0133]

[0147] In some embodiments, to perform eye analysis, the XR system can turn on an eye tracking camera and use the eye tracking camera to track the eyes on the virtual content. For example, the XR system can distinguish between attention paid to the virtual content and attention paid to something in the field of view behind the virtual content. If a low value (e.g., a value below a threshold, such as threshold 10) is determined for the virtual content, the XR system can erase the virtual content. If a high value (e.g., a value above a threshold) is determined for the virtual content, the XR system can keep the content in place for a period of time (e.g., 5 seconds, 10 seconds, etc.) to give the user time to reread the content. If the XR system determines that the user is rereading the virtual content, the XR system can leave the content in place.

[0134]

[0148] In some aspects, to perform secondary analysis, the XR system can check the user's pupils, whether the user is squinting, and / or the user's head movement. For example, the XR system can maintain the content in place if it determines that one or more of the user's pupils are not dilated, the user is squinting, the user's head is tilted to the side or forward, the user's blink rate, the user is staring at the virtual content for a particular duration (e.g., 3 seconds or more), the user is looking at the content at a particular frequency (e.g., twice per second).

[0135]

[0149] In some cases, over time, the XR system can learn and optimize (e.g., using ML-based pattern recognition) how long a message should remain in place for a user. The XR system may determine the likelihood that the virtual content will be read and / or viewed within a threshold time (e.g., within a threshold time period of 1 second, 2 seconds, 3 seconds, 4 seconds, 5 seconds, 10 seconds, 15 seconds, etc.). In some examples, the XR system may deploy a confidence level of the displayed virtual content (e.g., virtual message or notification) to determine this likelihood. In some cases, if a high likelihood is determined for a given item of virtual content, the XR system may not utilize the eye tracking camera for the eye analysis described above.

[0136]

[0150] 9 is a block diagram illustrating an example of a neural network (NN) 900 that may be used by a trained machine learning system to analyze users viewing extended reality content. The neural network 900 may include any type of deep network, such as a convolutional neural network (CNN), an autoencoder, a deep belief net (DBN), a recurrent neural network (RNN), a generative adversarial network (GAN), and / or other types of neural networks. The neural network 900 may be an example of one of the one or more trained neural networks: the first trained ML model 825, the second trained ML model 835, the third trained ML model 865, or a combination thereof.

[0137]

[0151] An input layer 910 of the neural network 900 includes input data. The input data of the input layer 910 can include data representing pixels of one or more input image frames. In some examples, the input data of the input layer 910 includes data representing pixels of image data (e.g., of images captured by the user-facing sensor 205, the third camera 330C, the fourth camera 330D, the first camera 430A, the second camera 430B, and / or the sensor 810) and / or metadata corresponding to the image data. In some examples, the input data of the input layer 910 includes images captured by the user-facing sensor 205, the third camera 330C, the fourth camera 330D, the first camera 430A, the second camera 430B, and / or the sensor 810.

[0138]

[0152] In some examples, the input data of the input layer 910 can include perception-related attribute data, such as perception-related attribute data 830 and / or perception-related attribute data generated by attribute engine 230. In some examples, the input data of the input layer 910 can include display settings for displaying the virtual content, such as first display setting 250, second display setting 255, display setting 6520, display setting 555, display setting 620, display setting 720, display setting 837, or a combination thereof.

[0139]

[0153] In some examples, the input data of the input layer 910 may include a perception level of the virtual content by the user through the user's eyes, such as perception level 850 and / or a perception level determined using the perception engine 235. In some examples, the input data of the input layer 910 may include historical data associated with the user, such as historical data 855. In some examples, the input data of the input layer 910 may include one or more ratings associated with the virtual content, such as ratings 860, ratings and / or metrics generated by the virtual content rating engine 245, or combinations thereof. In some examples, the input data of the input layer 910 may include contextual data and / or contextual information, such as context information 890. In some aspects, the ratings 860 may be referred to as metrics and / or scores.

[0140]

[0154] The image may include image data from an image sensor including raw pixel data (e.g., including a single color per pixel based on a Bayer filter) or processed pixel values ​​(e.g., RGB pixels for an RGB image). The neural network 900 includes multiple hidden layers 912A, 912B through 912N. The hidden layers 912A, 912B through 912N include “N” hidden layers, where “N” is an integer greater than or equal to 1. The number of hidden layers may be adapted to include as many layers as required for a given application. The neural network 900 further includes an output layer 914 that provides an output resulting from the processing performed by the hidden layers 912A, 912B through 912N. In some examples, the output layer 914 may provide an output image. In some examples, the output layer 914 may provide perception-related attribute data, such as perception-related attribute data 830 and / or perception-related attribute data generated by the attribute engine 230. In some examples, output layer 914 can provide a perception level for the virtual content, such as perception level 850 and / or a perception level determined using perception engine 235. In some examples, output layer 914 can provide a user's grasp and / or understanding level of the virtual content, such as grasp level 870 and / or a grasp level determined using grasp engine 240.

[0141]

[0155] Neural network 900 is a multi-layered neural network of interconnected filters. Each filter can be trained to learn features that represent the input data. Information related to the filters is shared between different layers, with each layer retaining the information as it is processed. In some cases, neural network 900 can include a feed-forward network, in which there are no feedback connections where the output of the network is fed back to itself. In some cases, network 900 can include a recurrent neural network, which can have loops that allow information to be carried across nodes while reading the input.

[0142]

[0156] In some cases, information may be exchanged between layers through interconnections of nodes and nodes between various layers. In some cases, the network may include a convolutional neural network, which may not connect every node in one layer to every other node in the next layer. In a network in which information is exchanged between layers, the nodes of the input layer 910 may activate a set of nodes in the first hidden layer 912A. For example, as shown, each of the input nodes of the input layer 910 may be connected to each of the nodes of the first hidden layer 912A. The nodes of the hidden layer may transform the information of each input node by applying an activation function (e.g., a filter) to this information. The information derived from the transformation may then be passed to the nodes of the next hidden layer 912B, activating those nodes, which may perform their own designated function. Exemplary functions include convolution functions, downsampling, upscaling, data transformation, and / or any other suitable function. The output of the hidden layer 912B may then activate the nodes of the next hidden layer, and so on. The output of the final hidden layer 912N may activate one or more nodes in the output layer 914, which provides the processed output image. In some cases, a node in the neural network 900 (e.g., node 916) is shown as having multiple output lines, however, the node has a single output and all lines shown as outputting from the node represent the same output value.

[0143]

[0157] In some cases, each node or interconnection between nodes can have a weight, which is a set of parameters derived from training of the neural network 900. For example, the interconnections between nodes can represent information learned about the interconnected nodes. The interconnections can have adjustable numerical weights that can be adjusted (e.g., based on a training data set), allowing the neural network 900 to be adaptive to the input and to learn as more and more data is processed.

[0144]

[0158] The neural network 900 is pre-trained to process features from the data in the input layer 910 using different hidden layers 912A, 912B through 912N to provide output through the output layer 914.

[0145]

[0159] FIG. 10 is a flow diagram illustrating a process for an extended reality (XR) display operation. The process 1000 may be implemented by an imaging system. In some examples, the imaging system may include, for example, an image capture and processing system 100, an image capture device 105A, an image processing device 105B, an image processor 150, an ISP 154, a host processor 152, an XR system 200, an HMD 310, a mobile handset 410, an imaging device for capturing an image 505 and overlaying virtual content 525-535, an imaging device for capturing an image 605 and overlaying virtual content 630, an imaging device for capturing an image 705 and overlaying virtual content 730, an imaging device of FIG. 8, a first trained ML model 825 of a second trained ML model 835, a third trained ML model 865, a neural network 900, a computing system 1100, a processor 1110, or a combination thereof.

[0146]

[0160] At operation 1005, the imaging system is configured to and can cause the virtual content to be displayed using the display according to a display setting associated with the virtual content, and the environment is viewable using the display when the virtual content is displayed by the display. In some examples, the imaging system includes a display. Examples of displays include display 225, display 340, display 440, display 845, output device 1135, other displays described herein, or combinations thereof. Examples of virtual content include virtual content generated by virtual content generator 215, virtual content 515, virtual content 525, virtual content 530, virtual content 535, virtual content 615, virtual content 630, virtual content 715, virtual content 730, virtual content 840, other virtual content described herein, or combinations thereof. Examples of display settings include first display setting 250, second display setting 255, display setting 520, display setting 555, display setting 620, display setting 720, display setting 837 (before modification 875), display setting 837 (after modification 875), other display settings described herein, or combinations thereof.

[0147]

[0161] In some examples, the imaging system is configured to and can generate the virtual content prior to displaying the virtual content using the display. For example, a virtual content generator 215 of the imaging system can generate the virtual content. In some examples, a compositor 220 of the imaging system can generate the display settings.

[0148]

[0162] In some examples, the environment is viewable using the display based at least in part on light from the environment passing through at least a portion of the display. For example, the display may be at least partially transparent, translucent, light receiving, light transmissive, and / or light permissive. In such examples, the imaging system may be referred to as having an optical see-through display. In such examples, the imaging system may cause at least a portion of the virtual content to be displayed over at least a portion of a view of the environment using the display according to a display configuration. In some aspects, the imaging system's compositor 220 may generate a display configuration to overlay at least a portion of the virtual content over the view of the environment. In some aspects, the imaging system's compositor 220 may generate a display configuration to display a displayed portion of the virtual content at a simulated depth such that at least a portion of the environment appears to be in front of at least a hidden portion of the virtual content that is not displayed per the display configuration.

[0149]

[0163] In some examples, the environment is viewable using the display based at least in part on the imaging system causing a view of the environment to be displayed by the display. For example, the imaging system can capture one or more images depicting a view of the environment using one or more environmental focus sensors 210 of the imaging system. The imaging system can cause the display to display one or more images of the environment in combination with virtual content. For example, the imaging system can display the virtual content by combining and / or compositing at least a portion of the virtual content with at least a portion of the one or more images of the environment to generate a composite image, e.g., using a compositor 220 of the imaging system, and causing the display to display the resulting composite image. In some aspects, the compositor 220 of the imaging system can generate a display configuration to overlay at least a portion of the virtual content over at least a portion of the view of the environment in the one or more images of the environment. In some aspects, the imaging system's compositor 220 can generate display settings to display the displayed portion of the virtual content at a simulated depth such that at least a portion of the environment (from one or more images of the environment) appears to be in front of at least the hidden portion of the virtual content that the environment overlays per the display setting.

[0150]

[0164] In operation 1010, the imaging system is configured to and can determine a perception level for the virtual content displayed by the user using the display based on one or more perception-related attributes of the user. The perception level may be referred to as a degree of perception and / or a perception metric. In some examples, the imaging system can determine the perception-related attributes of the user using the attribute engine 230 of the imaging system, the first trained ML model 825 of the imaging system, the neural network 900, or a combination thereof. Examples of perception-related attributes include perception-related attributes determined using the attribute engine 230, the perception-related attribute data 830, the perception-related attributes determined using the NN 900, or a combination thereof.

[0151]

[0165] In some examples, the imaging system can determine a perception level of the user to the virtual content using the perception engine 235 of the imaging system, the second trained ML model 835 of the imaging system, the neural network 900, or a combination thereof. In some examples, the perception level includes a grasping level, and the imaging system can determine the perception level and / or the grasping level using the perception engine 235 of the imaging system, the grasping engine 240 of the imaging system, the virtual content evaluation engine 245 of the imaging system, the second trained ML model 835 of the imaging system, the third trained ML model 865 of the imaging system, the neural network 900, or a combination thereof. Examples of the perception level of the user to the virtual content include a perception level determined using the perception engine 235, a perception level 850, a perception level determined using the NN 900, or a combination thereof. In some examples, the perception level includes a grasp level, and examples of the perception level and / or grasp level include a perception level determined using perception engine 235, a grasp level determined using grasp engine 240, a perception level 850, a grasp level 870, a perception level determined using NN900, a grasp level determined using NN900, or a combination thereof.

[0152]

[0166] In some examples, the one or more perception-related attributes of the user are associated with one or more eyes of the user. In some examples, the one or more perception-related attributes of the user include at least one of one or more attributes of one or more eyes of the user, one or more attributes of one or more facial expressions of the user, one or more gestures of the user, or a combination thereof. In some examples, determining the one or more perception-related attributes of the user includes tracking an eye position of the user's eye, tracking an eye position of the user's eye, tracking an eye movement of the user's eye, tracking a pupil dilation of the user's eye, tracking a saccade of the user's eye, tracking a fixation by the user's eye, tracking a blink by the user's eyelid, tracking a squint by the user's eyelid, tracking an optokinetic reflex by the user's eye, tracking a vestibulo-ocular reflex by the user's eye, tracking an accommodation reflex by the user's eye, tracking a facial expression of the user, tracking a gesture by the user, or a combination thereof. Tracking in the above listed actions may refer to tracking of timing, frequency, extent, amplitude, eye position, eye movement, eye velocity, or combinations thereof.

[0153]

[0167] In some examples, the one or more perception-related attributes of the user include one or more eye positions of one or more eyes of the user relative to the virtual content. In some examples, the one or more perception-related attributes of the user include one or more characteristics of one or more saccades by one or more eyes of the user. The one or more characteristics include at least one of frequency, duration, timing, saccade velocity, saccade amplitude, eye position, eye movement, other characteristics discussed herein, or combinations thereof. In some examples, the one or more perception-related attributes of the user include one or more characteristics of one or more fixations by one or more eyes of the user. The one or more characteristics include at least one of frequency, duration, timing, eye position, and eye movement. In some examples, the one or more perception-related attributes of the user include one or more characteristics of one or more pupil dilations by one or more eyes of the user. The one or more characteristics include at least one of frequency, duration, timing, pupil dilation level, eye position, and eye movement. In some examples, the one or more perception-related attributes of the user include one or more characteristics of one or more blinks by one or more eyelids of the user. The one or more characteristics include at least one of frequency, duration, timing, blink rate, eye position, and eye movement. In some examples, the one or more perception-related attributes of the user include one or more characteristics of one or more squints by one or more eyelids of the user. The one or more characteristics include at least one of frequency, duration, timing, squint level, eye position, and eye movement.

[0154]

[0168] In some examples, the imaging system is configured to receive and can receive sensor data captured by one or more sensors. The sensor data is indicative of one or more eyes of the user. The imaging system is further configured to determine and can determine one or more perception-related attributes of the user based on the sensor data. In some examples, the imaging system includes one or more sensors. In some examples, the one or more sensors are configured to capture and can capture the sensor data. Examples of the one or more sensors include the user-facing sensor 205, the additional sensor 208, the sensor 810, the third camera 330C, the fourth camera 330D, the first camera 430A, the second camera 430B, the input device 1145, other sensors described herein, or combinations thereof. Examples of sensor data include sensor data captured by any of the sensors listed in the previous sentence, including, for example, the sensor data 805. In some examples, the one or more sensors include one or more image sensors, and the sensor data includes one or more images, videos, or combinations thereof. In some examples, the sensor data includes a representation of one or more eyes of the user.

[0155]

[0169] In some examples, determining the user's level of perception of the virtual content includes using one or more perception-related attributes of the user as input to one or more trained machine learning systems. Examples of the one or more trained ML systems include the attribute engine 230, the perception engine 235, the understanding engine 240, the first trained ML model 825, the second trained ML model 835, the third trained ML model 865, the NN 900, or combinations thereof. In some examples, the imaging system is configured to receive, and can receive, feedback through a user interface corresponding to the user's level of perception of the virtual content. In some examples, the imaging system is configured to update, and can update, the one or more trained machine learning systems based on the feedback. In some examples, the imaging system includes a feedback engine 260 that the imaging system uses to receive the feedback and update the trained ML system with further training and / or learning of the trained ML system. An example of feedback includes feedback 880. An example of training includes training 885.

[0156]

[0170] In some examples, the imaging system is configured to and can determine a level of understanding of the virtual content by the user based on the user's level of perception of the virtual content. In some examples, determining the level of perception of the virtual content by the user includes determining a level of understanding of the virtual content by the user. The imaging system is configured to and can determine modifications to the display settings based on the level of understanding and / or the level of perception. In some examples, determining the level of understanding of the virtual content by the user is based on one or more perception-related attributes of the user, one or more characteristics of the virtual content, contextual data, historical information associated with the user, a user profile of the user, an assessment of the complexity of the virtual content, an assessment of the uniqueness of the virtual content, or a combination thereof. In some examples, the imaging system is configured to and can receive historical information associated with the user. In some examples, determining the level of understanding of the virtual content by the user is based on historical information about the user. In some examples, the user profile includes historical data associated with the user. In some examples, the contextual data includes one or more reactions by the user to the virtual content. In some examples, the contextual data includes a location of the XR system.

[0157]

[0171] In some examples, the imaging system can determine the grasping level using the imaging system's perception engine 235, the imaging system's grasping engine 240, the imaging system's virtual content assessment engine 245, the imaging system's second trained ML model 835, the imaging system's third trained ML model 865, the neural network 900, or a combination thereof. Examples of the grasping level include a grasping level determined using the grasping engine 240, a grasping level 870, a grasping level determined using the NN 900, or a combination thereof.

[0158]

[0172] In some examples, the imaging system is configured and can determine characteristics of the virtual content, for example using the virtual content rating engine 245, and determining the user's level of perception and / or understanding of the virtual content is based on the characteristics of the virtual content. Examples of characteristics include a rating by the virtual content rating engine 245, a rating 860 associated with the virtual content 840, or a combination thereof. In some examples, the imaging system is configured and can determine a complexity level of the virtual content. In some aspects, determining the user's level of perception and / or understanding of the virtual content is based on the complexity level of the virtual content. In some examples, the imaging system is configured and can determine a uniqueness level of the virtual content. In some aspects, determining the user's level of perception and / or understanding of the virtual content is based on the uniqueness level of the virtual content. In some examples, the imaging system is configured and can determine a uniqueness level of the virtual content relative to the environment. In some aspects, determining the user's level of perception and / or understanding of the virtual content is based on the uniqueness level of the virtual content relative to the environment.

[0159]

[0173] In some examples, determining a level of perception of the virtual content by the user includes determining that the user perceived the virtual content. In some examples, determining a level of perception of the virtual content by the user includes determining that the user does not perceive the virtual content. In some examples, determining a level of perception of the virtual content by the user includes determining that the user perceived the virtual content to a first perception level of the plurality of perception levels. In some examples, determining a level of perception of the virtual content by the user includes determining that the user perceived the virtual content to a second perception level of the plurality of perception levels. In some aspects, the first perception level is greater than the second perception level and the second perception level is less than the first perception level. In some aspects, the second perception level is greater than the first perception level and the first perception level is less than the second perception level.

[0160]

[0174] In some examples, determining the user's level of perception of the virtual content includes determining a confidence level corresponding to the user's level of perception of the virtual content. In some aspects, modifications to the display settings are based on the confidence level. The confidence level may be provided by a machine learning system, such as the attribution engine 230, the perception engine 235, the understanding engine 240, the first trained ML model 825, the second trained ML model 835, the third trained ML model 865, the NN 900, or a combination thereof.

[0161]

[0175] In some examples, determining the user's level of perception of the virtual content includes, for example, identifying one or more gestures of the user using one or more sensors (e.g., an image sensor, a camera, the user focus sensor 205, the environmental focus sensor 210). The imaging system can track the user's hand using sensor data from the environmental focus sensor 210. For example, the imaging system can determine if the user's hand is pointing and / or gesturing towards the virtual content and increase the perception level to a high level of perception. The imaging system can determine if the user's hand is pointing and / or gesturing away from the virtual content and decrease the perception level to a low level of perception.

[0162]

[0176] In some examples, the virtual content includes a string of characters. The imaging system is configured and capable of determining a degree of comprehension of the string by the user based on a perception level for the virtual content, a length of the string, a complexity of the string, and / or a uniqueness of the string. The complexity and / or uniqueness of the string may be an evaluation (e.g., evaluation 860) by the virtual content evaluation engine 245 of the imaging system. For example, if the perception level indicates that the user glanced at the virtual content quickly, but the string is short, not complex, and / or not unique, the degree of comprehension of the string may still be high. On the other hand, if the perception level indicates that the user glanced at the virtual content quickly, but the string is long, complex, and / or unique, the degree of comprehension of the string may be low. If the perception level indicates that the user viewed the virtual content for a very long time, the degree of comprehension of the string may be high even if the string is long, complex, and / or unique.

[0163]

[0177] In operation 1015, the imaging system is configured and capable of determining a modification to a display setting corresponding to the virtual content based on the user's level of perception of the virtual content. Examples of modifications to the display setting include a modification from first display setting 250 to second display setting 255 in FIG. 2, a modification from display setting 520 to display setting 555 in FIGS. 5A-5B, a modification 875 to display setting 837, or a combination thereof.

[0164]

[0178] In some examples, the modification to the display settings corresponding to the virtual content includes causing the display to stop displaying at least a portion of the virtual content. An example of this is shown in FIG. 5B, where the modification from display setting 520 to display setting 555 causes the virtual content 535 to no longer be overlaid on top of the image 505, as shown in FIG. 5B. In some examples, the modification to the display settings corresponding to the virtual content includes causing the display to display at least a portion of the virtual content more prominently than before the modification. An example of this is shown in FIG. 5B, where the modification from display setting 520 to display setting 555 increases the size, font size, and level of detail of the virtual content 525, as shown in FIG. 5B. In some examples, the modification to the display settings corresponding to the virtual content includes modification to one or more characteristics of the virtual content, where the one or more characteristics include at least one of a position, an orientation, a depth, a size, a color, a font size, a font color, a font, a language, a layout, or a combination thereof. An example of this is shown in FIG. 5B, where modifying display setting 520 to display setting 555 increases the size, font size, and level of detail of virtual content 525, as shown in FIG. 5B, and changes the position and depth of virtual content 530 so that it is partially visible behind the statue of Red Auerbach.

[0165]

[0179] In some examples, modifications to the display settings are based on the likelihood that the virtual content will be re-seen by the user within a threshold time. For example, in the example virtual content 730 of FIG. 7, the threshold time may be shortened as a car is rapidly approaching the user, and modifications may be to quickly increase the size of the virtual content 730 to ensure that the user is alerted if it appears unlikely (e.g., based on perception and / or grasp levels) that the user will re-seen the virtual content within the threshold time.

[0166]

[0180] In some examples, the imaging system may include means for displaying virtual content using the display in accordance with display settings associated with the virtual content, where an environment is viewable using the display when the virtual content is displayed by the display; means for determining a level of perception of the virtual content displayed by a user using the display based on one or more perception-related attributes of the user; and means for determining modifications to the display settings corresponding to the virtual content based on the level of perception of the virtual content by the user.

[0167]

[0181] In some examples, the means for displaying the virtual content includes image capture and processing system 100, image capture device 105A, image processing device 105B, image processor 150, ISP 154, host processor 152, XR system 200, virtual content generator 215, environmental facing sensor 210, compositor 220, display 225, display 340, first camera 330A, second camera 330B, display 440, third camera 430C, fourth camera 430D, virtual content 525, virtual content 530, virtual content 535, virtual content 630, virtual content 730, computing system 1100, or combinations thereof. In some examples, the means for determining the perception level includes the image capture and processing system 100, the image capture device 105A, the image processing device 105B, the image processor 150, the ISP 154, the host processor 152, the XR system 200, the user-facing sensor 205, the attribute engine 230, the perception engine 235, the understanding engine 240, the virtual content scoring engine 245, the feedback engine 260, the first trained ML model 825, the second trained ML model 835, the third trained ML model 865, the NN 900, the computing system 1100, or a combination thereof. In some examples, the means for determining modifications to the display settings includes the XR system 200, the synthesizer 220, the attribute engine 230, the perception engine 235, the understanding engine 240, the virtual content scoring engine 245, the feedback engine 260, the first trained ML model 825, the second trained ML model 835, the third trained ML model 865, the NN 900, the computing system 1100, or a combination thereof.

[0168]

[0182] In some examples, the processes described herein (e.g., the processes of FIGS. 1, 2, 8, 9, 10 and / or other processes described herein) may be performed by a computing device or apparatus. 1, 2, 8, 9, and / or 10 may be performed by image capture and processing system 100, image capture device 105A, image processing device 105B, image processor 150, ISP 154, host processor 152, XR system 200, HMD 310, mobile handset 410, an imaging device that captures image 505 and overlays virtual content 525-535, an imaging device that captures image 605 and overlays virtual content 630, an imaging device that captures image 705 and overlays virtual content 730, the imaging device of FIG. 8, the first trained ML model 825 of the second trained ML model 835, the third trained ML model 865, the neural network 900, the computing system 1100, the processor 1110, or a combination thereof.

[0169]

[0183] The computing device may include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, AR glasses, a network-connected watch or smartwatch, or other wearable device), a server computer, an autonomous vehicle or a computing device of an autonomous vehicle, a robotic device, a television, and / or any other computing device having resource capabilities to perform the processes described herein, including the processes of FIG. 1, FIG. 2, FIG. 8, FIG. 9, and / or FIG. 10. In some cases, a computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform steps of the processes described herein. In some examples, the computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive Internet Protocol (IP)-based data or other types of data.

[0170]

[0184] Components of a computing device may be implemented in circuitry. For example, components may include and / or be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other suitable electronic circuitry), and / or may include and / or be implemented using computer software, firmware, or any combination thereof, to perform various operations described herein.

[0171]

[0185] The processes of Figures 1, 2, 8, 9, and / or 10 are illustrated as logical flow diagrams, block diagrams, or conceptual diagrams, whose operations represent sequences of operations that may be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the described operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations can be combined in any order and / or in parallel to implement a process.

[0172]

[0186] Additionally, the processes of Figures 1, 2, 8, 9, and / or 10, and / or other processes described herein may be performed under the control of one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that collectively execute on one or more processors, by hardware, or a combination thereof. As mentioned above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0173]

[0187] 11 is a diagram illustrating an example of a system for implementing some aspects of the present technology. In particular, FIG. 11 illustrates an example of a computing system 1100, which may be, for example, an internal computing system, a remote computing system, a camera, or any computing device constituting any of the components thereof, in which the components of the system communicate with each other using a connection 1105. The connection 1105 may be a physical connection using a bus, or a direct connection to a processor 1110, such as in a chipset architecture. The connection 1105 may also be a virtual connection, a network connection, or a logical connection.

[0174]

[0188] In some embodiments, computing system 1100 is a distributed system in which the functionality described in this disclosure may be distributed across one data center, multiple data centers, a peer network, etc. In some embodiments, one or more of the system components described represent many components, each performing some or all of the functionality that is the subject of the component description. In some embodiments, the components may be physical or virtual devices.

[0175]

[0189] The exemplary system 1100 includes at least one processing unit (CPU or processor) 1110 and connections 1105 coupling various system components to the processor 1110, including system memory 1115, such as read only memory (ROM) 1120 and random access memory (RAM) 1125. The computing system 1100 may include a cache 1112 of high speed memory either directly connected to the processor 1110, in close proximity to the processor 1110, or integrated as part of the processor 1110.

[0176]

[0190] Processor 1110 may include any general purpose processor, as well as hardware or software services, such as services 1132, 1134, and 1136, stored in storage device 1130 and configured to control processor 1110, as well as special purpose processors whose software instructions are built into the actual processor design. Processor 1110 may essentially be a completely self-contained computing system, including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0177]

[0191] To enable user interaction, computing system 1100 includes input devices 1145, which can represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, speech, etc. Computing system 1100 may also include output devices 1135, which can be one or more of several output mechanisms. In some cases, a multi-modal system may enable a user to provide multiple types of input / output to communicate with computing system 1100. Computing system 1100 may generally include a communication interface 1140, which can govern and manage user input and system output.The communications interface may be any of the following: audio jack / plug, microphone jack / plug, universal serial bus (USB) port / plug, Apple® Lightning® port / plug, Ethernet® port / plug, fiber optic port / plug, proprietary wired port / plug, BLUETOOTH® wireless signal transmission, BLUETOOTH® low energy (BLE) wireless signal transmission, IBEACON® wireless signal transmission, radio-frequency identification (RFID) wireless signal transmission, near-field communications (NFC) wireless signal transmission, dedicated short range communication (DSRC) wireless signal transmission, 802.11 Wi-Fi wireless signal transmission, wireless local area network (WLAN) signal transmission, Visible Light Communication (VLC), Worldwide Interoperability for Microwave Access (WLAN), and Bluetooth® wireless signal transmission. The wireless communication device may perform or facilitate the reception and / or transmission of wired or wireless communications using wired and / or wireless transceivers, including those utilizing WiMAX (Wireless Access), infrared (IR) communications wireless signal transmission, Public Switched Telephone Network (PSTN) signal transmission, Integrated Services Digital Network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad-hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or any combination thereof.The communication interface 1140 may also include one or more Global Navigation Satellite System (GNSS) receivers or transceivers used to determine the location of the computing system 1100 based on reception of one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the United States Global Positioning System (GPS), the Russian Global Navigation Satellite System (GLONASS), the Chinese BeiDou Navigation Satellite system (BDS), and the European Galileo GNSS. There is no constraint to operate with any particular hardware arrangement, and therefore the basic features herein may be easily substituted for improved hardware or firmware arrangements as they are developed.

[0178]

[0192] The storage device 1130 may be a non-volatile and / or non-transitory and / or computer readable memory device, such as a magnetic cassette, a flash memory card, a solid state memory device, a digital versatile disk, a cartridge, a floppy disk, a flexible disk, a hard disk, a magnetic tape, a magnetic strip / stripe, any other magnetic storage medium, a flash memory, a memristor memory, any other solid state memory, a compact disc read only memory (CD-ROM) optical disk, a rewritable compact disc (CD) optical disk, a digital video disk (DVD) optical disk, a blu-ray disc (BDD) optical disk, a holographic optical disk, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a memory stick card, a smart card chip, an EMV chip, a subscriber identity module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (ICC), a circuit (IC) chips / cards, random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash EPROM (FLASH EPROM), cache memory (L1 / L2 / L3 / L4 / L5 / L#), resistive random access memory (RRAM),The memory may be a hard disk or other type of computer readable medium capable of storing data that is accessible by a computer, such as random-access memory (RRAM / ReRAM), phase change memory (PCM), spin transfer torque RAM (STT-RAM), another memory chip or cartridge, and / or a combination thereof.

[0179]

[0193] The storage devices 1130 may include software services, servers, services, etc. that cause the system to perform functions when code defining such software is executed by the processor 1110. In some embodiments, hardware services that perform certain functions may include software components stored in a computer-readable medium in conjunction with the necessary hardware components, such as the processor 1110, connections 1105, output devices 1135, etc., to perform the functions.

[0180]

[0194] The term "computer-readable medium" as used herein includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, storing, or transporting instructions and / or data. Computer-readable media may also include non-transitory media on which data is stored and does not include carrier waves and / or transitory electronic signals propagating wirelessly or via wired connections. Examples of non-transitory media may include, but are not limited to, magnetic disks or tapes, optical storage media such as compact disks (CDs) or digital versatile disks (DVDs), flash memory, memories, or memory devices. Computer-readable media may have code and / or machine-executable instructions stored thereon, which may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted using any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0181]

[0195] In some embodiments, computer readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when mentioned, non-transitory computer readable storage media specifically excludes media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0182]

[0196] Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, it will be understood by those skilled in the art that the embodiments may be practiced without these specific details. For ease of explanation, in some cases, the present technology may be presented as including individual functional blocks, including devices, device components, steps or routines in a method embodied in software, or functional blocks comprising a combination of hardware and software. Additional components may be used other than the components shown in the figures and / or described herein. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the embodiments in unnecessary detail. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail so as to avoid obscuring the embodiments.

[0183]

[0197] Individual embodiments may be described above as a process or method that is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although the flowcharts may describe operations as a sequential process, many of the operations may be performed in parallel or simultaneously. In addition, the order of operations may be rearranged. A process terminates when its operations are completed, but may have additional steps not included in the diagrams. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the function to the calling function or to the main function.

[0184]

[0198] The processes and methods according to the examples described above may be implemented using computer-executable instructions stored on or otherwise available from a computer-readable medium. Such instructions may include, for example, instructions and data that cause a general-purpose computer, a special-purpose computer, or a processing device to perform a function or group of functions, or in some cases configure a general-purpose computer, a special-purpose computer, or a processing device to perform a function or group of functions. Portions of the computer resources used may be accessible over a network. The computer-executable instructions may be, for example, binary, intermediate format instructions, such as assembly language, firmware, source code, etc. Examples of computer-readable media that may be used to store instructions, information used, and / or information created during the methods according to the described examples include magnetic or optical disks, flash memory, USB devices with non-volatile memory, network-attached storage devices, etc.

[0185]

[0199] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) to perform the necessary tasks may be stored in a computer-readable or machine-readable medium. A processor may perform the necessary tasks. Typical examples of form factors include laptops, smartphones, mobile phones, tablet devices or other small-footprint personal computers, personal digital assistants, rack-mounted devices, standalone devices, and the like. The functionality described herein may also be embodied in peripheral devices or add-in cards. Such functionality may also be implemented on a circuit board among different chips, or on different processes executing in a single device, as further examples.

[0186]

[0200] The instructions, media for carrying such instructions, computing resources for executing such instructions, and other structures for supporting such computing resources are exemplary means for providing the functionality described in this disclosure.

[0187]

[0201] In the above description, aspects of the present application are described with reference to specific embodiments thereof, but those skilled in the art will recognize that the present application is not limited thereto. Thus, while exemplary embodiments of the present application have been described in detail herein, it should be understood that the inventive concepts may be embodied and employed in various other ways, and that the appended claims are intended to be construed to include such variations, except as limited by the prior art. The various features and aspects of the present application described above may be used individually or jointly. Moreover, the embodiments may be utilized in any number of environments and applications other than those described herein without departing from the broader spirit and scope of the present specification. Thus, the present specification and drawings should be regarded as illustrative and not restrictive. For purposes of illustration, methods have been described in a particular order. It should be understood that in alternative embodiments, the methods may be performed in an order different from that described.

[0188]

[0202] Those skilled in the art will understand that the less than ("<") and greater than (">") symbols or terms used herein may be replaced with the less than or equal to ("≦") and greater than or equal to ("≧") symbols, respectively, without departing from the scope of this description.

[0189]

[0203] When a component is described as being "configured to" perform a particular operation, such configuration may be achieved, for example, by designing electronic circuitry or other hardware to perform the operation, by programming a programmable electronic circuitry (e.g., a microprocessor or other suitable electronic circuitry) to perform the operation, or any combination thereof.

[0190]

[0204] The phrase "coupled to" refers to any component that is physically connected, either directly or indirectly, to another component and / or that is in communication, either directly or indirectly, with another component (e.g., connected to the other component via a wired or wireless connection and / or other suitable communication interface).

[0191]

[0205] Claim language or other language reciting "at least one of" a set and / or "one or more" of a set indicates that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, a claim language reciting "at least one of A and B" means A, B, or A and B. In another example, a claim language reciting "at least one of A, B, and C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more" of a set does not limit the set to the items listed in the set. For example, a claim language reciting "at least one of A and B" can mean A, B, or A and B, and can further include items not listed in the set of A and B.

[0192]

[0206] The various exemplary logic blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly illustrate this interchangeability of hardware and software, the various exemplary components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in varying ways for each particular application, and such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0193]

[0207] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as a general purpose computer, a wireless communication device handset, or an integrated circuit device having multiple uses, including applications in wireless communication device handsets and other devices. Any features described as modules or components may be implemented together in an integrated logic device, or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be realized at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, perform one or more of the methods described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may comprise a memory or data storage medium, such as random access memory (RAM), such as synchronous dynamic random access memory (SDRAM), read only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read only memory (EEPROM), FLASH memory, magnetic or optical data storage medium, etc. The techniques may additionally or alternatively be realized at least in part by a computer-readable communications medium, such as a propagated signal or wave, which may carry or communicate program code in the form of instructions or data structures and which may be accessed, read, and / or executed by a computer.

[0194]

[0208] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Such a processor may be configured to perform any of the techniques described in this disclosure. A general purpose processor may be a microprocessor, alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Thus, the term "processor" as used herein may refer to any of the above structures, any combination of the above structures, or any other structure or apparatus suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein may be provided in dedicated software or hardware modules configured for encoding and decoding, or may be incorporated within a combined video encoder-decoder (CODEC).

[0195]

[0209] Exemplary aspects of the present disclosure include the following.

[0196]

[0210] Aspect 1: An extended reality (XR) system, wherein the device comprises a memory and one or more processors coupled to the memory, the one or more processors configured to: display virtual content using a display in accordance with display settings associated with the virtual content; when the virtual content is displayed by the display, an environment is viewable using the display; determine a perception level of the virtual content displayed by the user using the display based on one or more perception-related attributes of the user; and determine modifications to the display settings corresponding to the virtual content based on the perception level of the virtual content by the user.

[0197]

[0211] Aspect 2. The XR system of aspect 1, wherein one or more perception-related attributes of the user are associated with one or more eyes of the user.

[0198]

[0212] Aspect 3. An XR system as described in aspect 1 or 2, wherein the environment is viewable using the display based at least in part on light from the environment passing through at least a portion of the display.

[0199]

[0213] Aspect 4. An XR system as described in any of aspects 1 to 3, wherein the environment is viewable using a display based at least in part on one or more processors being configured to cause a view of the environment to be displayed by the display.

[0200]

[0214] Aspect 5. An XR system as described in any of aspects 1 to 4, wherein the one or more processors are configured to use one or more perception-related attributes of the user as input to one or more trained machine learning systems to determine the user's level of perception of the virtual content.

[0201]

[0215] Aspect 6. The XR system of aspect 5, wherein the one or more processors are configured to receive feedback through the user interface corresponding to a level of perception of the virtual content by the user, and update the one or more trained machine learning systems based on the feedback.

[0202]

[0216] Aspect 7. An XR system as described in any of aspects 1 to 6, wherein the one or more processors are configured to receive sensor data captured by one or more sensors, the sensor data being indicative of one or more eyes of a user, and determine one or more perception-related attributes of the user based on the sensor data.

[0203]

[0217] An XR system as described in embodiment 7, further comprising one or more sensors.

[0204]

[0218] Example 9. An XR system as described in any of Examples 1 to 8, wherein the one or more processors are configured to determine one or more perception-related attributes of a user based on sensor data captured by one or more image sensors, the sensor data including one or more images of one or more eyes of the user.

[0205]

[0219] Aspect 10. An XR system as described in any of aspects 1 to 9, wherein the one or more processors are configured to determine a user's level of understanding of the virtual content based on the user's level of perception of the virtual content, and to determine modifications to the display settings based on the level of perception, the one or more processors are configured to determine modifications to the display settings based on the level of understanding.

[0206]

[0220] Aspect 11. The XR system of aspect 10, wherein one or more processors are configured to receive historical information associated with a user, and to determine a level of understanding of the virtual content by the user, the one or more processors are configured to determine a level of understanding of the virtual content by the user based on the historical information about the user.

[0207]

[0221] Aspect 12. An XR system as described in any of aspects 1 to 11, wherein the one or more processors are configured to determine characteristics of the virtual content, and in order to determine a level of perception of the virtual content by the user, the one or more processors are configured to determine the level of perception further based on the characteristics of the virtual content.

[0208]

[0222] Aspect 13. An XR system described in any of aspects 1 to 12, wherein the one or more processors are configured to determine a complexity level of the virtual content, and in order to determine a user's perception level of the virtual content, the one or more processors are configured to determine the perception level based on the complexity level of the virtual content.

[0209]

[0223] Aspect 14. An XR system described in any of aspects 1 to 13, wherein one or more processors are configured to determine a uniqueness level of the virtual content, and in order to determine a user's perception level of the virtual content, the one or more processors are configured to determine the perception level based on the uniqueness level of the virtual content.

[0210]

[0224] Aspect 15. An XR system described in any of aspects 1 to 14, wherein modifications to display settings corresponding to the virtual content include causing the display to stop displaying at least a portion of the virtual content.

[0211]

[0225] Aspect 16. An XR system described in any of aspects 1 to 15, wherein modifications to display settings corresponding to virtual content include causing the display to display at least a portion of the virtual content more prominently than before the modifications.

[0212]

[0226] Aspect 17. An XR system described in any of aspects 1 to 16, wherein modifications to display settings corresponding to the virtual content include modifications to one or more characteristics of the virtual content, the one or more characteristics including at least one of position, orientation, depth, size, color, font size, font color, font, language, and layout.

[0213]

[0227] Aspect 18. An XR system described in any of aspects 1 to 17, wherein one or more processors are configured to determine that the user has perceived the virtual content, in order to determine a level of perception of the virtual content by the user.

[0214]

[0228] Aspect 19. An XR system described in any of aspects 1 to 18, wherein, in order to determine a user's level of perception of virtual content, one or more processors are configured to determine that the user does not perceive the virtual content.

[0215]

[0229] Aspect 20. An XR system according to any one of aspects 1 to 19, wherein modifications to the display settings are based on the likelihood that the virtual content will be re-viewed by the user within a threshold time.

[0216]

[0230] Aspect 21. An XR system described in any of aspects 1 to 20, wherein, to determine a user's level of perception of the virtual content, one or more processors are configured to determine a confidence level corresponding to the user's level of perception of the virtual content, and modifications to the display settings are based on the confidence level.

[0217]

[0231] Aspect 22. An XR system described in any of aspects 1 to 21, wherein the one or more perception-related attributes of the user include one or more eye positions of one or more eyes of the user relative to the virtual content.

[0218]

[0232] Aspect 23. An XR system described in any of aspects 1 to 22, wherein the one or more perception-related attributes of the user include one or more characteristics of one or more saccades by one or more eyes of the user, the one or more characteristics including at least one of frequency, duration, timing, saccade velocity, saccade amplitude, eye position, and eye movement.

[0219]

[0233] Aspect 24. An XR system described in any of aspects 1 to 23, wherein the one or more perception-related attributes of the user include one or more characteristics of one or more fixations by one or more eyes of the user, the one or more characteristics including at least one of frequency, duration, timing, eye position, and eye movement.

[0220]

[0234] Aspect 25. An XR system described in any of aspects 1 to 24, wherein the one or more perception-related attributes of the user include one or more characteristics of one or more pupil dilations by one or more eyes of the user, the one or more characteristics including at least one of frequency, duration, timing, pupil dilation level, eye position, and eye movement.

[0221]

[0235] Aspect 26. An XR system described in any of aspects 1 to 25, wherein the one or more perception-related attributes of the user include one or more characteristics of one or more blinks by one or more eyelids of the user, the one or more characteristics including at least one of frequency, duration, timing, blink rate, eye position, and eye movement.

[0222]

[0236] Aspect 27. An XR system described in any of aspects 1 to 26, wherein the one or more perception-related attributes of the user include one or more characteristics of one or more squintings by one or more eyelids of the user, the one or more characteristics including at least one of frequency, duration, timing, level of squinting, eye position, and eye movement.

[0223]

[0237] Aspect 28. An XR system described in any of aspects 1 to 27, wherein one or more processors are configured to determine a user's degree of comprehension of a string of characters based on a perceptual level of the virtual content and a length of the string of characters, and the virtual content includes the string of characters.

[0224]

[0238] Aspect 29. An XR system described in any of aspects 1 to 28, further comprising a display.

[0225]

[0239] Aspect 30. An XR system according to any one of aspects 1 to 29, wherein the XR system includes at least one of a mobile handset, a wireless communication device, and a head-mounted display.

[0226]

[0240] Aspect 31. An XR system described in any of aspects 1 to 30, wherein one or more processors are configured to determine a level of uniqueness of the virtual content relative to the environment, and in order to determine a user's level of perception of the virtual content, the one or more processors are configured to determine the level of perception based on the level of uniqueness of the virtual content relative to the environment.

[0227]

[0241] Aspect 32. An XR system as described in any of aspects 1 to 31, wherein one or more processors are configured to determine at least one of one or more perception-related attributes of a user based at least in part on sensor data including representations of one or more eyes of the user, and one or more sensors are configured to capture the sensor data.

[0228]

[0242] Aspect 33. An XR system described in any of aspects 1 to 32, wherein the one or more perception-related attributes of the user include at least one of: one or more attributes of the user's one or more eyes, one or more attributes of the user's one or more facial expressions, and one or more gestures of the user.

[0229]

[0243] Aspect 34. An XR system described in any of aspects 1 to 33, wherein, to determine a user's level of perception of the virtual content, one or more processors are configured to determine the user's level of understanding of the virtual content based on one or more perception-related attributes of the user.

[0230]

[0244] Aspect 35. The XR system of aspect 34, wherein to determine a user's level of understanding of the virtual content, one or more processors are configured to determine the user's level of understanding of the virtual content based on one or more perception-related attributes of the user and at least one of one or more characteristics of the virtual content, the contextual data, and a user profile of the user.

[0231]

[0245] Aspect 36. The XR system of aspect 35, wherein the user profile includes historical data associated with the user.

[0232]

[0246] Aspect 37. An XR system as described in aspect 35 or 36, wherein the context data includes one or more reactions by the user to the virtual content.

[0233]

[0247] Example 38. An XR system according to any of examples 35 to 37, wherein the context data includes a location of the XR system.

[0234]

[0248] Aspect 39. An XR system described in any of aspects 1 to 38, wherein to determine a user's perception level of the virtual content, one or more processors are configured to determine that the user has perceived the virtual content up to a first perception level among multiple perception levels.

[0235]

[0249] Aspect 40. An XR system described in any of aspects 1 to 39, wherein to determine a user's perception level of the virtual content, the one or more processors are configured to determine that the user has perceived the virtual content up to a second perception level of the multiple perception levels.

[0236]

[0250] Aspect 41. A method of extended reality (XR) operation, the method including: displaying virtual content using a display in accordance with display settings associated with the virtual content, wherein an environment is viewable using the display when the virtual content is displayed by the display; determining a level of perception of the virtual content displayed by a user using the display based on one or more perception-related attributes of the user; and determining modifications to the display settings corresponding to the virtual content based on the level of perception of the virtual content by the user.

[0237]

[0251] Aspect 42. The method of aspect 41, wherein one or more perception-related attributes of the user are associated with one or more eyes of the user.

[0238]

[0252] Aspect 43. The method of aspect 41 or 42, wherein the environment is viewable using the display based at least in part on light from the environment passing through at least a portion of the display.

[0239]

[0253] Embodiment 44. The method of any of embodiments 41 to 43, wherein the environment is viewable using the display based at least in part on causing a view of the environment to be displayed by the display.

[0240]

[0254] Aspect 45. A method as described in any of aspects 41 to 44, wherein determining a level of perception of the virtual content by the user includes using one or more perception-related attributes of the user as input to one or more trained machine learning systems.

[0241]

[0255] Aspect 46. The method of aspect 45, further comprising receiving feedback through a user interface corresponding to a user's level of perception of the virtual content, and updating one or more trained machine learning systems based on the feedback.

[0242]

[0256] Aspect 47. A method according to any of aspects 41 to 46, further comprising receiving sensor data captured by one or more sensors, the sensor data being indicative of one or more eyes of a user, and determining one or more perception-related attributes of the user based on the sensor data.

[0243]

[0257] Embodiment 48. A method according to any of embodiments 41 to 47, wherein the method is performed by an XR system including one or more sensors.

[0244]

[0258] Aspect 49. A method as described in any of aspects 41 to 48, further comprising determining one or more perception-related attributes of the user based on sensor data captured by one or more image sensors, the sensor data including one or more images of one or more eyes of the user.

[0245]

[0259] Aspect 50. The method of any of aspects 41 to 49, further comprising determining a user's level of understanding of the virtual content based on the user's level of perception of the virtual content, and determining modifications to the display settings based on the level of perception comprises determining modifications to the display settings based on the level of understanding.

[0246]

[0260] Aspect 51. The method of aspect 50, further comprising receiving historical information associated with the user, wherein determining a level of understanding of the virtual content by the user is based on the historical information regarding the user.

[0247]

[0261] Aspect 52. The method of any one of aspects 41 to 51, further comprising determining a characteristic of the virtual content, wherein determining a level of perception of the virtual content by the user is based on the characteristic of the virtual content.

[0248]

[0262] Aspect 53. The method of any of aspects 41 to 52, further comprising determining a complexity level of the virtual content, wherein determining the user's perception level of the virtual content is based on the complexity level of the virtual content.

[0249]

[0263] Aspect 54. The method of any one of aspects 41 to 53, further comprising determining a uniqueness level of the virtual content, wherein determining the user's perception level of the virtual content is based on the uniqueness level of the virtual content.

[0250]

[0264] Aspect 55. The method of any of aspects 41-54, wherein the modification to the display settings corresponding to the virtual content includes causing the display to stop displaying at least a portion of the virtual content.

[0251]

[0265] Aspect 56. The method of any of aspects 41-55, wherein a modification to a display setting corresponding to the virtual content includes causing the display to display at least a portion of the virtual content more prominently than before the modification.

[0252]

[0266] Aspect 57. A method as described in any of aspects 41 to 56, wherein the modifications to the display settings corresponding to the virtual content include modifications to one or more characteristics of the virtual content, the one or more characteristics including at least one of position, orientation, depth, size, color, font size, font color, font, language, and layout.

[0253]

[0267] Aspect 58. The method of any of aspects 41-57, wherein determining a level of perception of the virtual content by the user includes determining that the user perceived the virtual content.

[0254]

[0268] Aspect 59. The method of any of aspects 41-58, wherein determining a level of perception of the virtual content by the user includes determining that the user does not perceive the virtual content.

[0255]

[0269] Aspect 60. The method of any of aspects 41-59, wherein the modification to the display settings is based on a likelihood that the virtual content will be re-viewed by the user within a threshold time.

[0256]

[0270] Aspect 61. The method of any of aspects 41 to 60, wherein determining the user's level of perception of the virtual content includes determining a confidence level corresponding to the user's level of perception of the virtual content, and modifications to the display settings are based on the confidence level.

[0257]

[0271] Aspect 62. The method of any of aspects 41 to 61, wherein the one or more perception-related attributes of the user include one or more eye positions of one or more eyes of the user relative to the virtual content.

[0258]

[0272] Aspect 63. A method according to any of aspects 41 to 62, wherein the one or more perception-related attributes of the user include one or more characteristics of one or more saccades by one or more eyes of the user, the one or more characteristics including at least one of frequency, duration, timing, saccade velocity, saccade amplitude, eye position, and eye movement.

[0259]

[0273] Aspect 64. A method according to any of aspects 41 to 63, wherein the one or more perception-related attributes of the user include one or more characteristics of one or more fixations by one or more eyes of the user, the one or more characteristics including at least one of frequency, duration, timing, eye position, and eye movement.

[0260]

[0274] Aspect 65. A method as described in any of aspects 41 to 64, wherein the one or more perception-related attributes of the user include one or more characteristics of one or more pupil dilations by one or more eyes of the user, the one or more characteristics including at least one of frequency, duration, timing, pupil dilation level, eye position, and eye movement.

[0261]

[0275] Aspect 66. A method described in any of aspects 41 to 65, wherein the one or more perception-related attributes of the user include one or more characteristics of one or more blinks by one or more eyelids of the user, the one or more characteristics including at least one of frequency, duration, timing, blink rate, eye position, and eye movement.

[0262]

[0276] Aspect 67. A method as described in any of aspects 41 to 66, wherein the one or more perception-related attributes of the user include one or more characteristics of one or more squintings by one or more eyelids of the user, the one or more characteristics including at least one of frequency, duration, timing, level of squinting, eye position, and eye movement.

[0263]

[0277] Aspect 68. The method of any one of aspects 41 to 67, further comprising determining a degree of comprehension of the character string by the user based on a perceptual level of the virtual content and a length of the character string, wherein the virtual content includes the character string.

[0264]

[0278] Embodiment 69. A method according to any of embodiments 41 to 68, wherein the method is performed by an XR system including a display.

[0265]

[0279] Embodiment 70. The method of any of embodiments 41 to 69, wherein the method is implemented by an XR system including at least one of a mobile handset, a wireless communication device, and a head-mounted display.

[0266]

[0280] Aspect 71. The method of any of aspects 41 to 70, further comprising determining a level of uniqueness of the virtual content relative to the environment, wherein determining the user's level of perception of the virtual content is based on the level of uniqueness of the virtual content relative to the environment.

[0267]

[0281] Aspect 72. The method of any of aspects 41 to 71, further comprising determining at least one of the user's one or more perception-related attributes based at least in part on sensor data including an expression of one or more eyes of the user, wherein the one or more sensors are configured to capture the sensor data.

[0268]

[0282] Aspect 73. A method according to any of aspects 41 to 72, wherein the one or more perception-related attributes of the user include at least one of: one or more attributes of one or more eyes of the user, one or more attributes of one or more facial expressions of the user, and one or more gestures of the user.

[0269]

[0283] Aspect 74. The method of any of aspects 41 to 73, wherein determining a level of perception of the virtual content by the user includes determining a level of understanding of the virtual content by the user based on one or more perception-related attributes of the user.

[0270]

[0284] Aspect 75. The method of aspect 74, wherein determining a user's level of understanding of the virtual content is based on one or more perception-related attributes of the user and at least one of one or more characteristics of the virtual content, contextual data, and a user profile of the user.

[0271]

[0285] Aspect 76. The method of aspect 75, wherein the user profile includes historical data associated with the user.

[0272]

[0286] Aspect 77. The method of aspect 75 or 76, wherein the contextual data includes one or more reactions by the user to the virtual content.

[0273]

[0287] Example 78. The method of any of examples 75 to 77, wherein the method is performed by an XR system and the context data includes a location of the XR system.

[0274]

[0288] Aspect 79. The method of any of aspects 41-78, wherein determining a level of perception of the virtual content by the user includes determining that the user perceived the virtual content to a first perception level of a plurality of perception levels.

[0275]

[0289] Aspect 80. The method of any of aspects 41-79, wherein determining a level of perception of the virtual content by the user includes determining that the user perceived the virtual content to a second perception level of the plurality of perception levels.

[0276]

[0290] Aspect 81: A non-transitory computer-readable medium having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to display virtual content using a display in accordance with display settings associated with the virtual content, determine a level of perception for the virtual content displayed by a user using the display based on one or more perception-related attributes of the user when the virtual content is displayed by the display, and determine modifications to the display settings corresponding to the virtual content based on the user's level of perception for the virtual content.

[0277]

[0291] Example 82: The non-transitory computer-readable medium of example 81, further comprising any of examples 2 to 40 and / or any of examples 42 to 80.

[0278]

[0292] Aspect 83: An apparatus for image processing, comprising: means for displaying virtual content using a display in accordance with display settings associated with the virtual content, where an environment is viewable using the display when the virtual content is displayed by the display; means for determining a perception level of the virtual content displayed by a user using the display based on one or more perception-related attributes of the user; and means for determining modifications to the display settings corresponding to the virtual content based on the user's perception level of the virtual content.

[0279]

[0293] Embodiment 84: The device described in embodiment 83, further comprising any of embodiments 2 to 40 and / or any of embodiments 42 to 80.

Claims

1. at least one memory; one or more processors coupled to the at least one memory; An extended reality (XR) system comprising: the one or more processors: causing virtual content to be displayed using a display in accordance with display settings associated with the virtual content, wherein an environment is viewable using the display when the virtual content is displayed by the display; determining a complexity level of the virtual content; determining a perception level for the virtual content displayed by the user using the display based on one or more perception-related attributes of the user and the complexity level of the virtual content; determining modifications to the display settings corresponding to the virtual content based on the level of perception of the virtual content by the user; The XR system is configured as follows.

2. The XR system of claim 1 , wherein the one or more perception-related attributes of the user are associated with one or more eyes of the user.

3. The XR system of claim 1 , wherein the environment is viewable using the display based at least in part on light from the environment passing through at least a portion of the display.

4. 10. The XR system of claim 1, wherein the environment is viewable using the display based at least in part on the one or more processors being configured to cause a view of the environment to be displayed by the display.

5. wherein the one or more processors are configured to use the one or more perception-related attributes of the user as input to one or more trained machine learning systems to determine the perception level of the user to the virtual content, and optionally the one or more processors are configured to: receiving, through a user interface, feedback corresponding to the level of perception of the virtual content by the user; updating the one or more trained machine learning systems based on the feedback; The XR system of claim 1 , configured to:

6. the one or more processors: receiving sensor data captured by one or more sensors, wherein the sensor data is indicative of one or more eyes of the user; determining the one or more sensory-related attributes of the user based on the sensor data; The XR system of claim 1 , configured to:

7. the one or more processors:

10. The XR system of claim 1, configured to determine the one or more perception-related attributes of the user based on sensory data captured by one or more image sensors, the sensor data including one or more images of one or more eyes of the user.

8. the one or more processors: configured to determine a level of understanding of the virtual content by the user based on the level of perception of the virtual content by the user, and to determine the modifications to the display settings based on the level of perception, the one or more processors are configured to determine the modifications to the display settings based on the level of understanding, and optionally the one or more processors:

2. The XR system of claim 1, further comprising: a processor configured to receive historical information associated with the user, wherein the processor or processors are configured to determine the user's level of understanding of the virtual content based on the historical information about the user to determine the user's level of understanding of the virtual content.

9. the one or more processors:

2. The XR system of claim 1, wherein the one or more processors are configured to determine a uniqueness level of the virtual content, and to determine the level of perception of the virtual content by the user, the one or more processors are configured to determine the level of perception based on the uniqueness level of the virtual content.

10. 10. The XR system of claim 1, wherein the modification to the display settings is based on a confidence level, a likelihood that the virtual content will be re-viewed by the user within a certain time.

11. 2. The XR system of claim 1, wherein to determine the level of perception of the user toward the virtual content, the one or more processors are configured to determine a confidence level corresponding to the level of perception of the user toward the virtual content, and wherein the modification to the display settings is based on the confidence level.

12. 2. The XR system of claim 1, wherein the one or more perception-related attributes of the user include one or more characteristics of one or more pupil dilations by one or more eyes of the user, the one or more characteristics including at least one of frequency, duration, timing, pupil dilation level, eye position, and eye movement.

13. 2. The XR system of claim 1, wherein the one or more perception-related attributes of the user include one or more characteristics of one or more blinks by one or more eyelids of the user, the one or more characteristics including at least one of frequency, duration, timing, blink rate, eye position, and eye movement.

14. 1. A method of extended reality (XR) operation, comprising: displaying virtual content using a display according to display settings associated with the virtual content, wherein an environment is viewable using the display when the virtual content is displayed by the display; determining a complexity level of the virtual content; determining a perception level for the virtual content displayed by the user using the display based on one or more perception-related attributes of the user and the complexity level of the virtual content; determining modifications to the display settings corresponding to the virtual content based on the level of perception of the virtual content by the user; A method comprising:

15. A computer program comprising instructions that, when executed by a computer, cause the computer to: displaying virtual content using a display according to display settings associated with the virtual content, wherein an environment is viewable using the display when the virtual content is displayed by the display; determining a complexity level of the virtual content; determining a perception level for the virtual content displayed by the user using the display based on one or more perception-related attributes of the user and the complexity level of the virtual content; determining modifications to the display settings corresponding to the virtual content based on the level of perception of the virtual content by the user; A computer program that performs the following:

Images