Context-Aware Extended Reality System
Patent Information
- Application Number
- JP2023543354
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-01-27
- Filing Date
- 2021-12-01
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2041-12-01
AI Technical Summary
Extended reality systems often fail to dynamically adjust virtual content based on real-world environmental factors, leading to potential user distraction and safety hazards due to the immersive nature of the systems obscuring real-world interactions.
A context-aware extended reality system that adjusts virtual content based on user distraction levels and environmental factors, using sensors to monitor user engagement and environmental safety, and dynamically modifies content prominence to ensure safe interaction with the real world.
Enables users to fully immerse themselves in XR environments without risking negative interactions with real-world objects by adjusting virtual content to focus on relevant environmental factors, enhancing safety and usability.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
[Technical field]
[0001]
[0001] The present disclosure relates generally to context-aware extended reality systems. In some examples, aspects of the present disclosure relate to adjusting virtual content provided by an extended reality system based on characteristics of a real world environment in which the extended reality system is located. [Background technology]
[0002]
[0002] Extended reality technology may be used to present virtual content to a user and / or combine a real environment from the physical world with a virtual environment to provide the user with an extended reality experience. The term extended reality may encompass virtual reality, augmented reality, mixed reality, and the like. Extended reality systems may enable a user to experience an extended reality environment by overlaying virtual content onto an image of a real-world environment that may be viewed by the user through an extended reality device (e.g., a head-mounted display, extended reality glasses, or other device). Although the goal of many extended reality systems is to create a realistic, interactive, and fully immersive extended reality environment, the extended reality system should also ensure that the virtual content does not create a potentially dangerous situation for the user or otherwise prevent the user from properly interacting with the real-world environment. Improved extended reality systems are needed to dynamically adapt the virtual content based on characteristics of the real-world environment. Summary of the Invention
[0003]
[0003] Systems and techniques for adjusting virtual content in an extended reality system are described herein. According to at least one example, an apparatus for adjusting virtual content in an extended reality system is provided. The exemplary apparatus may include a memory and one or more processors (e.g., configured in a circuit). The one or more processors are configured to: determine a level of distraction of a user of the extended reality system by virtual content provided by the extended reality system; determine whether the level of distraction of the user by the virtual content exceeds a distraction threshold level; and adjust one or more characteristics of the virtual content based on a determination of whether the level of distraction of the user by the virtual content exceeds the distraction threshold level, where the distraction threshold level is determined at least in part based on one or more environmental factors associated with a real-world environment in which the user is located.
[0004] In another example, an apparatus is provided for adjusting virtual content in an extended reality system, the apparatus comprising: means for determining a level of distraction of a user of the extended reality system due to virtual content provided by the extended reality system, means for determining whether the level of distraction of the user due to the virtual content exceeds a distraction threshold level, and means for adjusting one or more characteristics of the virtual content based on the determination of whether the level of distraction of the user due to the virtual content exceeds the distraction threshold level, where the distraction threshold level is determined at least in part based on one or more environmental factors associated with a real-world environment in which the user is located.
[0005]
[0005] In another example, a method for adjusting virtual content in an extended reality system is provided. An exemplary method can include determining a level of distraction of a user of the extended reality system due to virtual content provided by the extended reality system. The method can also include determining whether a level of distraction of the user due to the virtual content exceeds or is less than a distraction threshold level, the distraction threshold level being determined at least in part based on one or more environmental factors associated with a real-world environment in which the user is located. The method further includes adjusting one or more characteristics of the virtual content based on a determination of whether a level of distraction of the user due to the virtual content exceeds or is less than the distraction threshold level.
[0006]
[0006] In another example, a non-transitory computer-readable medium for adjusting virtual content in an extended reality system is provided. The exemplary non-transitory computer-readable medium can store instructions that, when executed by one or more processors, cause the one or more processors to: determine a level of distraction of a user of the extended reality system due to virtual content provided by the extended reality system; determine whether the level of distraction of the user due to the virtual content exceeds or is less than a distraction threshold level; and adjust one or more characteristics of the virtual content based on the determination of whether the level of distraction of the user due to the virtual content exceeds or is less than a distraction threshold level, where the distraction threshold level is determined at least in part based on one or more environmental factors associated with a real-world environment in which the user is located.
[0007]
[0007] In some aspects, determining a level of distraction of a user of an extended reality system by virtual content provided by the extended reality system can include determining a level of engagement between the user and the virtual content. In some examples, determining a level of engagement between the user and the virtual content can include monitoring one or more physical behaviors of the user. The one or more physical behaviors can indicate at least one of a virtual object and a physical object on which the user is focusing. Additionally or alternatively, determining a level of distraction of a user of an extended reality system by virtual content provided by the extended reality system can include determining a prominence of the virtual content in a display of the extended reality system, with high prominence being associated with a high level of distraction. In some examples, determining the prominence of the virtual content within the display of the extended reality system may include determining at least one of: an audio volume of the virtual content; a size of the virtual content; a type of the virtual content; a brightness of the virtual content; a transparency of the virtual content; a level of movement of the virtual content; one or more color characteristics of the virtual content; a vibration intensity of the virtual content; a vibration frequency of the virtual content; a vibration duration of the virtual content; a field of view of the virtual content; a visual depth of the virtual content; and a position of the virtual content.
[0008] In some aspects, the one or more environmental factors include physical objects in a real-world environment. In some examples, the one or more environmental factors include an interaction between a user of the extended reality system and a person.
[0009] In some aspects, the threshold level of distraction may be determined based at least in part on determining a level of importance of one or more environmental factors, where a low threshold level of distraction is associated with a high level of importance of the one or more environmental factors. In some examples, an environmental factor with a high safety risk to the user may be associated with a high level of importance.
[0010] In some examples, determining whether a level of distraction of the user by the virtual content exceeds a distraction threshold level includes determining that a level of distraction of the user exceeds a distraction threshold level. In these examples, adjusting one or more characteristics of the virtual content can include outputting an indication of one or more environmental factors via a display of the extended reality system. Additionally or alternatively, adjusting one or more characteristics of the virtual content can include reducing a prominence of the virtual content within a display of the extended reality system.
[0011]
[0011] In some examples, determining whether a level of distraction of the user due to the virtual content exceeds a threshold level of distraction includes determining that the level of distraction of the user is less than the threshold level of distraction. In these examples, adjusting one or more characteristics of the virtual content can include increasing the prominence of the virtual content in a display of the extended reality system. In some examples, an updated level of distraction of the user of the extended reality system due to the virtual content with the adjusted one or more characteristics can be determined. These examples can include determining an updated threshold level of distraction based at least in part on one or more updated environmental factors related to the real-world environment, the one or more updated environmental factors being at least in part different from the one or more environmental factors. These examples can also include determining that an updated level of distraction of the user due to the virtual content with the adjusted one or more characteristics exceeds an updated threshold level of distraction, and further adjusting one or more characteristics of the virtual content based on determining that the updated level of distraction exceeds the updated threshold level of distraction. Further adjusting the one or more characteristics of the virtual content can decrease the salience of the virtual content provided by the extended reality system. These examples can also include determining that an updated level of distraction of the user by the virtual content with the adjusted one or more characteristics is less than an updated threshold level of distraction. These examples can also include further adjusting the one or more characteristics of the virtual content based on determining that the updated level of distraction is less than the updated threshold level of distraction, where further adjusting the one or more characteristics of the virtual content can increase the salience of the virtual content provided by the extended reality system.
[0012]
[0012] In some aspects, the methods, apparatus, and computer-readable media described herein may include, be a part of, and / or be implemented by an extended reality device (e.g., a virtual reality device, an augmented reality device, and / or a mixed reality device), a mobile device (e.g., a mobile phone or so-called "smartphone" or other mobile device), a wearable device, a personal computer, a laptop computer, a server computer, or other device. In some aspects, the apparatus includes one or more cameras for capturing one or more images. In some aspects, the apparatus includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the apparatus described above may include one or more sensors (e.g., one or more accelerometers, gyroscopes, inertial measurement units (IMUs), motion detection sensors, and / or other sensors).
[0013]
[0013] This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used independently to determine the scope of the claimed subject matter, which should be understood by reference to the entire specification of this patent, any or all drawings, and appropriate portions of each claim.
[0014]
[0014] The above, together with other features and examples, will become more apparent with reference to the following specification, claims, and accompanying drawings.
[0015]
[0015] Illustrative examples of the present application are described in detail below with reference to the following figures: [Brief description of the drawings]
[0016] [Figure 1]
[0016] A block diagram illustrating an exemplary architecture of an extended reality system, according to some examples. [Diagram 2]
[0017] 1 is a block diagram illustrating an example of a system for coordinating virtual content in an extended reality environment, according to some examples. [Diagram 3]
[0018] 1 is a diagram of a display of an extended reality system, according to some examples. [Figure 4A]
[0019] 1 illustrates the adjustment of virtual content in a display of an extended reality system, according to some examples. [Figure 4B] 1 illustrates the adjustment of virtual content in a display of an extended reality system, according to some examples. [Diagram 5]
[0020] 1 is a flow diagram illustrating an example of a process for coordinating virtual content in an extended reality environment, according to some examples. [Figure 6]
[0021] 1 is a graph illustrating exemplary thresholds for adjusting virtual content in an extended reality system, according to some examples. [Figure 7]
[0022] 1 is a flow diagram illustrating an example of a process for coordinating virtual content in an extended reality environment, according to some examples. [Figure 8]
[0023] FIG. 1 illustrates an example of a system for implementing certain aspects described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0017]
[0024] Some aspects and examples of the present disclosure are provided below. As will be apparent to those skilled in the art, some of these aspects and examples can be applied independently, and some of them can be applied in combination. In the following description, for the purpose of explanation, specific details are described to provide a thorough understanding of the subject matter of the present application. However, it will be apparent that various examples can be implemented without these specific details. The figures and descriptions are not limiting.
[0018]
[0025] The following description provides only illustrative examples and is not intended to limit the scope, applicability, or configuration of the present disclosure. Instead, the following description provides those skilled in the art with an enabling description for implementing the illustrative examples. It should be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the present application, as set forth in the appended claims.
[0019]
[0026] An extended reality (XR) device or system can facilitate interaction with different types of XR environments, including virtual reality (VR) environments, augmented reality (AR) environments, mixed reality (MR) environments, and / or other XR environments. The XR device or system can be used by a user to interact with the XR environment. Examples of XR devices or systems include head mounted displays (HMDs), XR glasses, among others. The XR device or system can overlay virtual content onto an image of a real-world environment, which can be viewed by the user through the XR device or system. Some XR devices or systems can provide accompanying audio content. The real-world environment can include physical objects, people, or other real-world objects. The XR device or system can track parts of the user (e.g., the user's hands and / or fingertips) to allow the user to interact with items of virtual content.
[0020]
[0027] In some cases, the immersive nature of an XR device or system may prevent a user from properly or safely interacting with the user's real-world environment, such as when moving around the environment. For example, the virtual content presented to the user may obscure or conceal safety hazards, such as heavy furniture, stairs, traffic, etc. Additionally, a user immersed in virtual content may have problems seeing or hearing other people, or may even be unaware of other people attempting to communicate with the user. These problems are exacerbated when the user is particularly distracted by the virtual content. While the goal of many XR systems is to create a realistic, interactive, and fully immersive XR environment, the XR system should also ensure that the virtual content does not create potentially dangerous situations for the user or otherwise prevent the user from properly interacting with the real-world environment.
[0021]
[0028] This disclosure describes systems, apparatus, methods, and computer-readable media (collectively "systems and techniques") for providing a context-aware XR environment. The systems and techniques described herein provide the ability for an XR system or device (e.g., HMD, AR glasses, etc.) to dynamically adjust virtual content provided to a user based on the user's level of distraction from the virtual content. For example, the XR system may adjust the virtual content to reduce the user's level of distraction. By adjusting the virtual content in such a manner, the XR system triggers or otherwise enables the user to focus on environmental factors related to the real-world environment. As used herein, the term "environmental factors" may refer to any element or aspect of a real-world environment that potentially requires or attracts the user's attention. In one example, the environmental factors may include attempts by one or more people to communicate with the user. Additionally or alternatively, environmental factors may include safety hazards or risks (e.g., cars, stairs, animals, furniture, physical obstacles, etc.) near the user, on the user's path or route, or otherwise in the vicinity of the user. In some cases, the XR system may monitor the real-world environment (e.g., using various types of sensors) to detect environmental factors that may be relevant to the user. The XR system may adjust the virtual content based on the detected environmental factors.
[0022]
[0029] In some cases, the XR system can determine the level of the user's distraction by the virtual content by determining a level of engagement between the user and the virtual content. A high level of engagement with the virtual content can indicate that the user is highly distracted by the virtual content and is therefore unlikely to be focusing on environmental factors in the real-world environment. In some examples, the XR system can determine the level of engagement between the user and the virtual content by identifying an object (e.g., a virtual object and / or a physical object) on which the user is focusing, detecting the user's movements associated with providing input to the XR system, monitoring the user's physiological responses, and / or in any other suitable manner. In some cases, the XR system can determine the level of the user's distraction by the virtual content based on the prominence of the virtual content. The prominence of the virtual content can be based on the size of the virtual content, the placement of the virtual content on the display of the XR system (e.g., virtual content in the center of the display may be more prominent than virtual content in the corners of the display), the audio level of the virtual content, the brightness of the virtual content, the amount of movement of the virtual content, any combination thereof, and / or other characteristics of the virtual content. For example, highly salient virtual content (e.g., virtual content that is loud, loud, animated, etc.) may be more distracting than less salient virtual content. In some examples, the XR system may determine a level of user distraction by virtual content based on both the level of engagement between the user and the virtual content and the salience of the virtual content. For example, the XR system may combine a quantitative measure of the level of engagement with a quantitative measure of content salience to determine an overall level of user distraction.
[0023]
[0030] The XR system can determine whether a level of distraction of the user by the virtual content is likely and / or expected to prevent the user from properly and / or safely handling environmental factors in the real-world environment. For example, the XR system can determine an environmental safety score (ESS) that represents a quantitative assessment of safety hazards (or other features requiring the user's attention) in the environment. A high ESS can indicate a relatively safe environment. Additionally, the XR system can determine a user distraction score (UDS) that represents a quantitative assessment of the level of distraction of the user by the virtual content. The XR system can compare the UDS to the ESS to determine whether to adjust the virtual content being presented by the XR system. For example, if the UDS exceeds the ESS (e.g., by at least a threshold amount), the XR system can reduce the salience of the virtual content to allow the user to focus on the environmental factors. In an illustrative example, if the XR system determines the ESS and UDS as values in the range of 1 to 10, the XR system may determine to reduce the prominence of the virtual content if the UDS exceeds the ESS by at least a value of 1 (or any other suitable threshold amount). This threshold amount may be referred to as a safety threshold because reducing the prominence of the virtual content may help mitigate safety risks presented to the user by environmental factors. In one example, the XR system may reduce the prominence of the virtual content by a weighted or proportional amount (e.g., relative to the difference between the ESS and the UDS). Examples of reducing the prominence of the virtual content may include lowering the volume of the audio content and / or reducing the size, positioning, brightness, movement, etc. of the visual content being displayed by the XR system.
[0024]
[0031] In some examples, the XR system can present one or more environmental factors to the user by outlining, highlighting, pointing, and / or otherwise emphasizing physical objects in a display of the XR system. For example, using an augmented reality (AR) system as an example of an XR system, the AR system can display virtual content on a display. The display of the AR system (e.g., the lenses of the AR glasses) can enable the user to also view the real-world environment. Using the AR system, the user can view virtual content overlaid on the real-world environment. In such examples, the AR system can display outlines of physical objects of the virtual environment being viewed through the display of the AR system, and can highlight, point, and / or otherwise emphasizing the physical objects.
[0025]
[0032] In some cases, the XR system may determine that the level of distraction of the user by the virtual content is unlikely and / or is not expected to prevent the user from properly and / or safely handling the environmental factors. For example, the XR system may determine that the UDS does not exceed the ESS (e.g., by at least a threshold amount). In some cases, the XR system may determine that adjustment of the virtual content is unnecessary if the UDS does not exceed the ESS (e.g., by at least a threshold amount). In other cases, the XR system may adjust the virtual content by increasing the prominence of the virtual content. For example, the XR system may determine that the user is capable of experiencing a greater amount of the virtual content while properly handling the environmental factors. In an illustrative example, if the XR system determined the ESS and UDS as values in the range of 1 to 10, the XR system may determine to increase the prominence of the virtual content if the UDS falls below the ESS by at least a value of 1 (or any other suitable threshold amount). This threshold amount may be referred to as a usability threshold, as increasing the prominence of the virtual content may increase the usability of the XR system and / or improve the user's experience with the XR system. In one example, the XR system may increase the prominence of the virtual content by a weighted or proportional amount (e.g., relative to the difference between the ESS and the UDS). In some cases, increasing the prominence of the virtual content may increase the usability of the XR system and / or improve the user's experience with the XR system. Examples of increasing the prominence of the virtual content may include increasing the volume of audio content and / or increasing the size, positioning, brightness, movement, etc. of visual content.
[0026]
[0033] In some cases, a context-aware XR system may continuously or periodically adjust the virtual content based on changes in environmental factors in the real-world environment and / or based on changes in the user's level of distraction due to the virtual content. For example, the XR system may include and / or correspond to a closed-loop system, where the salience of the virtual content may represent feedback used to determine the user's level of distraction. In a closed-loop system, changes in the salience of the virtual content may represent a system output, and a new state of the salience of the virtual content may be fed back to the system (e.g., to account for changes in the user's level of distraction due to changes in content salience).
[0027]
[0034] The disclosed context-aware XR systems and techniques can enable users to become fully immersed in an XR environment without the risk of negative or harmful interactions with objects or elements in the real-world environment. Furthermore, by tailoring virtual content based on the surrounding environment, the systems and techniques can expand the types of environments in which users can enjoy XR applications.
[0028]
[0035] Further details regarding the context-aware XR system are provided herein with respect to various figures. FIG. 1 illustrates an exemplary extended reality system 100 according to some aspects of the disclosure. The extended reality system 100 can run (or execute) XR applications and implement XR operations. In some examples, the extended reality system 100 can perform tracking and localization, mapping of the physical world (e.g., a scene), and placing and rendering of virtual content on a display (e.g., a screen, a visible plane / area, and / or other display) as part of an XR experience. For example, the extended reality system 100 can generate a map (e.g., a 3D map) of a scene in the physical world, track the pose (e.g., location and position) of the extended reality system 100 relative to the scene (e.g., relative to the 3D map of the scene), place and / or anchor virtual content at a particular location(s) on the map of the scene, and render the virtual content on the display. The extended reality system 100 can render virtual content on the display such that the virtual content appears to be at a location in the scene that corresponds to a particular location on a map of the scene where the virtual content is placed and / or anchored. In some examples, the display can include glasses, a screen, lenses, and / or other display mechanisms that allow a user to view the real-world environment and also allow XR content to be displayed thereon.
[0029]
[0036] As shown in FIG. 1, the extended reality system 100 may include one or more image sensors 102, accelerometers 104, gyroscopes 106, storage 108, computational components 110, an XR engine 120, a content adjustment engine 122, an image processing engine 124, and a rendering engine 126. It should be noted that the components 102-126 shown in FIG. 1 are non-limiting examples provided for purposes of illustration and description, and other examples may include more, fewer, or different components than those shown in FIG. 1. For example, in some cases, the extended reality system 100 may include one or more other sensors (e.g., one or more inertial measurement units (IMUs), radar, light detection and ranging (LIDAR) sensors, audio sensors, etc.), one or more display devices, one or more other processing engines, one or more other hardware components, and / or one or more other software and / or hardware components not shown in FIG. 1. An example architecture and example hardware components that may be implemented by the extended reality system 100 are further described below with respect to FIG.
[0030]
[0037] For simplicity and purposes of explanation, one or more image sensors 102 will be referred to herein (e.g., in the singular) as image sensor 102. However, one of ordinary skill in the art will recognize that extended reality system 100 can include a single image sensor or multiple image sensors. Additionally, reference to any of the components (e.g., 102-126) of extended reality system 100 in the singular or plural should not be construed as limiting the number of such components implemented by extended reality system 100 to one, or two or more. For example, reference to accelerometer 104 in the singular should not be construed as limiting the number of accelerometers implemented by extended reality system 100 to one. One of ordinary skill in the art will recognize that for any of components 102-126 shown in FIG. 1, extended reality system 100 can include only one of such component(s) or two or more of such component(s).
[0031]
[0038] The extended reality system 100 may include or be in communication (wired or wireless) with input devices. The input devices may include any suitable input devices, such as a touch screen, a pen or other pointer device, a keyboard, a mouse, buttons or keys, a microphone for receiving voice commands, a gesture input device for receiving gesture commands, any combination thereof, and / or other input devices. In some cases, the image sensor 102 may capture images that may be processed to interpret the gesture commands.
[0032]
[0039] The extended reality system 100 may be part of or implemented by a single computing device or multiple computing devices. In some examples, the extended reality system 100 may be part of an electronic device(s), such as an extended reality head mounted display (HMD) device, extended reality glasses (e.g., augmented reality or AR glasses), a camera system (e.g., digital camera, IP camera, video camera, security camera, etc.), a telephone system (e.g., smartphone, cellular phone, conferencing system, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a smart television, a display device, a gaming console, a video streaming device, an Internet of Things (IoT) device, and / or any other suitable electronic device(s).
[0033]
[0040] In some implementations, one or more of the image sensor 102, the accelerometer 104, the gyroscope 106, the storage 108, the computational component 110, the XR engine 120, the content adjustment engine 122, the image processing engine 124, and the rendering engine 126 may be part of the same computing device. For example, in some cases, one or more of the image sensor 102, the accelerometer 104, the gyroscope 106, the storage 108, the computational component 110, the XR engine 120, the content adjustment engine 122, the image processing engine 124, and the rendering engine 126 may be integrated into an HMD, extended reality glasses, a smartphone, a laptop, a tablet computer, a gaming system, and / or any other computing device. However, in some implementations, one or more of the image sensor 102, accelerometer 104, gyroscope 106, storage 108, computational components 110, XR engine 120, content adjustment engine 122, image processing engine 124, and rendering engine 126 may be part of two or more separate computing devices. For example, in some cases, some of the components 102-126 may be part of or implemented by one computing device, and the remaining components may be part of or implemented by one or more other computing devices.
[0034]
[0041] The storage 108 may be any storage device(s) for storing data. Moreover, the storage 108 may store data from any of the components of the extended reality system 100. For example, the storage 108 may store data from the image sensor 102 (e.g., image or video data), data from the accelerometer 104 (e.g., measurements), data from the gyroscope 106 (e.g., measurements), data from the computational component 110 (e.g., processing parameters, preferences, virtual content, rendering content, scene maps, tracking and localization data, object detection data, privacy data, XR application data, face recognition data, occlusion data, etc.), data from the XR engine 120, data from the content adjustment engine 122, data from the image processing engine 124, and / or data from the rendering engine 126 (e.g., output frames). In some examples, the storage 108 may include a buffer for storing frames for processing by the computational component 110.
[0035]
[0042] The one or more computing components 110 may include a central processing unit (CPU) 112, a graphics processing unit (GPU) 114, a digital signal processor (DSP) 116, and / or an image signal processor (ISP) 118. The computing components 110 may perform various operations, such as image enhancement, computer vision, graphics rendering, extended reality (e.g., tracking, localization, pose estimation, mapping, content anchoring, content rendering, etc.), image / video processing, sensor processing, recognition (e.g., text recognition, face recognition, object recognition, feature recognition, tracking or pattern recognition, scene recognition, occlusion detection, etc.), machine learning, filtering, and any of the various operations described herein. In this example, the computing components 110 implement an XR engine 120, a content adjustment engine 122, an image processing engine 124, and a rendering engine 126. In other examples, the computing components 110 may also implement one or more other processing engines.
[0036]
[0043] The image sensor 102 can include any image and / or video sensor or capture device. In some examples, the image sensor 102 can be part of a multiple camera assembly, such as a dual camera assembly. The image sensor 102 can capture image and / or video content (e.g., raw image and / or video data), which can then be processed by the computing component 110, the XR engine 120, the content adjustment engine 122, the image processing engine 124, and / or the rendering engine 126 described herein.
[0037]
[0044] In some examples, the image sensor 102 can capture image data, generate frames based on the image data, and / or provide image data or frames to the XR engine 120, the content adjustment engine 122, the image processing engine 124, and / or the rendering engine 126 for processing. A frame can include a video frame or a still image of a video sequence. A frame can include an array of pixels representing a scene. For example, a frame can be a Red-Green-Blue (RGB) frame having red, green, and blue color components per pixel, a Luma, Chroma Red, Chroma Blue (YCbCr) frame having a luma component and two chroma (color) components (chroma red and chroma blue) per pixel, or any other suitable type of color or monochrome picture.
[0038]
[0045] In some cases, the image sensor 102 (and / or other image sensors or cameras of the extended reality system 100) may be configured to capture depth information as well. For example, in some implementations, the image sensor 102 (and / or other cameras) may include an RGB-depth (RGB-D) camera. In some cases, the extended reality system 100 may include one or more depth sensors (not shown) that are separate from the image sensor 102 (and / or other cameras) and can capture depth information. For example, such depth sensors may obtain depth information independent of the image sensor 102. In some examples, the depth sensor may be physically located in the same general location as the image sensor 102 but may operate at a different frequency or frame rate than the image sensor 102. In some examples, the depth sensor may take the form of a light source that can project a structured or textured light pattern, which may include one or more narrow bands of light, onto one or more objects in a scene. The depth information may then be obtained by exploiting geometric distortions of the projected pattern caused by the surface shape of the object. In one example, depth information may be obtained from a stereo sensor, such as a combination of an infrared structured light projector and an infrared camera registered to a camera (eg, an RGB camera).
[0039]
[0046] As mentioned above, in some cases, the extended reality system 100 may also include one or more sensors (not shown) other than the image sensor 102. For example, the one or more sensors may include one or more accelerometers (e.g., accelerometer 104), one or more gyroscopes (e.g., gyroscope 106), and / or other sensors. The one or more sensors may provide velocity, orientation, and / or other position-related information to the computational component 110. For example, the accelerometer 104 may detect acceleration by the extended reality system 100 and generate acceleration measurements based on the detected acceleration. In some cases, the accelerometer 104 may provide one or more translational vectors (e.g., up / down, left / right, forward / backward) that may be used to determine a position or attitude of the extended reality system 100. The gyroscope 106 may detect and measure the orientation and angular velocity of the extended reality system 100. For example, the gyroscope 106 may be used to measure the pitch, roll, and yaw of the extended reality system 100. In some cases, the gyroscope 106 can provide one or more rotation vectors (e.g., pitch, yaw, roll). In some examples, the image sensor 102 and / or the XR engine 120 can use measurements obtained by the accelerometer 104 (e.g., one or more translation vectors) and / or the gyroscope 106 (e.g., one or more rotation vectors) to calculate the attitude of the extended reality system 100. As mentioned above, in other examples, the extended reality system 100 can also include other sensors, such as an inertial measurement unit (IMU), magnetometer, machine vision sensor, smart scene sensor, voice recognition sensor, impact sensor, shock sensor, position sensor, tilt sensor, etc.
[0040]
[0047] In some cases, the one or more sensors may include at least one IMU. An IMU is an electronic device that measures specific forces, angular velocities, and / or orientations of the extended reality system 100 using a combination of one or more accelerometers, one or more gyroscopes, and / or one or more magnetometers. In some examples, the one or more sensors may output measured information related to the capture of images captured by the image sensor 102 (and / or other cameras of the extended reality system 100) and / or depth information obtained using one or more depth sensors of the extended reality system 100.
[0041]
[0048] The output of one or more sensors (e.g., accelerometer 104, gyroscope 106, one or more IMUs, and / or other sensors) may be used by extended reality engine 120 to determine the pose of extended reality system 100 (also referred to as head pose) and / or the pose of image sensor 102 (or other camera of extended reality system 100). In some cases, the pose of extended reality system 100 and the pose of image sensor 102 (or other camera) may be the same. The pose of image sensor 102 refers to the position and orientation of image sensor 102 relative to a reference frame. In some implementations, the camera pose may be determined for six degrees of freedom (6DOF), which refers to three translational components (which may be given by X (horizontal), Y (vertical), and Z (depth) coordinates relative to a reference frame, such as an image plane, for example) and three angular components (e.g., roll, pitch, and yaw relative to the same reference frame).
[0042]
[0049] In some cases, a device tracker (not shown) can use measurements from one or more sensors and image data from the image sensor 102 to track the pose (e.g., 6DOF pose) of the extended reality system 100. For example, the device tracker can fuse visual data from the captured image data (e.g., using a visual tracking solution) with inertial measurement data to determine the position and movement of the extended reality system 100 relative to the physical world (e.g., scene) and a map of the physical world. As described below, in some examples, when tracking the pose of the extended reality system 100, the device tracker can generate a three-dimensional (3D) map of the scene (e.g., real world) and / or generate updates for the 3D map of the scene. The 3D map updates can include, for example, but are not limited to, new or updated features and / or features or landmark points associated with the scene and / or the 3D map of the scene, localization updates that identify or update the position of the extended reality system 100 within the scene and the 3D map of the scene, and / or the like. The 3D map can provide a digital representation of a scene in the real / physical world. In some examples, the 3D map can anchor location-based objects and / or content to real-world coordinates and / or objects. The extended reality system 100 can use the mapped scene (e.g., a scene in the physical world represented and / or associated with the 3D map) to merge the physical and virtual worlds and / or merge virtual content or objects with the physical environment.
[0043]
[0050] In some aspects, the pose of the image sensor 102 and / or the extended reality system 100 as a whole may be determined and / or tracked by the computational component 110 using a visual tracking solution based on images captured by the image sensor 102 (and / or other cameras of the extended reality system 100). For example, in some examples, the computational component 110 may perform tracking using computer vision-based tracking techniques, model-based tracking techniques, and / or simultaneous localization and mapping (SLAM) techniques. For example, the computational component 110 may perform SLAM or may be in communication (wired or wireless) with a SLAM engine (not shown). SLAM refers to a class of techniques in which a map of an environment (e.g., a map of the environment being modeled by the extended reality system 100) is created and simultaneously the pose of the camera (e.g., the image sensor 102) and / or the extended reality system 100 relative to the map is tracked. The map may be referred to as a SLAM map and may be 3D. SLAM techniques may be implemented using color or grayscale image data captured by image sensor 102 (and / or other cameras of extended reality system 100) and used to generate estimates of 6DOF attitude measurements of image sensor 102 and / or extended reality system 100. Such SLAM techniques configured to implement 6DOF tracking may be referred to as 6DOF SLAM. In some cases, the output of one or more sensors (e.g., accelerometer 104, gyroscope 106, one or more IMUs, and / or other sensors) may be used to estimate, correct, and / or otherwise adjust the estimated attitude.
[0044]
[0051] In some cases, 6DOF SLAM (e.g., 6DOF tracking) can associate observed features from several input images from the image sensor 102 (and / or other cameras) to a SLAM map. For example, 6DOF SLAM can use feature point associations from the input images to determine the pose (position and orientation) of the image sensor 102 and / or the extended reality system 100 with respect to the input images. 6DOF mapping can also be performed to update the SLAM map. In some cases, a SLAM map maintained using 6DOF SLAM can include 3D feature points triangulated from two or more images. For example, key frames can be selected from the input images or video stream to represent the observed scene. For each key frame, a respective 6DOF camera pose associated with the image can be determined. The pose of the image sensor 102 and / or the extended reality system 100 can be determined by projecting features from the 3D SLAM map onto the image or video frame and updating the camera pose from the verified 2D-3D correspondence.
[0045]
[0052] In one illustrative example, the computational component 110 can extract feature points from every input image or from each key frame. A feature point (also referred to as a registration point) as used herein is a distinctive or identifiable part of an image, such as a part of a hand, an edge of a table, among others. The features extracted from the captured image can represent distinct feature points along a three-dimensional space (e.g., coordinates on the X, Y, and Z axes), and every feature point can have an associated feature location. A feature point in a key frame can either match (be the same as or correspond to) a feature point of a previously captured input image or key frame, or cannot match it. Feature detection can be used to detect feature points. Feature detection can include image processing operations used to examine one or more pixels of an image to determine if a feature is present at a particular pixel. Feature detection can be used to process the entire captured image or some portion of the image. For each image or key frame, once a feature is detected, a local image patch around the feature can be extracted. Features may be extracted using any suitable technique, such as Scale Invariant Feature Transform (SIFT) (which locates features and generates descriptions of them), Speed Up Robust Features (SURF), Gradient Location-Orientation histogram (GLOH), Normalized Cross Correlation (NCC), or other suitable techniques.
[0046]
[0053] In some cases, the extended reality system 100 may also track the user's hands and / or fingers to enable the user to interact with and / or control virtual content in the virtual environment. For example, the extended reality system 100 may track the pose and / or movement of the user's hands and / or fingertips to identify or translate user interactions with the virtual environment. User interactions may include, for example, without limitation, moving items of virtual content, resizing items of virtual content and / or locations of virtual private spaces, selecting input interface elements in a virtual user interface (e.g., a virtual representation of a mobile phone, a virtual keyboard, and / or other virtual interfaces), providing input through a virtual user interface, etc.
[0047]
[0054] Operations for the XR engine 120, the content adjustment engine 122, the image processing engine 124, and the rendering engine 126 (and any image processing engines) may be implemented by any of the computing components 110. In one illustrative example, the operations of the rendering engine 126 may be implemented by the GPU 114, and the operations of the XR engine 120, the content adjustment engine 122, and the image processing engine 124 may be implemented by the CPU 112, the DSP 116, and / or the ISP 118. In some cases, the computing components 110 may include other electronic circuitry or hardware, computer software, firmware, or any combination thereof to perform any of the various operations described herein.
[0048]
[0055] In some examples, the XR engine 120 can perform XR operations to generate an XR experience based on data from the image sensor 102, accelerometer 104, gyroscope 106, and / or one or more sensors on the extended reality system 100, such as one or more IMUs, radars, etc. In some examples, the XR engine 120 can perform tracking, localization, pose estimation, mapping, content anchoring operations, and / or any other XR operations / functions. The XR experience can include the use of the extended reality system 100 to present XR content (e.g., virtual reality content, augmented reality content, mixed reality content, etc.) to a user during a virtual session. In some examples, the XR content and experiences can be provided by the extended reality system 100 through an XR application (e.g., executed or implemented by the XR engine 120) that provides a particular XR experience, such as, for example, an XR gaming experience, an XR classroom experience, an XR shopping experience, an XR entertainment experience, an XR activity (e.g., a motion, a troubleshooting activity, etc.), among others. During an XR experience, a user can view and / or interact with virtual content using extended reality system 100. In some cases, a user can view and / or interact with virtual content while also being able to view and / or interact with the physical environment around the user, which allows the user to have an immersive experience between the physical environment and the virtual content mixed or integrated with the physical environment.
[0049]
[0056] The content adjustment engine 122 can perform various operations to determine adjustments to the virtual content (or other types of XR content) provided by the extended reality system 100. As used herein, adjustments to the virtual content can include modifications, alterations, and / or deviations of the virtual content as configured by an XR application. For example, the XR application can generate and / or configure virtual content to be displayed to a user while operating within the extended reality system 100. The content adjustment engine 122 can determine one or more adjustments to be applied to the generated and / or configured virtual content. The image processing engine 124 and / or the rendering engine 126 can then apply the adjustments to the virtual content such that the virtual content experienced by the user reflects the adjustments. As described below, the content adjustment engine 122 can determine the adjustments to the virtual content based on current contextual factors associated with the user's XR experience. The contextual factors can include environmental factors associated with the real-world environment, the salience of the virtual content (e.g., before the virtual content is adjusted), and / or the level of engagement between the user and the virtual content.
[0050]
[0057] 2 is a block diagram illustrating an example of a content adjustment system 200. In some cases, the content adjustment system 200 may include and / or be a part of the extended reality system 100 in FIG. 1. For example, the content adjustment system 200 may correspond to all or a portion of the content adjustment engine 122. As shown in FIG. 2, the content adjustment system 200 may include one or more engines, including an environmental factors engine 202, a user engagement engine 204, a content salience engine 206, and a content adjustment engine 214. The engines of the content adjustment system 200 may be configured to determine a content adjustment 224. The content adjustment 224 may include any type or number of adjustments of virtual content (e.g., virtual content 212) that may be displayed on a display of the extended reality system 100 and viewed by a user of the extended reality system 100.
[0051]
[0058] In some cases, the content adjustment system 200 can determine the content adjustment 224 based at least in part on one or more environmental factors (e.g., environmental factors 208) associated with the real-world environment. As described above, the environmental factors can include any element or aspect of the real-world environment that potentially requires and / or involves the user's attention. In some examples, the environmental factors 208 can include safety hazards, such as a flight of stairs, traffic, physical objects in the user's path (e.g., street signs, a piece of furniture, etc.), among others. In other examples, the environmental factors 208 can include the complexity of the environment. For example, the complexity of the environment can be based on an overall assessment of the level of clutter, movement, noise, etc. in the environment. In some cases, a complex environment can cause the user to direct a greater amount of attention to the virtual content in order to process the virtual content (e.g., to separate the virtual content from the complex environment). In some examples, the environmental factors 208 can include potential interactions with one or more people. For example, the environmental factors 208 can include attempts by a person to communicate with the user. Additionally or alternatively, environmental factors 208 can include surrounding and / or geographic characteristics of the real-world environment. For example, environmental factors 208 can include the temperature of the environment, the temperature of one or more physical objects in the environment, the level of lighting in the environment, the type of environment (e.g., whether the environment is a public space, whether it is inside a building, etc.), and / or other surrounding and / or geographic characteristics of the real-world environment. In one example, a high level of environmental lighting can reduce the visibility of the virtual content and thus cause the user to direct a greater amount of attention to the virtual content. In another example, a hot object (e.g., a fireplace or range top) can represent a safety hazard to the user. In some cases, environmental factors 208 can include the location of the environment (e.g., a particular street, city, or other geographic area associated with the environment). In some cases, environmental factors 208 can include historical and / or previously identified environmental factors associated with the environment.For example, environmental factors 208 may include trends in traffic, crime, injuries, etc., within the environment, and / or documented traffic volumes, crime rates, injury rates, etc.
[0052]
[0059] The environmental factor engine 202 may determine the environmental factors 208 using various techniques and / or devices. In some examples, the environmental factor engine 202 may analyze image or video data captured by one or more cameras or image sensors (e.g., the image sensor 102 of the extended reality system 100) to determine the environmental factors 208. For example, the environmental factor engine 202 may use various image processing techniques to detect and / or identify physical objects, safety hazards, people, and the like. Such safety hazards may include stairs, parked vehicles, furniture, pets, crowded streets, among other safety hazards. Additionally, in some cases, the environmental factor engine 202 may use various image processing techniques to detect visual markers (e.g., quick response (QR) codes) placed in the environment to indicate safety hazards. The image processing techniques may include object detection algorithms, object recognition algorithms, face recognition algorithms, any combination thereof, and / or other image processing techniques. In some examples, the environmental factor engine 202 may determine the complexity of the environment using spatial frequency analysis or similar analysis. In some examples, the environmental factors engine 202 may determine the ambient and / or physical conditions of the environment using sensors, such as LIDAR sensors, audio sensors, thermometers, any combination thereof, and / or other sensors or devices. In further examples, the environmental factors engine 202 may determine the location of the real-world environment in which the extended reality system 100 is located using a network connection and / or a positioning system (e.g., Global Positioning System (GPS)). The environmental factors engine 202 may detect any additional or alternative types of environmental factors using any suitable techniques and / or devices.
[0053]
[0060] As shown in FIG. 2, the environmental factor engine 202 can determine an environmental safety score (ESS) 216 based at least in part on the environmental factors 208. The ESS 216 can represent a quantitative assessment of the risk presented to the user due to safety hazards (or other factors requiring user attention) in the environment. For example, the value of the ESS 216 can correspond to the amount of attention that a combination of the environmental factors 208 may require from the user (e.g., for the user to safely and / or properly handle or address the environmental factors 208). In some cases, a high ESS 216 can indicate that a low level of user attention is required to address objects or other items in the environment), and a low ESS 216 can indicate a high level of user attention. For example, an environmental factor associated with a high safety risk may be associated with a low environmental safety score. In some cases, an environmental factor that is expected to require a large amount of attention from the user may be referred to as a "critical" environmental factor, and an environmental factor that is not expected to require a large amount of attention from the user may be referred to as a "non-critical" environmental factor.
[0054]
[0061] In some examples, the environmental factor engine 202 can determine the ESS 216 by weighting individual environmental factors in the environmental factors 208. For example, the environmental factor engine 202 can assign a numerical score to each environmental factor determined to be present and / or relevant in the real-world environment in which the XR system is located. In a non-limiting example, the environmental factor engine 202 can assign a score to each environmental factor within a predetermined range (e.g., a range of 1 to 10, where a score of 1 corresponds to the lowest level of environmental safety and a score of 10 corresponds to the highest level of environmental safety). The numerical value of the score assigned to the environmental factor can correspond to the relative importance of the environmental factor. For example, an environmental score with a high importance can be assigned a low score (corresponding to a high level of user caution). In some cases, the environmental factor engine 202 may combine the individual scores of each of the environmental factors 208 (e.g., by determining an average or mean or other representative value) to generate an ESS 216 that represents a cumulative assessment of the environmental factors 208. In a non-limiting example, the environmental factor engine 202 may identify and assign scores to four environmental factors as follows: (1) a chair located 3 feet to the right of the user, assigned a score of 7; (2) a flight of stairs descending 3 feet in front of the user, assigned a score of 2; (3) a television providing both audio and visual content in the same room as the user, assigned a score of 4; and (4) a high level of ambient light in the environment, assigned a score of 5. In this example, the environmental factor engine 202 may determine the ESS 216 by averaging the individual scores, resulting in an ESS 216 of 4.5.
[0055]
[0062] In some cases, the content adjustment system 200 can determine the content adjustment 224 based at least in part on one or more characteristics of the user (e.g., user characteristics 210). The user characteristics 210 can include any number or type of behaviors, reactions, physiological states, and / or other characteristics of the user associated with a level of engagement between the user and the virtual content 212. In some cases, a high level of engagement between the user and the virtual content 212 can correspond to the user being highly distracted from the real-world environment by the virtual content 212. Thus, a high level of engagement between the user and the virtual content 212 can be associated with an increased risk that the user experiences an unsafe or otherwise undesirable interaction with the real-world environment. For example, a user who is highly distracted by the virtual content may be unable to notice safety hazards (e.g., physical objects in the user's path, such as a street, stairs, etc.) and / or may be unable to notice that a person is attempting to communicate with the user.
[0056]
[0063] In one example, the user characteristics 210 can include physical behaviors indicative of one or more objects the user is viewing and / or tracking. For example, the user engagement engine 204 can track the direction of the user's gaze (e.g., using one or more eye tracking techniques) to determine whether the one or more objects include virtual content (indicating a high level of engagement with the virtual content) or physical objects (indicating a lower level of engagement with the virtual content). In another example, the user characteristics 210 can include movements associated with providing input to the XR application. For example, the user engagement engine 204 can detect when the user performs a particular gesture (e.g., a hand gesture) that corresponds to an input to the XR application and / or when the user interacts with a controller of the XR application. Frequent movements associated with providing input to the XR application can indicate a high level of engagement with the virtual content 212. In a further example, the user characteristics 210 can include the user's voice. For example, the user engagement engine 204 can determine whether a voice by the user is related to the virtual content 212 (e.g., interacting with other users of the XR application, giving instructions to the XR application, etc.) or related to the real-world environment (e.g., voice directed to people not using the XR application). Voices related to the virtual content 212 may be associated with a higher level of engagement with the virtual content 212. In another example, the user characteristics 210 can include the user's movements. For example, the user engagement engine 204 can detect when the user is sitting, walking, running, etc. In some cases, fast and / or varied movements (e.g., movements in more than one direction and / or speed) can indicate that the user is highly engaged with the virtual content 212. Furthermore, fast and / or varied movements can be associated with an increased risk of injury to the user due to a safety hazard in the real-world environment. In some cases, the user characteristics 210 can include the user's physiological state and / or physiological response.For example, the user engagement engine 204 may monitor the user's heart rate, breathing rate, and / or any additional biological metrics associated with the user. In some cases, a biological metric with an elevated value (e.g., compared to a baseline or resting value) may be associated with a high level of engagement with the virtual content 212.
[0057]
[0064] The user engagement engine 204 can determine the user characteristics 210 using various techniques and / or devices. In some examples, the user engagement engine 204 can utilize eye tracking techniques and / or focal plane detection sensors and / or algorithms to track the user's gaze. Such sensors and / or algorithms can determine whether the user is visually focused on (e.g., viewing and / or tracking) the virtual or real-world content. In further examples, the user engagement engine 204 can analyze image or video data captured by one or more cameras or image sensors (e.g., the image sensor 102 of the extended reality system 100) to detect the user's hand gestures or other input gestures. In other examples, the user engagement engine 204 can utilize one or more motion detection sensors (e.g., an IMU, an accelerometer, a gyroscope, etc.) to detect the user's movements. The motion detection sensors can enable the user engagement engine 204 to determine the speed, direction, and / or type of the user's movements. Additionally or alternatively, the user engagement engine 204 may utilize physiological sensors (e.g., galvanic skin response (GSR) sensors, electroencephalogram (EEG) devices, etc.) to monitor the user's physiological state and / or response. The user engagement engine 204 may detect any additional or alternative types of user characteristics using any suitable techniques and / or devices.
[0058]
[0065] 2, the user engagement engine 204 can determine a user engagement score (UES) 218 based at least in part on the user characteristics 210. The UES 218 can represent a quantitative assessment of an overall level of engagement between the user and the virtual content 212. The level of engagement between the user and the virtual content 212 can correspond to and / or be indicative of the amount of attention the user directs toward the virtual content 212 (e.g., as compared to the amount of attention the user directs toward a real-world environment). In some cases, a high UES 218 can indicate a high level of engagement with the virtual content and a low UES 218 can indicate a low level of engagement with the virtual content.
[0059]
[0066] In some cases, the user engagement engine 204 can determine the UES 218 by weighting the individual user characteristics in the user characteristics 210. For example, the user engagement engine 204 can assign a numerical score to each user characteristic detected and / or determined to be relevant (e.g., currently displayed by the user). In a non-limiting example, the user engagement engine 204 can assign each user characteristic a score within a predetermined range (e.g., a range of 1 to 10, where a score of 1 corresponds to the lowest level of user engagement and a score of 10 corresponds to the highest level of user engagement). In some cases, the user engagement engine 204 can combine the individual scores of each of the user characteristics 210 (e.g., by determining an average or mean or other representative value) to generate the UES 218, which represents a cumulative assessment of the user characteristics 210. In some cases, the predetermined range used to determine the UES 218 can be the same as the predetermined range used by the environmental factors engine 202 to determine the ESS 216. In this manner, the ESS 216 and the UES 218 may be normalized (e.g., directly compared). In some cases, the predetermined range used to determine the UES 218 may differ from the predetermined range used by the environmental factors engine 202 to determine the ESS 216. In a non-limiting example, the user engagement engine 204 may identify and assign scores to three user characteristics 210 as follows: (1) the user is running in a straight line, assigned a score of 8; (2) the user is providing verbal commands to the XR application, assigned a score of 7; and (3) the user is visually tracking a virtual object, assigned a score of 6. In this example, the user engagement engine 204 may determine the UES 218 by averaging the individual scores, resulting in a UES of 7.
[0060]
[0067] 2, the content prominence engine 206 can determine a content prominence score (CPS) 220 based at least in part on the virtual content 212. The virtual content 212 can include any virtual content (e.g., audio virtual content, visual virtual content, and / or haptic virtual content) that is displayed to a user of the extended reality system 100. The CPS 220 can represent a quantitative assessment of the amount and / or type of virtual content 212 currently being displayed to the user. In some examples, a high CPS 220 can correspond to a high prominence of the virtual content 212 and a low CPS 220 can correspond to a low prominence of the virtual content 212.
[0061]
[0068] The content salience engine 206 can determine the CPS 220 based on various characteristics of the virtual content 212. In some examples, the content salience engine 206 can determine a content salience score for each type (e.g., modality) of virtual content in the virtual content 212. For example, the content salience engine 206 can determine a first content salience score associated with audio virtual content, a second content salience score associated with visual virtual content, and a third content salience score associated with haptic virtual content. In some examples, the content salience engine 206 can determine that audio virtual content with speech has greater salience than other audio virtual content (e.g., sound effects, music, etc.). In other examples, the content salience engine 206 can determine that audio virtual content with higher volume of audio and / or higher complexity has greater salience than audio virtual content with lower volume of audio and / or lower complexity.
[0062]
[0069] In a further example, the content saliency engine 206 can determine that visual virtual content with higher brightness, color saturation and other color characteristics, visual complexity, size, transparency (and / or opacity), etc., has a higher saliency than visual virtual content with lower levels of such characteristics. For example, the content saliency engine 206 can determine that visual virtual content with lower transparency has a higher saliency than visual virtual content with higher transparency. In a further example, the content saliency engine 206 can determine that moving virtual content (e.g., animated virtual content) has a higher saliency than stationary virtual content (e.g., especially when in the periphery of the user's field of view). In an additional example, the content saliency engine 206 can determine that visual virtual content with a shorter depth (e.g., virtual content that appears closer to the user) has a higher saliency than visual virtual content with a longer depth (e.g., virtual content that appears farther away from the user). In a further example, the content saliency engine 206 can determine that visual virtual content that occupies a larger amount of the field of view (FOV) of the display has a higher saliency than visual virtual content that occupies a smaller amount of the FOV. In another example, the content saliency engine 206 can determine that visual virtual content with a higher complexity (e.g., variance) has a higher saliency than visual virtual content with a lower complexity. In another example, the content saliency engine 206 can determine that visual virtual content at some locations in the display (e.g., locations within the user's primary line of sight) has a higher saliency than visual virtual content at other locations.
[0063]
[0070] In some cases, the haptic virtual content may include vibrations induced in a handheld or wearable XR device (e.g., an XR controller, an XR watch, etc.) In one example, the content salience engine 206 may determine that haptic virtual content with a higher vibration intensity, a higher vibration duration, and / or a higher vibration frequency has a higher salience than haptic virtual content with a lower vibration intensity, a lower vibration duration, and / or a lower vibration intensity, respectively.
[0064]
[0071] In some cases, the content salience engine 206 may determine the CPS 220 by assigning numerical scores to individual characteristics associated with the virtual content 212. In a non-limiting example, the content salience engine 206 may assign each characteristic a score within a predetermined range (e.g., a range of 1 to 10, where a score of 1 corresponds to the lowest level of salience and a score of 10 corresponds to the highest level of salience). In some cases, the content salience engine 206 may combine the individual scores of each characteristic (e.g., by determining an average or mean or other representative value) to generate a CPS 220 that represents a cumulative assessment of the salience of the virtual content 212. In some cases, the predetermined range used to determine the CPS 220 may be the same as the predetermined range used by the user engagement engine 204 to determine the UES 218 and / or the predetermined range used by the environmental factors engine 202 to determine the ESS 216. In this manner, the CPS 220, the ESS 216, and / or the UES 218 may be normalized (e.g., directly compared). In some cases, the predetermined range used to determine the CPS 220 may differ from the predetermined range used to determine the UES 218 and / or the predetermined range used to determine the ESS 216. In some examples, if the content salience engine 206 determines individual content salience scores for multiple types of virtual content, the content salience engine 206 may determine the CPS 220 by averaging the individual content salience scores. In a non-limiting example, the average of the scores may be a weighted average (e.g., a weight of 0.3 for the visual content salience score, a weight of 0.5 for the audio content salience score, and a weight of .2 for the haptic content salience score).
[0065]
[0072] As shown in FIG. 2, the content adjustment system 200 can determine a user distraction score (UDS) 222. The UDS 222 can represent a quantitative assessment of an overall level of the user's distraction by the virtual content 212. In some cases, a high UDS 222 can indicate a high level of user distraction and a low UDS 222 can indicate a low level of user distraction. The content adjustment system 200 can determine the UDS 222 in various ways. In one example, the content adjustment system 200 can determine the UDS 222 as the UES 218. For example, the content adjustment system 200 can equate a level of engagement between the user and the virtual content 212 to a level of user distraction by the virtual content 212. In another example, the content adjustment system 200 can determine the UDS 222 based on both the UES 218 and the CPS 220. For example, even if a user does not highly and / or actively engage with the virtual content 212 (corresponding to a low UES 218), the user may still be distracted by highly salient content (corresponding to a high CPS 220). In some cases, the content adjustment system 200 may determine the UDS 222 as an average (e.g., a weighted average) of the UES 218 and the CPS 220. In a non-limiting example, the content adjustment system 200 may assign a weight of 0.7 to the UES 218 and a weight of 0.3 to the CPS 220.
[0066]
[0073] In some cases, the content adjustment engine 214 may determine the content adjustment 224 based at least in part on a comparison between the ESS 216 and the UDS 222. For example, the ESS 216 may represent a “threshold” user distraction score. If the UDS 222 is below the ESS 216, the content adjustment engine 214 may determine that the level of user distraction (and / or the level of engagement between the user and the virtual content 212) does not exceed the threshold level (e.g., is equal to or less than the threshold level). Thus, the content adjustment engine 214 may determine not to adjust the virtual content 212 based on a determination that the level of user distraction (and / or the level of engagement between the user and the virtual content 212) does not exceed the threshold level. For example, the content adjustment engine 214 may determine that no adjustment is necessary if the level of user distraction (and / or the level of engagement between the user and the virtual content 212) does not exceed the threshold level (or is equal to or less than the threshold level). In other examples, the content adjustment engine 214 may determine to adjust the virtual content 212 (e.g., by increasing the salience of the virtual content 212). If the UDS 222 exceeds the ESS 216, the content adjustment engine 214 may determine that the level of user distraction (and / or the level of engagement between the user and the virtual content 212) exceeds a threshold level. The content adjustment engine 214 may determine an appropriate content adjustment 224 based on the determination that the level of user distraction (and / or the level of engagement between the user and the virtual content 212) exceeds a threshold level. The content adjustment engine 214 may then apply the appropriate content adjustment 224 to the virtual content 212. As described above, the ESS 216, the UES 218, the CPS 220, and the UDS 222 may be normalized (e.g., determined using a scale with the same range). This normalization may facilitate a direct comparison between the ESS 216 and the UDS 222.However, in some cases, the content adjustment system 200 may utilize scores with different ranges and / or the content adjustment engine 214 may determine whether the UDS 222 and the ESS 216 differ by a threshold amount (defined by a safety threshold or a usability threshold).
[0067]
[0074] The content adjustment engine 214 can determine the appropriate content adjustment 224 in a variety of ways. In some cases, the content adjustment 224 can be designed to decrease or increase the CPS 220. For example, if the UDS 222 is determined using the CPS 220, applying the content adjustment 224 to the virtual content 212 can cause a corresponding decrease or increase in the UDS 222. In some cases, the content adjustment 224 can be configured to cause a decrease in the CPS 220, which results in a UDS 222 that falls below the ESS 216. In other cases, the content adjustment 224 can be configured to cause an increase in the CPS 220, which results in a UDS 222 that meets or approaches the ESS 216. In some examples, the content adjustment engine 214 can determine to adjust a particular type (e.g., modality) of virtual content within the virtual content 212. For example, the content adjustment engine 214 can adjust the type of virtual content with the highest salience or adjust the type of virtual content with the lowest salience. In another example, the content adjustment engine 214 can adjust multiple types of virtual content (e.g., two or more of visual virtual content, audio virtual content, and haptic virtual content). In a non-limiting example, the content adjustment engine 214 can adjust the types of virtual content according to weights assigned to the content types when determining the CPS 220.
[0068]
[0075] The content adjustment engine 214 can configure content adjustments to reduce the salience of the virtual content 212 in various ways. For example, the content adjustments 224 can include reducing the volume of the audio virtual content. Additionally or alternatively, the content adjustments 224 can include reducing the complexity of the audio virtual content (e.g., removing voices while allowing sound effects and / or music). In additional examples, the content adjustments 224 can include reducing the brightness, size, color saturation (and / or other color characteristics), opacity, and / or visual complexity of the visual virtual content within the display of the XR device. Additionally or alternatively, the content adjustments 224 can include changing the position of the visual virtual content within the display of the XR device. For example, the content adjustments 224 can include increasing the depth of the visual virtual content (e.g., to make the visual virtual content appear further ahead in the background and / or further away from the user). In another example, if an environmental factor (e.g., a physical obstacle) is obstructed from the user's view by a portion of the visual virtual content, the content adjustment 224 may include moving the visual virtual content to a different area of the display such that the environmental factor is visible to the user. In a further example, the content adjustment 224 may include reducing and / or stopping the movement of animated virtual content. In a further example, the content adjustment 224 may include changing the FOV occupied by the visual virtual content. For example, the content adjustment 224 may include reducing the size of the FOV to increase visibility of real-world objects.
[0069]
[0076] In some cases, the content adjustment 224 may include displaying additional virtual content to the user. The additional virtual content may indicate to the user the presence of one or more environmental factors. For example, the content adjustment 224 may include providing an audio message through a headset of the XR device that alerts the user to a nearby environmental factor. In another example, the content adjustment 224 may include displaying within a display of the XR device an outline of the environmental factor, a label identifying the environmental factor, an arrow pointing to the environmental factor, and / or any suitable visual identification of the environmental factor.
[0070]
[0077] The content adjustment engine 214 can configure the content adjustment to increase the prominence of the virtual content 212 in various ways. For example, the content adjustment 224 can include increasing the volume of the audio virtual content. Additionally or alternatively, the content adjustment 224 can include increasing the complexity of the audio virtual content (e.g., enabling voices, sound effects, and / or music). In additional examples, the content adjustment 224 can include increasing the brightness, size, color saturation (and / or other color characteristics), opacity, and / or visual complexity of the visual virtual content within the display of the XR device. Additionally or alternatively, the content adjustment 224 can include changing the position of the visual virtual content within the display of the XR device. For example, the content adjustment 224 can include decreasing the depth of the visual virtual content (e.g., to make the visual virtual content appear closer to the foreground and / or closer to the user). In further examples, the content adjustment 224 can include increasing and / or enabling the movement of animated virtual content. In another example, the content adjustment 224 can include modifying the FOV occupied by the visual virtual content. For example, the content adjustment 224 can include increasing the size of the FOV. Additionally, the content adjustment 224 can include removing all or a portion of a previous limitation imposed on the prominence of the virtual content 212. For example, if the content adjustment engine 214 previously reduced the prominence of the virtual content 212 based on detecting one or more environmental factors that presented a safety risk to the user, the content adjustment engine 214 can return the virtual content 212 to its original and / or intended state after detecting that the one or more environmental factors are no longer present.
[0071]
[0078] FIG. 3 illustrates an exemplary XR display 300 displayed to a user of an XR device. The XR display 300 includes real-world content and virtual content overlaid on the real-world content. For example, the table 304 and person 306 illustrated in FIG. 3 correspond to physical objects in a real-world environment that may be viewed on or through the XR display 300. The XR display 300 may also include an avatar 302 that corresponds to the virtual content overlaid on the real-world content viewed through the XR display 300. In some cases, the avatar 302 may accompany audio virtual content. In some examples, the content adjustment system 200 of FIG. 2 may determine an environmental safety score (ESS) associated with the real-world environment based at least in part on detecting the table 304 and / or the person 306. The content adjustment system 200 may also determine a user distraction score (UDS) associated with the user based at least in part on characteristics of the avatar 302 and / or characteristics of the user. In one example, the content adjustment system 200 may determine that the UDS does not exceed the ESS. Based on determining that the UDS does not exceed the ESS, the content adjustment system 200 may determine not to adjust the characteristics of the virtual content in the XR display 300 (e.g., the content adjustment system 200 may determine that adjustment is not currently necessary). In another example, based on determining that the UDS does not exceed the ESS, the content adjustment system 200 may determine to adjust the characteristics of the virtual content to increase the prominence of the virtual content in the XR display 300. In another example, the content adjustment system 200 may determine that the UDS exceeds the ESS. Based on determining that the UDS exceeds the ESS, the content adjustment system 200 may determine to adjust one or more characteristics of the virtual content in the XR display 300. In a further example, the content adjustment system 200 may determine that the UDS meets (or is sufficiently close to) the ESS.Based on determining that the UDS meets or is sufficiently close to the ESS, the content adjustment system 200 may decide not to adjust the characteristics of the virtual content in the XR display 300.
[0072]
[0079] FIG. 4A illustrates an example of adjusting one or more characteristics of virtual content in the XR display 300 of FIG. 3. In this example, the content adjustment system 200 may determine that the person 306 is attempting to communicate with the user (e.g., based on the person 306 being in close proximity to the user and / or addressing the user by name). In response, the content adjustment system 200 may reduce the prominence of the avatar 302. For example, the adjusted avatar 402 illustrated in FIG. 4A corresponds to the avatar 302 in FIG. 3 after the content adjustment system 200 has reduced the size of the avatar 302. In some cases, the content adjustment system 200 may also reduce the volume and / or complexity of audio content associated with the avatar 302. By reducing the prominence of the avatar 302 and / or the associated audio content, the content adjustment system 200 may reduce the level of user distraction associated with the avatar 302, thereby allowing the user to direct a sufficient amount of attention to the person 306. In some cases, the content adjustment system 200 may revert the adjusted avatar 402 to its original state (e.g., avatar 302) based on determining that the user is no longer communicating with the person 306.
[0073]
[0080] FIG. 4B illustrates another example of adjusting one or more characteristics of virtual content in the XR display 300 of FIG. 3. In this example, the content adjustment system 200 can determine that the user is traveling in a path that includes the table 304. In response, the content adjustment system 200 can indicate the table 304 to the user by displaying additional virtual content in the XR display 300. For example, as illustrated in FIG. 4B, the content adjustment system 200 can add a virtual outline 404 that highlights the table 304. The virtual outline 404 can reduce a level of user distraction associated with the avatar 302, thereby allowing the user to change course before colliding with the table 304. In some cases, the content adjustment system 200 can remove the outline 404 from the XR display 300 based on determining that the table 304 is no longer in the user's path.
[0074]
[0081] In some examples, the content adjustment system 200 can continue to adjust characteristics of the virtual content 212 based on changes in environmental factors in the real-world environment and / or based on changes in the level of user distraction by the virtual content 212. For example, the XR system 200 can include and / or correspond to a closed-loop system. When implemented as a closed-loop system, the CPS 220 determined by the content salience engine 206 can represent feedback used to determine the UDS 222. The adjustment of the salience of the virtual content 212 based at least in part on the UDS 222 can represent a system output. The updated CPS 220 resulting from the adjustment of the virtual content 212 can be fed back to the content adjustment engine 214. In some cases, the content adjustment system 200 (e.g., implemented as a closed-loop system) can maintain an appropriate and / or desired level of virtual content salience by periodically determining updated environmental safety scores, user engagement scores, content salience scores, and / or user distraction scores. Updated scores may be determined at periodic intervals (eg, every second, every five seconds, etc.) and / or as needed (eg, based on changes in the environment and / or the virtual content 212).
[0075]
[0082] In an open-loop system for adjusting virtual content in an XR environment, the adjustment of the salience of the virtual content may be made based on contextual factors related to the real-world environment and / or other contextual factors. For example, the adjustment of the salience of the virtual content may represent a system output. However, an open-loop system is not configured to receive the adjustment of the salience of the virtual content as a system input (and thus, an open-loop system does not include a feedback loop). As a result, an open-loop system may not have direct knowledge of how the adjustment of the virtual content affects the level of user distraction by the virtual content. Although the content adjustment system 200 may be implemented as an open-loop system, implementing the content adjustment system 200 as a closed-loop system may enable faster, higher quality, and / or more dynamic adjustment of the virtual content.
[0076]
[0083] Figure 5 is a flow diagram illustrating an example process 500 for coordinating virtual content in an extended reality system. For clarity, the process 500 is described with reference to the content coordination system 200 shown in Figure 2. The steps or operations outlined herein are examples and may be implemented in any combination thereof, including combinations that exclude, add, or modify some steps or operations.
[0077]
[0084] At block 502, the process 500 includes determining an environmental safety score (ESS) based on the detected environmental factors. For example, the environmental factors engine 202 of the content moderation system 200 can determine the ESS 216 based at least in part on the environmental factors 208. At block 504, the process 500 includes determining a content salience score (CPS) based on the salience of the virtual content displayed to the user. For example, the content salience engine 206 of the content moderation system 200 can determine the CPS 220 based at least in part on the virtual content 212.
[0078]
[0085] At block 506, the process 500 includes determining a user engagement score (UES) based on a level of engagement between the user and the virtual content. For example, the user engagement engine 204 can determine the UES 218 based at least in part on the user characteristics 210. At block 508, the process 500 includes determining a user distraction score (UDS) based on the user engagement score and / or the content salience score. For example, the content moderation system 200 can determine the UDS 222 based at least in part on the UES 218 and / or the CPS 220.
[0079]
[0086] At block 510, the process 500 includes determining whether the UDS exceeds the ESS. For example, the content adjustment engine 214 of the content adjustment system 200 can determine whether the UDS 222 exceeds the ESS 216 (e.g., by at least a threshold amount, such as an amount defined by a safety threshold). If the process 500 includes determining that the UDS exceeds the ESS (e.g., by at least a threshold amount), the process 500 proceeds to block 512. This determination can indicate that a current level of user distraction by the virtual content prevents (or is expected to prevent) the user from paying sufficient attention to the current environmental factors. At block 512, the process 500 includes adjusting the virtual content to lower the CPS. For example, the content adjustment engine 214 can determine an adjustment to one or more characteristics of the virtual content 212 such that, when the adjustment is applied to the virtual content 212, the UDS 222 is reduced to a score below the ESS 216. From block 512, the process 500 returns to block 502. For example, the content coordination system 200 may determine an updated ESS, an updated CPS, an updated UES, and / or an updated UDS to continue to provide proper coordination of the virtual content 212.
[0080]
[0087] If process 500 includes determining at block 510 that the UDS does not exceed the ESS (e.g., determining that the UDS falls below the ESS by at least a threshold amount, such as an amount defined by a usability threshold), process 500 proceeds to block 514. This determination may indicate that a current level of user distraction by the virtual content does not (or is not expected to) prevent the user from paying a sufficient amount of attention to the current environmental factors. In some cases, the determination may also indicate that the salience of the virtual content may be increased without endangering the user due to the environmental factors. For example, content adjustment system 200 may increase the salience of virtual content 212 such that UDS 222 meets or approaches (but does not exceed) ESS 216. In one example, if content adjustment system 200 previously reduced the salience of virtual content 212, content adjustment system 200 may return virtual content 212 to its original and / or intended state. From block 514, process 500 returns to block 502. For example, the content coordination system 200 may determine an updated ESS, an updated CPS, an updated UES, and / or an updated UDS to continue to provide proper coordination of the virtual content 212.
[0081]
[0088] FIG. 6 is an exemplary graph 602 illustrating various thresholds that may be used by the content adjustment system 200 of FIG. 2 to determine appropriate adjustments to the virtual content 212. In this example, the content adjustment system 200 may determine the ESS 216 and the UDS 222 as values in the range of 0 to 10. In some cases, the content adjustment system 200 may determine whether an (x,y) coordinate pair, where the ESS 216 is the x-axis coordinate and the UDS 222 is the y-axis coordinate, falls within a region 604, a region 606, or a region 608 shown in the graph 602. As shown, the region 604 may be defined by a threshold 610 and a threshold 612, with the regions 606 and 608 located on either side of the region 604. FIG. 6 illustrates that the thresholds 610 and 612 are offset by a value of 1 from a line 614 corresponding to UDS=ESS. However, the thresholds 610 and 612 may be offset from the line 614 by any suitable amount (including no offset). In one example, the threshold 612 may correspond to a safety threshold. For example, the content adjustment system 200 may determine that a situation in which the UDS 222 exceeds the ESS 216 by more than the safety threshold is likely unsafe for a user. Thus, if an (x,y) coordinate pair corresponding to the ESS 216 and the UDS 222 falls within the region 606, the content adjustment system 200 may reduce the salience of the virtual content 212. For example, the content adjustment system 200 may reduce the salience of the virtual content 212 until the UDS 222 meets or falls below a preferred value, such as the threshold 612 or the ESS 216.
[0082]
[0089] In one example, the threshold 610 may correspond to a usability threshold. For example, the content adjustment system 200 may determine that a situation in which the UDS 222 falls below the ESS 216 by more than the usability threshold likely corresponds to an unsatisfactory and / or undesirable XR experience for the user. Thus, if the (x,y) coordinate pair corresponding to the ESS 216 and the UDS 222 falls within the region 608, the content adjustment system 200 may increase the salience of the virtual content 212. For example, the content adjustment system 200 may increase the salience of the virtual content 212 until the UDS 222 meets or approaches a preferred value, such as the threshold 610 or the ESS 216. In one example, the region 604 may correspond to an intermediate region or a hysteresis region. For example, if the (x,y) coordinate pair corresponding to the ESS 216 and the UDS 222 falls within the region 604, the content adjustment system 200 may determine not to adjust the virtual content 212. In this manner, the content adjustment system 200 can avoid unnecessary, bothersome, and / or distracting adjustment of the virtual content 212.
[0083]
[0090] In some cases, the content adjustment system 200 may increase or decrease the salience of the virtual content 212 based on the magnitude of the difference between the ESS 216 and the UDS 222. For example, the content adjustment system 200 may determine and apply a proportional and / or weighted adjustment of the virtual content 212. In an illustrative example, the content adjustment system 200 may determine the magnitude of the adjustment of the virtual content 212 using the formula P(t)=P(t-1)-X(UDS(t)-ESS(t)), where P is the CPS 220, t is the current time step, and X is a weighting factor. The time step may correspond to any suitable time interval, such as 1 second, 5 seconds, 30 seconds, etc.
[0084]
[0091] Figure 7 is a flow diagram illustrating an example process 700 for coordinating virtual content provided by an extended reality system. For clarity, the process 700 is described with reference to the content coordination system 200 shown in Figure 2. The steps or operations outlined herein are examples and may be implemented in any combination thereof, including combinations that exclude, add, or modify some steps or operations.
[0085]
[0092] At block 702, process 700 includes determining a level of distraction of a user of the extended reality system due to virtual content provided by the extended reality system. For example, content adjustment system 200 can determine a level of distraction of a user of the extended reality system due to virtual content provided by the extended reality system by determining a level of engagement of the user with the virtual content. Additionally or alternatively, content adjustment system 200 can determine a level of distraction of the user by determining a prominence of the virtual content within a display of the extended reality system, where high prominence is associated with a high level of distraction. In some cases, the content adjustment system 200 can determine the prominence of the virtual content within the display of the extended reality system by determining at least one of: the audio volume of the virtual content, the size of the virtual content, the type of virtual content, the brightness of the virtual content, the transparency of the virtual content, the level of movement of the virtual content, one or more color characteristics of the virtual content, the vibration strength of the virtual content, the vibration frequency of the virtual content, the vibration duration of the virtual content, the field of view occupied by the virtual content, the visual depth of the virtual content, and the location of the virtual content. In a further example, the content adjustment system 200 can determine the level of engagement between the user and the virtual content, at least in part, by monitoring one or more physical behaviors of the user. The one or more physical behaviors can be indicative of at least one of a virtual object and a physical object on which the user is focusing, among other factors described herein.
[0086]
[0093] At block 704, the process 700 includes determining whether a level of the user's distraction by the virtual content exceeds or is less than a distraction threshold level. The distraction threshold level may be determined based at least in part on one or more environmental factors associated with a real-world environment in which the user is located. In some cases, the one or more environmental factors may include physical objects in the real-world environment. Additionally or alternatively, the one or more environmental factors may include an interaction between a user of the extended reality system and a person. In some cases, the content adjustment system 200 may determine a distraction threshold level based at least in part on determining a level of importance of the one or more environmental factors, where a lower threshold level of distraction is associated with a higher level of importance of the one or more environmental factors. In one example, an environmental factor with a higher safety risk to the user is associated with a higher level of importance. In some cases, the content adjustment system 200 may determine whether a level of the user's distraction exceeds a distraction threshold level by comparing the level of the user's distraction to the distraction threshold level.
[0087]
[0094] At block 706, the process 700 includes adjusting one or more characteristics of the virtual content based on a determination of whether the level of the user's distraction from the virtual content exceeds or is less than the distraction threshold level. In one example, if the content adjustment system 200 determines (at block 704) that the level of the user's distraction from the virtual content exceeds the distraction threshold level, the content adjustment system may adjust one or more characteristics of the virtual content by outputting an indication of one or more environmental factors via the display of the extended reality system. Additionally or alternatively, the content adjustment system 200 may adjust one or more characteristics of the virtual content by decreasing the prominence of the virtual content within the display of the extended reality system. In another example, if the content adjustment system 200 determines (at block 704) that the level of the user's distraction from the virtual content is less than the distraction threshold level, the content adjustment system may adjust one or more characteristics of the virtual content by increasing the prominence of the virtual content within the display of the extended reality system.
[0088]
[0095] In some cases, the process 700 can include determining an updated level of distraction of the user of the extended reality system due to the virtual content with the adjusted one or more characteristics. In these cases, the process 700 can also include determining an updated threshold level of distraction based at least in part on one or more updated environmental factors associated with the real-world environment, where the one or more updated environmental factors are at least in part different from the one or more environmental factors. In some examples, the process 700 can include determining that the updated level of distraction of the user due to the virtual content with the adjusted one or more characteristics exceeds the updated threshold level of distraction, and then further adjusting one or more characteristics of the virtual content based on the determination that the updated level of distraction exceeds the updated threshold level of distraction. Further adjusting one or more characteristics of the virtual content can reduce the salience of the virtual content provided by the extended reality system. In some examples, process 700 can include determining that an updated level of distraction of the user by the virtual content with the adjusted one or more characteristics is less than an updated threshold level of distraction, and further adjusting one or more characteristics of the virtual content based on determining that the updated level of distraction is less than the updated threshold level of distraction. Further adjusting the one or more characteristics of the virtual content can increase the salience of the virtual content provided by the extended reality system.
[0089]
[0096] 8 is a diagram illustrating an example of a system for implementing some aspects of the present technology. In particular, FIG. 8 illustrates an example of a computing system 800, which may be, for example, an internal computing system, a remote computing system, a camera, or any computing device making up any of the components thereof, in which the components of the system communicate with each other using a connection 805. The connection 805 may be a physical connection to the processor 810 using a bus, or a direct connection to the processor 810, such as in a chipset architecture. The connection 805 may also be a virtual connection, a networked connection, or a logical connection.
[0090]
[0097] In some examples, computing system 800 is a distributed system in which the functionality described in this disclosure may be distributed within a data center, multiple data centers, a peer network, etc. In some examples, one or more of the described system components represent many such components, each performing some or all of the functionality described therefor. In some cases, the components may be physical or virtual devices.
[0091]
[0098] The exemplary system 800 includes at least one processing unit (CPU or processor) 810 and a connection 805 that couples various system components, including system memory 815, such as read only memory (ROM) 820 and random access memory (RAM) 825, to the processor 810. The computing system 800 may include a cache 812 of high-speed memory directly connected to the processor 810, in close proximity to the processor 810, or integrated as part of the processor 810.
[0092]
[0099] Processor 810 may include any general purpose processor and hardware or software services, such as services 832, 834, and 836 stored in storage device 830, configured to control processor 810 as well as special purpose processors, where software instructions are incorporated into the actual processor design. Processor 810 may essentially be a fully self-contained computing system, including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.
[0093]
[0100] To enable user interaction, computing system 800 includes input device(s) 845, which may represent any number of input mechanisms, such as a microphone for audio, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, etc. Computing system 800 may also include output device(s) 835, which may be one or more of several output mechanisms. In some instances, a multimodal system may allow a user to provide multiple types of input / output to communicate with computing system 800. Computing system 800 may include a communication interface 840, which may generally govern and manage user input and system output.The communication interface may be an audio jack / plug, a microphone jack / plug, a Universal Serial Bus (USB) port / plug, an Apple® Lightning® port / plug, an Ethernet® port / plug, a fiber optic port / plug, a proprietary wired port / plug, BLUETOOTH® wireless signal transmission, BLUETOOTH® Low Energy (BLE) wireless signal transmission, IBEACON® wireless signal transmission, Radio Frequency Identification (RFID) wireless signal transmission, Near Field Communication (NFC) wireless signal transmission, Dedicated Short Range Communications (DSRC) wireless signal transmission, 802.11 Wi-Fi® wireless signal transmission, a Wireless Local Area Network (WLAN) wireless signal transmission, a Wireless LAN ... The communication interface 840 may implement or facilitate receipt and / or transmission of wired or wireless communications using wired and / or wireless transceivers, including those utilizing wireless LAN (WLAN) signal transmission, visible light communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX), infrared (IR) communication wireless signal transmission, public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad-hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or any combination thereof. The communication interface 840 may also include one or more Global Navigation Satellite System (GNSS) receivers or transceivers used to determine a location of the computing system 800 based on reception of one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the US-based Global Positioning System (GPS), the Russian-based Global Navigation Satellite System (GLONASS), the Chinese-based BeiDou Navigation Satellite System (BDS), and the European-based Galileo GNSS.There is no restriction to operating on any particular hardware configuration, and therefore the basic features herein can be easily substituted for improved hardware or firmware configurations as they are developed.
[0094]
[0101] The storage device 830 may be a non-volatile and / or non-transitory and / or computer readable memory device, and may be a magnetic cassette, a flash memory card, a solid state memory device, a digital versatile disk, a cartridge, a floppy disk, a flexible disk, a hard disk, a magnetic tape, a magnetic strip / stripe, any other magnetic storage medium, a flash memory, a memristor memory, any other solid state memory, a compact disk read only memory (CD-ROM) optical disk, a rewritable compact disk (CD) optical disk, a digital video disk (DVD) optical disk, a blu-ray disk (BDD) optical disk, a holographic optical disk, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a memory The memory may be a hard disk or other type of computer readable medium capable of storing data that is accessible by a computer, such as a Stick card, a smart card chip, an EMV chip, a Subscriber Identity Module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, a random access memory (RAM), a static RAM (SRAM), a dynamic RAM (DRAM), a read only memory (ROM), a programmable read only memory (PROM), an erasable programmable read only memory (EPROM), an electrically erasable programmable read only memory (EEPROM), a flash EPROM (FLASHEPROM), a cache memory (L1 / L2 / L3 / L4 / L5 / L#), a resistive random access memory (RRAM / ReRAM), a phase change memory (PCM), a spin transfer torque RAM (STT-RAM), another memory chip or cartridge, and / or a combination thereof.
[0095]
[0102] Storage devices 830 may include software services, servers, services, etc., which cause the system to perform functions when code defining such software is executed by processor 810. In some examples, hardware services that perform a particular function may include software components stored in a computer-readable medium in conjunction with the necessary hardware components, such as processor 810, connections 805, output devices 835, etc., to perform that function.
[0096]
[0103] The term "computer-readable medium" as used herein includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instruction(s) and / or data. Computer-readable media may include non-transitory media on which data may be stored, which does not include carrier waves and / or transitory electronic signals propagating wirelessly or over wired connections. Examples of non-transitory media may include, but are not limited to, magnetic disks or tapes, optical storage media such as compact disks (CDs) or digital versatile disks (DVDs), flash memory, memory or memory devices. A computer-readable medium may have code and / or machine-executable instructions stored thereon, which may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted using any suitable means including memory sharing, message passing, token passing, network transmission, etc.
[0097]
[0104] In some examples, computer-readable storage devices, media, and memories may include cable or wireless signals containing bit streams, etc. However, when stated, non-transitory computer-readable storage media specifically excludes media such as energy, carrier signals, electromagnetic waves, and the signals themselves.
[0098]
[0105] Specific details are provided in the above description to provide a thorough understanding of the examples provided herein. However, those skilled in the art will understand that the examples may be practiced without these specific details. For clarity of explanation, in some cases, the technology may be presented as including individual functional blocks, including devices, device components, steps or routines in a method implemented in software, or functional blocks comprising a combination of hardware and software. Additional components other than those shown in the figures and / or described herein may be used. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the examples with unnecessary detail. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail to avoid obscuring the examples.
[0099]
[0106] The particular examples may be described above as a process or method that is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although the flowcharts may describe operations as a sequential process, many of the operations may be performed in parallel or simultaneously. Moreover, the order of operations may be rearranged. A process is terminated when an operation of a process is completed, but may have additional steps not included in the diagram. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the function to the calling function or to the main function.
[0100]
[0107] The processes and methods according to the examples described above may be implemented using computer-executable instructions stored or otherwise available from a computer-readable medium. Such instructions may include, for example, instructions and data that cause or otherwise configure a general purpose computer, a special purpose computer, or a processing device to perform a certain function or group of functions. Portions of the computer resources used may be accessible over a network. The computer-executable instructions may be, for example, binaries, intermediate format instructions such as assembly language, firmware, source code, etc. Examples of computer-readable media that may be used to store instructions, information used, and / or information created during methods according to the described examples include magnetic or optical disks, flash memory, USB devices with non-volatile memory, networked storage devices, etc.
[0101]
[0108] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, the program code or code segments (e.g., computer program product) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. A processor(s) may perform the necessary tasks. Common examples of form factors include laptops, smartphones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rackmount devices, standalone devices, etc. The functionality described herein may also be embodied in peripheral devices or add-in cards. Such functionality may also be implemented on a circuit board among different chips or different processes executing in a single device, as further examples.
[0102]
[0109] The instructions, media for carrying such instructions, computing resources for executing them, and other structures for supporting such computing resources are exemplary means for providing the functionality described in this disclosure.
[0103]
[0110] In the above description, the aspects of the present application have been described with reference to specific examples thereof, but those skilled in the art will recognize that the present application is not limited thereto. Thus, although illustrative examples of the present application have been described in detail herein, it should be understood that the inventive concept may be embodied and employed in various ways, and the appended claims are to be construed to include such variations, except as limited by the prior art. Various features and aspects of the application examples described above may be used individually or together. Moreover, the examples may be utilized in any number of environments and applications other than those described herein without departing from the broader spirit and scope of the present specification. Thus, the specification and drawings should be considered illustrative and not limiting. For purposes of illustration, the methods have been described in a particular order. It should be appreciated that in alternative examples, the methods may be performed in an order different from that described.
[0104]
[0111] Those skilled in the art will appreciate that the less than ("<") and greater than (">") symbols or terminology used herein may be replaced with the less than or equal to ("≦") and greater than or equal to ("≧") symbols, respectively, without departing from the scope of this description.
[0105]
[0112] When a component is described as being "configured to" perform some operation, such configuration may be achieved, for example, by designing electronic circuitry or other hardware to perform the operation, by programming a programmable electronic circuit (e.g., a microprocessor or other suitable electronic circuitry) to perform the operation, or any combination thereof.
[0106]
[0113] The phrase "coupled to" refers to any component that is physically connected, either directly or indirectly, to another component, and / or that is in communication, either directly or indirectly, with another component (e.g., connected to another component via a wired or wireless connection, and / or other suitable communications interface).
[0107]
[0114] Claim language or other language reciting "at least one of" a set and / or "one or more" of a set indicates that one member of the set or more than one member of the set (in any combination) satisfies the claim. For example, claim language reciting "at least one of A and B" means A, B, or A and B. In another example, claim language reciting "at least one of A, B, and C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more" of a set does not limit the set to the items listed in the set. For example, claim language reciting "at least one of A and B" can mean A, B, or A and B, and can further include items not listed in the set of A and B.
[0108]
[0115] The various exemplary logic blocks, modules, circuits, and algorithm steps described with respect to the examples disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly illustrate this interchangeability of hardware and software, the various exemplary components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in various ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
[0109]
[0116] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as a general purpose computer, a wireless communication device handset, or an integrated circuit device having multiple uses, including applications in wireless communication device handsets and other devices. Features described as modules or components may be implemented together in an integrated logic device, or separately as separate but interoperable logic devices. If implemented in software, the techniques may be realized at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, perform one or more of the methods described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may comprise a memory or data storage medium, such as a random access memory (RAM), such as a synchronous dynamic random access memory (SDRAM), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic or optical data storage medium, or the like. The techniques may additionally or alternatively be realized at least in part by a computer-readable communications medium, such as a propagated signal or radio wave, that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer.
[0110]
[0117] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such a processor may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Thus, the term "processor" as used herein may refer to any of the above structures, any combination of the above structures, or any other structure or apparatus suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described herein may be provided within dedicated software or hardware modules configured for encoding and decoding, or may be incorporated into a composite video encoder / decoder (codec).
[0111]
[0118] Exemplary aspects of the present disclosure include the following.
[0112]
[0119] Aspect 1: An apparatus for adjusting virtual content provided by an extended reality system, the apparatus including: a memory; and a processor coupled to the memory, the processor configured to: determine a level of distraction of a user of the extended reality system due to virtual content provided by the extended reality system; determine whether the level of distraction of the user due to the virtual content exceeds or is less than a distraction threshold level; and adjust one or more characteristics of the virtual content based on a determination of whether the level of distraction of the user due to the virtual content exceeds or is less than a distraction threshold level, the distraction threshold level being determined at least in part based on one or more environmental factors related to a real-world environment in which the user is located.
[0113]
[0120] Aspect 2: The device of aspect 1, wherein the processor is configured to determine a level of distraction of a user of the extended reality system by virtual content provided by the extended reality system, at least in part, by determining a level of engagement between the user and the virtual content.
[0114]
[0121] Aspect 3: The apparatus of aspect 2, wherein the processor is configured to determine a level of engagement between the user and the virtual content, at least in part, by monitoring one or more physical behaviors of the user.
[0115]
[0122] Aspect 4: The apparatus of aspect 3, wherein the one or more physical behaviors of the user indicate at least one of a virtual object and a physical object on which the user is focusing.
[0116]
[0123] Aspect 5: An apparatus as described in any one of aspects 1 to 4, wherein the processor is configured to determine a level of distraction of a user of the extended reality system by virtual content provided by the extended reality system, at least in part by determining a prominence of the virtual content within a display of the extended reality system, with high prominence being associated with a high level of distraction.
[0117]
[0124] Aspect 6: The device of aspect 5, wherein the processor is configured to determine prominence of the virtual content within the display of the extended reality system by determining, at least in part, at least one of: an audio volume of the virtual content, a size of the virtual content, a type of the virtual content, a brightness of the virtual content, a transparency of the virtual content, a level of movement of the virtual content, one or more color characteristics of the virtual content, a vibration strength of the virtual content, a vibration frequency of the virtual content, a vibration duration of the virtual content, a field of view occupied by the virtual content, a visual depth of the virtual content, and a position of the virtual content.
[0118]
[0125] Aspect 7: An apparatus described in any one of aspects 1 to 6, wherein the one or more environmental factors include physical objects in a real-world environment.
[0119]
[0126] Aspect 8: An apparatus described in any one of aspects 1 to 7, wherein the one or more environmental factors include an interaction between a user of the extended reality system and a person.
[0120]
[0127] Aspect 9: The device of any one of aspects 1 to 8, wherein the processor is configured to determine a threshold level of distraction based at least in part on determining a level of importance of one or more environmental factors, a lower threshold level of distraction being associated with a higher level of importance of the one or more environmental factors.
[0121]
[0128] Aspect 10: The apparatus of aspect 9, wherein environmental factors that pose a high safety risk to the user are associated with a high level of importance.
[0122]
[0129] Aspect 11: The device of any one of aspects 1 to 10, wherein the processor is configured to determine that a level of distraction of the user due to the virtual content exceeds a threshold level of distraction, and adjust one or more characteristics of the virtual content by outputting an indication of one or more environmental factors via a display of the extended reality system based on the level of distraction of the user due to the virtual content exceeding the threshold level of distraction.
[0123]
[0130] Aspect 12: The device of any one of aspects 1 to 11, wherein the processor is configured to determine that a level of distraction of the user by the virtual content exceeds a distraction threshold level, and adjust one or more characteristics of the virtual content by reducing the prominence of the virtual content in the display of the extended reality system based on the level of distraction of the user by the virtual content exceeding the distraction threshold level.
[0124]
[0131] Aspect 13: The device of any one of aspects 1 to 12, wherein the processor is further configured to determine that a level of distraction of the user by the virtual content is less than a distraction threshold level, and adjust one or more characteristics of the virtual content by increasing the prominence of the virtual content within the display of the extended reality system based on the level of distraction of the user by the virtual content being less than the distraction threshold level.
[0125]
[0132] Aspect 14: The device of any one of aspects 1 to 13, wherein the processor is further configured to determine an updated level of distraction of the user of the extended reality by virtual content having one or more adjusted characteristics, and to determine an updated threshold level of distraction based at least in part on one or more updated environmental factors related to the real-world environment, the one or more updated environmental factors being at least in part different from the one or more environmental factors.
[0126]
[0133] Aspect 15: The device of aspect 14, wherein the processor is further configured to determine that an updated level of distraction of the user by the virtual content having the adjusted one or more characteristics exceeds an updated threshold level of distraction, and further adjust one or more characteristics of the virtual content based on a determination that the updated level of distraction exceeds the updated threshold level of distraction, wherein further adjusting the one or more characteristics of the virtual content reduces salience of the virtual content provided by the extended reality system.
[0127]
[0134] Aspect 16: The device of any one of aspects 14 or 15, wherein the processor is further configured to determine that an updated level of distraction of the user by the virtual content having the adjusted one or more characteristics is less than an updated threshold level of distraction, and further adjust one or more characteristics of the virtual content based on determining that the updated level of distraction is less than the updated threshold level of distraction, wherein further adjusting the one or more characteristics of the virtual content increases salience of the virtual content provided by the extended reality system.
[0128]
[0135] Aspect 17: A method for adjusting virtual content provided by an extended reality system, the method comprising: determining a level of distraction of a user of the extended reality system due to virtual content provided by the extended reality system; determining whether the level of distraction of the user due to the virtual content exceeds or is less than a distraction threshold level; and adjusting one or more characteristics of the virtual content based on a determination of whether the level of distraction of the user due to the virtual content exceeds or is less than a distraction threshold level, the distraction threshold level being determined at least in part based on one or more environmental factors related to a real-world environment in which the user is located.
[0129]
[0136] Aspect 18: The method of aspect 17, wherein determining a level of distraction of a user of an extended reality system by virtual content provided by the extended reality system includes determining a level of engagement between the user and the virtual content.
[0130]
[0137] Aspect 19: The method of aspect 18, further comprising determining a level of engagement between the user and the virtual content, at least in part, by monitoring one or more physical behaviors of the user.
[0131]
[0138] Aspect 20: The method of aspect 19, wherein the user's one or more physical behaviors indicate at least one of a virtual object and a physical object on which the user is focusing.
[0132]
[0139] Aspect 21: A method as described in any one of aspects 17 to 20, wherein determining a level of distraction of a user of an extended reality system due to virtual content provided by the extended reality system includes determining a prominence of the virtual content within a display of the extended reality system, with high prominence being associated with a high level of distraction.
[0133]
[0140] Aspect 22: The method of aspect 21, wherein determining the prominence of the virtual content within the display of the extended reality system includes determining at least one of: audio volume of the virtual content, size of the virtual content, type of virtual content, brightness of the virtual content, transparency of the virtual content, level of movement of the virtual content, one or more color characteristics of the virtual content, vibration intensity of the virtual content, vibration frequency of the virtual content, vibration duration of the virtual content, field of view of the virtual content, visual depth of the virtual content, and position of the virtual content.
[0134]
[0141] Aspect 23: A method as described in any one of aspects 17 to 22, wherein the one or more environmental factors include physical objects in a real-world environment.
[0135]
[0142] Aspect 24: A method as described in any one of aspects 17 to 23, wherein the one or more environmental factors include an interaction between a user of the extended reality system and a person.
[0136]
[0143] Aspect 25: A method as described in any one of aspects 17 to 24, further comprising determining a threshold level of distraction based at least in part on determining a level of importance of one or more environmental factors, wherein a lower threshold level of distraction is associated with a higher level of importance of the one or more environmental factors.
[0137]
[0144] Aspect 26. The method of aspect 25, wherein environmental factors that pose a high safety risk to the user are associated with a high level of importance.
[0138]
[0145] Aspect 27: The method of any one of aspects 17 to 26, further comprising determining that the user's level of distraction exceeds a threshold level of distraction, wherein adjusting one or more characteristics of the virtual content includes outputting an indication of one or more environmental factors via a display of the extended reality system.
[0139]
[0146] Aspect 28: The method of any one of aspects 17 to 27, further comprising determining that the user's level of distraction exceeds a threshold level of distraction, wherein adjusting one or more characteristics of the virtual content includes reducing prominence of the virtual content within the display of the extended reality system.
[0140]
[0147] Aspect 29: The method of any one of aspects 17 to 28, further comprising determining that the user's level of distraction is less than a threshold level of distraction, wherein adjusting one or more characteristics of the virtual content includes reducing prominence of the virtual content within the display of the extended reality system.
[0141]
[0148] Aspect 30: A method as described in any one of aspects 17 to 29, further comprising determining an updated level of distraction of a user of an extended reality system by virtual content having one or more adjusted characteristics, and determining an updated threshold level of distraction based at least in part on one or more updated environmental factors related to the real-world environment, the one or more updated environmental factors being at least in part different from the one or more environmental factors.
[0142]
[0149] Aspect 31: The method of aspect 30, further comprising determining that an updated level of distraction of the user by virtual content having one or more adjusted characteristics exceeds an updated threshold level of distraction, and further adjusting one or more characteristics of the virtual content based on determining that the updated level of distraction exceeds the updated threshold level of distraction, wherein further adjusting the one or more characteristics of the virtual content reduces salience of the virtual content provided by the extended reality system.
[0143]
[0150] Aspect 32: The method of any one of aspects 30 or 31, further comprising determining that an updated level of distraction of the user by the virtual content having one or more adjusted characteristics is less than an updated threshold level of distraction, and further adjusting one or more characteristics of the virtual content based on determining that the updated level of distraction is less than the updated threshold level of distraction, wherein further adjusting the one or more characteristics of the virtual content increases salience of the virtual content provided by the extended reality system.
[0144]
[0151] Aspect 33: A non-transitory computer-readable storage medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to perform operations as recited in any of aspects 1 to 32.
[0145]
[0152] Example 34: An apparatus comprising means for performing the operations recited in any of examples 1 to 32.
Claims
1. 1. An apparatus for adjusting virtual content provided by an extended reality system, the apparatus comprising: Memory, a processor coupled to the memory; wherein the processor comprises: determining a user distraction score (UDS) representative of a level of distraction of a user of the extended reality system by virtual content presented on a display of the extended reality system, the UDS being determined, at least in part, by determining a level of engagement between the user and the virtual content and / or a prominence of the virtual content within a display of the extended reality system, with high prominence being associated with a high level of distraction; determining whether the UDS exceeds a threshold level of distraction, and the threshold level of distraction is determined based at least in part on one or more environmental factors related to a real-world environment in which the user is located, the real-world environment being viewable on or through at least a portion of the display of the extended reality system when the virtual content is displayed. adjusting one or more characteristics of the virtual content based on the determination of whether the UDS exceeds the threshold level of distraction, comprising outputting an indication of the one or more environmental factors via the display of the extended reality system and / or reducing prominence of the virtual content within the display of the extended reality system if the UDS is determined to exceed the threshold level of distraction; An apparatus configured to:
2. 10. The apparatus of claim 1, wherein the processor is configured to determine the level of engagement between the user and the virtual content, at least in part, by monitoring one or more physical behaviors of the user.
3. The apparatus of claim 2 , wherein the one or more physical behaviors of the user are indicative of at least one of a virtual object and a physical object on which the user is focusing.
4. 2. The device of claim 1, wherein the processor is configured to determine the prominence of the virtual content within the display of the extended reality system by determining at least one of: an audio volume of the virtual content; a size of the virtual content; a type of the virtual content; a brightness of the virtual content; a transparency of the virtual content; a level of movement of the virtual content; one or more color characteristics of the virtual content; a vibration strength of the virtual content; a vibration frequency of the virtual content; a vibration duration of the virtual content; a field of view occupied by the virtual content; a visual depth of the virtual content; and a position of the virtual content.
5. The apparatus of claim 1 , wherein the one or more environmental factors include physical objects in the real-world environment or interactions between the user and a person of the extended reality system.
6. 10. The apparatus of claim 1, wherein the processor is configured to determine the threshold level of distraction based at least in part on determining a level of importance of the one or more environmental factors, a lower threshold level of distraction being associated with a higher level of importance of the one or more environmental factors.
7. The apparatus of claim 6 , wherein environmental factors that pose a high safety risk to the user are associated with a high level of importance.
8. The processor, determining that the UDS exceeds the threshold level of distraction; adjusting the one or more characteristics of the virtual content by outputting an indication of the one or more environmental factors via the display of the extended reality system based on the UDS exceeding the threshold level of distraction; The apparatus of claim 1 configured to:
9. The processor, determining that the UDS exceeds the threshold level of distraction; adjusting the one or more characteristics of the virtual content by reducing salience of the virtual content within a display of the extended reality system based on UDS exceeding the threshold level of distraction; The apparatus of claim 1 configured to:
10. The processor, determining that the UDS is less than the threshold level of distraction; adjusting the one or more characteristics of the virtual content by increasing the prominence of the virtual content within a display of the extended reality system based on the UDS being less than the threshold level of distraction; The apparatus of claim 1 , further configured to:
11. The processor, determining an updated UDS having the adjusted one or more characteristics; determining an updated threshold level of distraction based at least in part on one or more updated environmental factors associated with the real-world environment, the one or more updated environmental factors being at least in part different from the one or more environmental factors; The apparatus of claim 1 , further configured to:
12. The processor, determining that the updated UDS with the adjusted one or more characteristics exceeds the updated threshold level of distraction; further adjusting the one or more characteristics of the virtual content based on the determination that the updated UDS exceeds the updated threshold level of distraction, wherein further adjusting the one or more characteristics of the virtual content reduces salience of the virtual content provided by the extended reality system. The apparatus of claim 11 , further configured to:
13. The processor, determining that the UDS with the adjusted one or more characteristics is less than the updated threshold level of distraction; further adjusting the one or more characteristics of the virtual content based on determining that the updated UDS is less than the updated threshold level of distraction, wherein further adjusting the one or more characteristics of the virtual content increases salience of the virtual content provided by the extended reality system. The apparatus of claim 11 , further configured to:
14. 1. A method for adjusting virtual content provided by an extended reality system, the method comprising: determining a user distraction score (UDS) representative of a level of distraction of a user of the extended reality system by virtual content presented on a display of the extended reality system, the UDS being determined, at least in part, by determining a level of engagement between the user and the virtual content and / or a prominence of the virtual content within a display of the extended reality system, with high prominence being associated with a high level of distraction; determining whether the UDS exceeds a threshold level of distraction, and the threshold level of distraction is determined based at least in part on one or more environmental factors related to a real-world environment in which the user is located, the real-world environment being viewable on or through at least a portion of the display of the extended reality system when the virtual content is displayed. adjusting one or more characteristics of the virtual content based on the determination of whether the UDS exceeds the threshold level of distraction, comprising outputting an indication of the one or more environmental factors via the display of the extended reality system and / or reducing prominence of the virtual content within the display of the extended reality system if the UDS is determined to exceed the threshold level of distraction; A method comprising:
15. 15. A non-transitory computer readable storage medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to perform the method of claim 14.
