Night mode for XR systems

Abstract

An XR system is disclosed that has a night mode to optimize performance in low light conditions. The system estimates the user's level of dark adaptation based on captured image data and selects a rendering configuration that is consistent with the user's vision. For example, the spatial resolution, temporal resolution, color mode, and luminance are adjusted based on whether the adaptation level is scotopic, mesopic, or photopic. In the case of eye tracking, the fovea is rendered for cone cells and the periphery is rendered for rod cells. In the case of no eye tracking, a pattern with different wavelengths and coloration is employed. The night mode aims to maximize the information visible on the display, save power by only displaying what the eye can perceive, and preserve night vision capability by avoiding overstimulation of rod cells.

Description

Priority Statement

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 600,698, filed November 19, 2023, which is incorporated herein by reference in its entirety. Technical Field

[0002] This disclosure generally relates to extended reality systems, and more specifically, to extended reality system user interfaces. Background Technology

[0003] Head-mounted wearable devices can be implemented with transparent or semi-transparent displays, through which users can view their surroundings. Such head-mounted wearable devices allow users to view their real-world environment through transparent or semi-transparent displays, and also to see objects generated for display that appear as part of the surrounding environment and / or superimposed on it (e.g., renderings of two-dimensional (2D) or three-dimensional (3D) graphical models, images, videos, text, and other virtual objects). This is often referred to as “augmented reality” or “AR.” Head-mounted wearable devices can also completely obscure the user’s field of view and display a virtual environment that the user can move or be moved by. This is often referred to as “virtual reality” or “VR.” In a hybrid form, a view of the surrounding environment is captured using a camera device, and then that view, along with augmentations, is displayed to the user on a display that obscures their eyes. As used herein, unless the context otherwise indicates, the term Extended Reality (XR) refers to AR, VR, and / or any combination of these technologies. Attached Figure Description

[0004] In accompanying drawings that are not necessarily drawn to scale, similar reference numerals may describe similar parts in different views. To facilitate identification of any discussion of a particular element or action, one or more of the highest-order digits of the reference numerals indicate the drawing number in which the element was first introduced. Some non-limiting examples are shown in the figures below:

[0005] Figure 1A These are stereoscopic images based on some examples of wearable head devices.

[0006] Figure 1B It is based on some examples Figure 1A Another stereoscopic view of the head-mounted wearable device.

[0007] Figure 2 It is a graphical representation of a machine based on some examples, which executes instructions that cause the machine to perform any or more of the methods discussed herein.

[0008] Figure 3It is a diagram based on some examples of adaptation and visual pattern types.

[0009] Figure 4A It is a diagram of the field of view or ring for foveal vision and the field of view or ring for peripheral vision of a user's eye, based on some examples.

[0010] Figure 4B and Figure 4C The following are examples of various aspects of the foveation night mode.

[0011] Figure 5 Figure 502 shows estimated rhodopsin levels for average ambient light levels and Figure 504 shows estimated dark adaptation levels for average ambient light levels, based on some examples.

[0012] Based on some examples, Figure 6A Table 602 includes eye response characteristics, and Figure 6B The following aspects of the night mode AR content processing method are shown 606.

[0013] Figure 7 It is a collaborative diagram of the components of an XR system based on some examples.

[0014] Figure 8 The night mode method is shown based on some examples.

[0015] Figure 9 Physiological dark adaptation estimation methods based on some examples are shown.

[0016] Figure 10 An eye-tracking-based adaptation estimation method is shown based on some examples.

[0017] Figure 11 Behavior-based dark adaptation estimation methods are shown based on some examples.

[0018] Figure 12 Dark adaptation estimation methods based on machine learning models are shown based on some examples.

[0019] Figure 13 The following demonstrates an eye-tracking-based night mode rendering method based on some examples.

[0020] Figure 14 The following examples illustrate non-eye-tracking-based rendering methods.

[0021] Figure 15 The machine learning pipeline is shown based on some examples.

[0022] Figure 16 The training and use of machine learning programs are illustrated using some examples.

[0023] Figure 17 A system of head-worn wearable devices based on some examples is shown.

[0024] Figure 18 It is a collaboration diagram based on some examples of networked environments.

[0025] Figure 19 It is a graphical representation based on examples such as data structures maintained in a database.

[0026] Figure 20 It is a graphical representation of a messaging system with client-side and server-side functionalities, based on some examples.

[0027] Figure 21 This is a block diagram illustrating a software architecture based on some examples. Detailed Implementation

[0028] AR systems allow users to view virtual content overlaid on the real world. The key difference between AR and VR is that AR allows users to still perceive their real environment, while VR completely replaces reality with a virtual environment.

[0029] Human vision relies on two types of photoreceptor cells—cones for daytime / bright light vision and rods for nighttime / low light vision. The transition from cone-based vision to rod-based vision is called dark adaptation, and it takes 1 to 25 minutes when moving from bright light conditions to low light conditions. Exposure to bright light during dark adaptation can impair night vision.

[0030] Traditional AR systems do not take into account the differences in human vision between bright and low-light environments. Rendering virtual content optimized for daytime use may impair or disrupt a user's dark adaptation at night, thus preventing them from seeing both virtual and real-world elements simultaneously.

[0031] To address this issue, a night mode feature for XR systems is used to optimize performance in low-light conditions. In night mode, rendering settings (such as spatial resolution, temporal resolution, color mode, and brightness) are adjusted based on ambient light levels to match the user's dark adaptation level. This provides the following advantages: maximizing the information visible on the display, saving power by showing only what the eye can perceive, and maintaining night vision capabilities to see the real-world environment. An estimate of the user's current night vision state can be used to further optimize rendering.

[0032] In some examples, the XR system uses its camera to capture image data and estimates the user's dark adaptation level based on that data. The XR system then selects a night mode rendering configuration based on the estimated dark adaptation level and uses the selected rendering configuration to generate the XR display. Finally, the method displays the XR display to the user.

[0033] In some examples, physiological dark adaptation estimation methods are used to estimate the level of dark adaptation. Physiological dark adaptation estimation methods involve estimating rod cell photoreceptor responses or analyzing eye-tracking data by flashing a weak light signal in the XR system display and then measuring electroencephalogram (EEG) signals captured from the user's visual cortex. EEG / event-related potential (EEG / ERP) signals can indicate whether photoreceptors can detect the weak light signal. Rendering configuration selection includes choosing peripheral and foveal rendering settings. Rendering configurations can include lower spatial resolution in the peripheral region compared to the foveal region, and lower temporal resolution in the peripheral region compared to the foveal region. Rendering configurations can also utilize longer wavelengths of light in the peripheral region compared to the foveal region.

[0034] In some examples, an eye-tracking-based adaptation estimation method is used to estimate the user's dark adaptation level. This method involves flashing a weak light signal in an XR system and capturing eye-tracking data of the user's eyes. The method then analyzes the eye-tracking data to detect whether the user's eyes have noticed the flash and estimates the dark adaptation level based on whether the flash has been noticed.

[0035] In some examples, behavior-based dark adaptation estimation methods are used to estimate the user's eye dark adaptation level. Behavior-based dark adaptation estimation methods involve displaying interactive instructions using low-intensity, low-contrast text in an XR system, where the instructions are displayed at different contrast levels. The method detects when the user executes the displayed interactive instructions and estimates the dark adaptation level based on the contrast level of the displayed instructions that the user can detect.

[0036] In some examples, estimating a user's eye dark adaptation level involves sensing ambient light levels in the surrounding environment and measuring the user's pupil size over a period of time. This method inputs the ambient light level and pupil size into a machine learning model and estimates the dark adaptation level based on the model's output. In some examples, this time period ranges from 25 to 35 minutes. In some examples, physiological dark adaptation estimation methods, eye-tracking-based adaptation estimation methods, and / or behavior-based dark adaptation estimation methods can be used to obtain ground truth values ​​for retraining the machine learning model.

[0037] Other technical features will be readily apparent to those skilled in the art from the following figures, description and claims.

[0038] Head wearable devices

[0039] Figure 1A This is a perspective view of an XR user device in the form of a head-mounted wearable device 100, based on some examples. The head-mounted wearable device 100 can be an XR system (e.g., Figure 18 The client device of the XR system 1802, or the head-worn device 100, can be a standalone XR system. The head-worn device 100 may include a frame 102 made of any suitable material, such as plastic or metal (including any suitable shape memory alloy). In one or more examples, the frame 102 includes a first or left optical element holder 104 (e.g., a display or lens holder) and a second or right optical element holder 106 connected by a nose bridge 112. A first or left optical element 108 and a second or right optical element 110 may be disposed within the left optical element holder 104 and the right optical element holder 106, respectively. The right optical element 110 and the left optical element 108 may be a lens, a display, a display assembly, or a combination thereof. Any suitable display assembly may be disposed in the head-worn device 100.

[0040] Frame 102 further includes a left arm or left temple piece 122 and a right arm or right temple piece 124. In some examples, frame 102 may be formed from a single piece of material to have a uniform or monolithic construction.

[0041] The head-mounted wearable device 100 may include a computing device (e.g., computer 120), which may be of any suitable type to be carried by the frame 102, and in one or more examples, may have a suitable size and shape to be partially housed in one of the left temple piece 122 or the right temple piece 124. Computer 120 may include one or more hardware processors with memory, a wireless communication circuitry, and a power supply. As discussed below, computer 120 includes a low-power circuitry system 1726, a high-speed circuitry system 1728, and a display processor. Various other examples may include these elements in different configurations or integrated in different ways. Additional details of various aspects of computer 120 may be implemented as shown by the machine 200 discussed herein.

[0042] The computer 120 also includes a battery 118 or other suitable portable power source. In some examples, the battery 118 is disposed in the left temple 122 and electrically coupled to the computer 120 disposed in the right temple 124. The head-worn wearable device 100 may include a connector or port (not shown) for charging the battery 118, a wireless receiver, a transmitter or transceiver (not shown), or a combination of such devices.

[0043] The head-mounted wearable device 100 includes a first or left camera device 114 and a second or right camera device 116. Although two camera devices are depicted, other examples envision the use of a single or additional (i.e., more than two) camera devices.

[0044] In some examples, in addition to the left camera device 114 and the right camera device 116, the head-mounted wearable device 100 includes any number of input sensors or other input / output devices. Such sensors or input / output devices may also include biometric sensors, positioning sensors, motion sensors, etc. For example, a biometric sensor may include components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body posture, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), and identifying a person (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition). Any biometric data collected by the biometric components is captured and stored only with user approval and is deleted upon user request. Furthermore, such biometric data may be used for very limited purposes (e.g., authentication). To ensure the restricted and authorized use of biometric information and other personally identifiable information (PII), access to this data is limited to authorized personnel (if access to the data occurs). Any use of biometric data may be strictly limited to identification and verification purposes, and the biometric data may not be shared or sold to any third party without the user's explicit consent. In addition, appropriate technical and organizational measures have been implemented to ensure the security and confidentiality of this sensitive information.

[0045] In some examples, the head-mounted wearable device 100 includes one or more sensors (e.g., sensor 154) for capturing data of a real-world scene. In some examples, one or more sensors include one or more scanning sensors having sensing or image-forming components movably coupled to frame 102. In some examples, one or more scanning sensors are point scanning sensors (e.g., but not limited to LiDAR sensors) that determine the distance or depth of points on a physical object or surface in a real-world scene. In some examples, one or more scanning sensors have a fixed field of view (FOV) (e.g., but not limited to scanning camera devices) used to capture image data of a real-world scene. In some examples, one or more sensors are fixed sensors (e.g., right camera device 116 and left camera device 114) fixedly coupled to frame 102 and cannot move relative to frame 102.

[0046] Position and motion sensors may include accelerometer components (e.g., accelerometers), gravity sensor components, rotation sensor components (e.g., gyroscopes), etc. In some examples, position and motion sensors may be incorporated into inertial measurement units (IMUs), etc.

[0047] In some examples, the head-worn device 100 identifies its position and orientation in three-dimensional (3D) space, where position and orientation together constitute the pose of the head-worn device 100. The pose comprises six values: three for position in a 3D Cartesian coordinate system with three orthogonal axes (horizontal or X-axis, vertical or Y-axis, and depth or Z-axis), and three for rotation about each corresponding axis (e.g., Euler angles, such as (α, β, γ), or pitch, yaw, and roll). These six values ​​are simply referred to as the device's six-dimensional (6D) pose. The pose tracking components (not shown) of the head-worn device 100 may include sensors and components such as, but not limited to, a right camera device 116, a left camera device 114, a Global Positioning System (GPS), an IMU, a gravimeter, etc., whose outputs are combined to track the movement, orientation, and position of the head-worn device 100. The task of determining the pose of the head-worn device 100 is called pose estimation.

[0048] In some examples, the pose tracking component uses the IMU of the head wearable device 100 and the output of one or more camera devices to track the pose of the head wearable device 100 based on the visual simultaneous localization and mapping (vSLAM) method.

[0049] During an XR experience, the head-worn device 100 can continuously estimate its pose in a 3D coordinate system. The position of the head-worn device 100 is measured by its displacement relative to the origin of the 3D coordinate system, and the orientation is measured by the angular (rotational) displacement of the head-worn device 100's axes relative to the axes of the 3D coordinate system. Position is represented by a set of points (e.g., Cartesian coordinates, such as (x, y, z)). Orientation is typically represented by a set of rotation angles (e.g., Euler angles, such as (α, β, γ)). Other parameterizations for expressing rotational displacement can be used, such as quaternions or angular axis representations. The pose can be expressed as a transformation matrix or a mapping.

[0050] In some examples, the left camera device 114 and the right camera device 116 provide video frame data for use by the head-worn device 100 to extract 3D information from the real-world scene, including depth or displacement relative to the head-worn device 100 along the Z-axis.

[0051] The head-mounted wearable device 100 can also construct and maintain one or more 3D reference frames, each including a corresponding coordinate system. For example, the head-mounted wearable device 100 can construct a local real-world scene reference frame, a global real-world reference frame, and a reference frame associated with the head-mounted wearable device 100, etc. Each reference frame can be associated with transformations that relate position and orientation in different reference frames. As an example, depth measurements and orientations from the head-mounted wearable device 100 can be transformed into the local coordinate system of the local real-world scene reference frame to identify the position of the corresponding physical object in the real-world scene reference frame and coordinate system.

[0052] The head-worn device 100 may also include a touchpad 126 mounted to or integrated with one or both of the left temple 122 and the right temple 124. The touchpad 126 is arranged generally vertically, and in some examples is approximately parallel to the user's temple. As used herein, generally vertical alignment means that the touchpad is more vertical than horizontal, though it may be more vertical than this. Additional user input may be provided via one or more buttons 128, which, in the illustrated example, are located on the outer upper edges of the left optics retainer 104 and the right optics retainer 106. The touchpad 126 and buttons 128 provide a means by which the head-worn device 100 can receive input from the user of the head-worn device 100.

[0053] Figure 1B The head-worn device 100 is shown from the perspective of a user wearing it. For clarity, [the following text is incomplete and likely refers to a different viewpoint]. Figure 1B The middle part is omitted Figure 1A Many elements are shown in the diagram. For example... Figure 1A The above, Figure 1B The wearable head device 100 shown includes a left optical element 140 and a right optical element 144, which are respectively fixed in a left optical element holder 132 and a right optical element holder 136.

[0054] The head-mounted wearable device 100 includes: a right front optical assembly 130, which includes a left near-eye display 150 and a right near-eye display 134; and a left front optical assembly 142, which includes a left projector 146 and a right projector 152.

[0055] In some examples, the near-eye display is a waveguide. The waveguide includes reflective or diffractive structures (e.g., gratings and / or optical elements such as mirrors, lenses, or prisms). Light 138 emitted by the right projector 152 encounters the diffractive structure of the waveguide of the right near-eye display 134, which directs the light toward the user's right eye to provide an image on or in the right optical element 144, superimposed with a view of the real-world scene seen by the user. Similarly, light 148 emitted by the left projector 146 encounters the diffractive structure of the waveguide of the left near-eye display 150, which directs the light toward the user's left eye to provide an image on or in the left optical element 140, superimposed with a view of the real-world scene seen by the user. A combination of a graphics processing unit (GPU), an image display driver, the right front optical element 130, the left front optical element 142, the left optical element 140, and the right optical element 144 provides the optical engine for the head-mounted wearable device 100. The head-mounted wearable device 100 uses an optical engine to generate an overlay of the user's real-world scene view, including displaying a user interface to the user of the head-mounted wearable device 100.

[0056] However, it should be understood that other display technologies or configurations can be used within the optical engine to display images to the user within their field of view. For example, instead of projectors and waveguides, LCDs, LEDs, or other display panels or surfaces can be provided.

[0057] In use, the user of the head-mounted wearable device 100 will see information, content, and various user interfaces on a near-eye display. As described in more detail herein, the user can then use the touchpad 126 and / or buttons 128 on associated devices (e.g., Figure 17 Interact with the head-worn device 100 by touch input or voice input on the mobile device 1714 shown, and / or by hand movements, positioning and location recognized by the head-worn device 100.

[0058] In some examples, the optical engine of the XR system is incorporated into a lens, such as a contact lens, that comes into contact with the user's eye. The XR system uses the contact lens to generate images for the XR experience.

[0059] In some examples, the head-worn wearable device 100 includes one or more eye-tracking sensors (such as eye-tracking sensor 156 and eye-tracking sensor 158) operable to capture eye-tracking data of one or more eyes of the user of the head-worn wearable device 100.

[0060] In some examples, the head-mounted wearable device 100 includes an XR system. In some examples, the head-mounted wearable device 100 is a component of an XR system that includes additional computing components. In some examples, the head-mounted wearable device 100 is a component of an XR system that includes additional user input systems or devices.

[0061] Machine architecture

[0062] Figure 2This is a schematic representation of machine 200, within which instructions 202 (e.g., software, programs, applications, applets, or other executable code) can be executed to cause machine 200 to perform any or more of the methods discussed herein for head-worn or mobile devices. For example, instructions 202 can cause machine 200 to perform any or more of the methods described herein. Instructions 202 transform the general, unprogrammed machine 200 into a specific machine 200 programmed to perform the described and illustrated functions in the described manner. Machine 200 can operate as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, machine 200 can operate as a server machine or client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. Machine 200 may include, but is not limited to, server computers, client computers, personal computers (PCs), tablet computers, laptop computers, netbooks, set-top boxes (STBs), personal digital assistants (PDAs), entertainment media systems, cellular phones, smartphones, mobile devices, wearable devices (e.g., smartwatches), smart home devices (e.g., smart appliances), other smart devices, web devices, network routers, network switches, network bridges, or any machine capable of sequentially or otherwise executing instructions 202 specifying actions to be taken by machine 200. Furthermore, while a single machine 200 is shown, the term "machine" should also be considered as a collection of machines that individually or jointly execute instructions 202 to perform any or more of the methods discussed herein. For example, machine 200 may include any one of XR system 1802 or any of a plurality of server devices forming part of interactive server system 1812. In some examples, machine 200 may also include both client and server systems, wherein certain operations of a particular method or algorithm are performed on the server side and certain operations of a particular method or algorithm are performed on the client side.

[0063] Machine 200 may include hardware processor 204, memory 206, and input / output (I / O) components 208 that can be configured to communicate with each other via bus 210. In the example, processor 204 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a GPU, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, processors 212 and 214 that execute instruction 202. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") capable of executing instructions simultaneously. Figure 2 Multiple processors 204 are shown, but machine 200 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0064] Memory 206 includes main memory 216, static memory 240, and storage cells 218, all of which are accessible by processor 204 via bus 210. Main memory 206, static memory 240, and storage cells 218 store instructions 202 embodying any one or more of the methods or functions described herein. Instructions 202 may also reside wholly or partially in main memory 216, in static memory 240, in machine-readable medium 220 within storage cell 218, in at least one processor of processor 204 (e.g., in the processor's cache memory), or in any suitable combination thereof during execution by machine 200.

[0065] I / O component 208 may include various components for receiving input, providing output, generating output, sending information, exchanging information, capturing measurements, etc. The specific I / O component 208 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine may not include such a touch input device. It should be understood that I / O component 208 may include... Figure 2Many other components are not shown. In various examples, I / O component 208 may include user output component 222 and user input component 224. User output component 222 may include visual components (e.g., displays such as plasma display panels (PDPs), light-emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tube (CRT) displays), acoustic components (e.g., speakers), haptic components (e.g., vibrating motors, resistive mechanisms), other signal generators, etc. User input component 224 may include alphanumeric input components (e.g., keyboards, touchscreens configured to receive alphanumeric input, photoelectric keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touchpads, trackballs, joysticks, motion sensors, or other pointing instruments), haptic input components (e.g., physical buttons, touchscreens or other haptic input components that provide positioning and force for touch or touch gestures), audio input components (e.g., microphones), etc.

[0066] For example, environmental component 230 may include one or more camera devices (with still image / photograph and video capabilities), lighting sensor components (e.g., photometers), temperature sensor components (e.g., one or more thermometers for detecting ambient temperature), humidity sensor components, pressure sensor components (e.g., barometers), acoustic sensor components (e.g., one or more microphones for detecting background noise), proximity sensor components (e.g., infrared sensors for detecting nearby objects), gas sensors (e.g., gas detection sensors for detecting the concentration of hazardous gases for safety purposes or for measuring pollutants in the atmosphere), depth or distance sensors (e.g., sensors for determining the distance to an object or for determining the depth of object features in a 3D coordinate system), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment.

[0067] Positioning component 232 and motion component 228 include positioning sensor components (e.g., GPS receiver components), altitude sensor components (e.g., altimeters or barometers that detect air pressure, from which altitude can be determined), orientation sensor components (e.g., magnetometers), etc. In some examples, positioning component 232 and motion component 228 may be incorporated into an IMU, etc.

[0068] Biometric component 226 may include components for detecting expressions (e.g., hand gestures, facial expressions, voice expressions, body posture, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), and identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based identification). Any biometric data collected by the biometric component is captured and stored only with user approval and is deleted upon user request. Furthermore, such biometric data may be used for very limited purposes (e.g., authentication). To ensure the restricted and authorized use of biometric information and other personally identifiable information (PII), access to this data is limited to authorized personnel (if access to the data occurs). Any use of biometric data may be strictly limited to identification and verification purposes, and the biometric data may not be shared or sold to any third party without the user's explicit consent. In addition, appropriate technical and organizational measures are implemented to ensure the security and confidentiality of this sensitive information.

[0069] Regarding the camera device, machine 200 may have a camera device system including, for example, a front-facing camera on the front surface of the housing of machine 200 and a rear-facing camera on the rear surface of the housing of machine 200. The front-facing camera may be used, for example, to capture still images and videos (e.g., "selfies") of the user of the machine, which can then be enhanced with the enhancement data (e.g., filters) described above. The rear-facing camera may be used, for example, to capture still images and videos in a more conventional camera device mode, wherein these images are similarly enhanced with enhancement data. In addition to the front-facing and rear-facing cameras, XR system 1802 may also include a 360° camera for capturing 360° photos and videos.

[0070] Furthermore, the camera system of the machine 200 may include dual rear cameras (e.g., a main camera and a depth-sensing camera), or even triple, quadruple, or quintuple rear camera configurations on the front and rear sides of the machine 200. For example, these multiple camera systems may include wide-angle cameras, ultra-wide-angle cameras, telephoto cameras, macro cameras, and depth sensors.

[0071] A wide variety of technologies can be used to implement communication. I / O component 208 also includes a communication component 234 operable to couple machine 200 to network 236 or device 238 via a suitable coupling or connection. For example, communication component 234 may include a network interface component or other suitable device that interfaces with network 236. In further examples, communication component 234 may include a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, or a Bluetooth component.® Components (e.g., Bluetooth) ® Low energy consumption), Wi-Fi ® Components and other communication components that provide communication via other modes. Device 238 can be any peripheral device from another machine or various peripheral devices (e.g., a peripheral device coupled via USB).

[0072] Furthermore, the communication component 234 can detect identifiers, or include components operable to detect identifiers. For example, the communication component 234 may include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional barcodes such as Universal Product Code (UPC) barcodes, multi-dimensional barcodes such as Quick Response (QR) codes, Aztec codes, Data Matrix, Dataglyph, MaxiCode, PDF417, UltraCode, UCC RSS-2D barcodes, and other optical codes), or an acoustic detection component (e.g., a microphone for identifying audio signals from the tag). Additionally, various information can be derived via the communication component 234, such as location derived via Internet Protocol (IP) geolocation, location derived via Wi-Fi® signal triangulation, location derived by detecting NFC beacon signals that can indicate a specific location, etc.

[0073] Various memories (e.g., main memory 216, static memory 240, and the memory of processor 204) and storage units 218 may store one or more sets of instructions and data structures (e.g., software) embodied or used by any or more of the methods or functions described herein. These instructions (e.g., instruction 202) cause various operations to implement the disclosed examples when executed by processor 204.

[0074] Instructions 202 can be sent or received over network 236 via a transmission medium using a network interface device (e.g., a network interface component included in communication component 234) and using any of several known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, instructions 202 can be sent or received via a transmission medium through a coupling to device 238 (e.g., peer-to-peer coupling).

[0075] Physiological characteristics of dark adaptation

[0076] Figure 3 This is a diagram based on some examples of adaptation and visual mode types. Visual mode diagram 304 shows the relationship between three types of vision (scotopic vision, mesovision, and photovision) and brightness levels. In some examples, the XR system provides a night mode within the night mode range 302, such as... Figure 8 A more comprehensive description is provided in the text.

[0077] Scotopic vision refers to human vision under low-light conditions, in which only the photoreceptors in the rod cells of the retina are actively involved in viewing. Rod cells are extremely sensitive to light and can perceive light levels as low as 10⁻⁶ cd / m². 2 Scotopia allows humans to see in starlight, but it lacks color sensitivity, has poor visual acuity, and relies on peripheral vision. Full dark adaptation to achieve scotopia takes approximately 30 minutes. Scotopia allows humans to see things under starlight, but the vision is achromatic (e.g., black and white or gray shadows).

[0078] Mesovision refers to human vision under moderate lighting conditions, in which both rod and cone photoreceptors in the retina are involved in viewing. It occurs at approximately 0.01 cd / m². 2 With 3 cd / m 2 Mesovision is a brightness level between scotopic and photopic vision, where there is enough light to stimulate some cone cells but not enough to fully saturate the rod cells. Mesovision allows for some color discrimination and better visual acuity than scotopic vision, but less sensitive than full photopic vision. It is a hybrid of rod-mediated and cone-mediated vision and allows humans to see things in environments such as moonlight or dim indoor lighting. During mesovision, visual functions such as acuity, color vision, and time response fall between scotopic and photopic levels.

[0079] Photopic vision refers to human vision under good lighting conditions, in which only the cone photoreceptors in the retina are actively engaged in viewing. It occurs at brightness levels above 3 cd / m², which fully saturates the rod cells. Photopic vision allows for excellent visual acuity, rapid flicker response, and color vision mediated by three types of cone cells. Cone cells are concentrated in the fovea and provide high-resolution central vision. Photopic vision enables humans to see fine details and colors in daylight, indoor lighting, and other bright illumination. Full visual abilities such as reading, object recognition, and color perception are achieved under photopic conditions.

[0080] Figure 4A Figure 406 shows the foveal and peripheral vision of a user's eye based on some examples, including the field of view or ring 402 for foveal vision and the field of view or ring 404 for peripheral vision. Figure 4B and Figure 4CAspects of a foveated night mode 408 according to some examples are illustrated. In some examples, the XR system provides a night mode by offering different content to the user's field of vision depending on whether the content is to be displayed in the field of vision or ring 402 of foveal vision or the field of vision or ring 404 of peripheral vision. For example, a raw image 416 of a real-world scene can be captured. The image is displayed to the user as a display image 418 using a red light wavelength (e.g., 640 nm), thus preserving the night vision capability of the user's rod cells. When gaze tracking, such as eye tracking, is used, the display image 418 can be a foveated region that includes a 5-degree field of vision of the user's foveated region 420. By doing so, the night vision capability of the rod cells is preserved, and more energy is saved due to the smaller display area and less energy required for processing. RGB content 422 can be displayed in the foveated region of the user's vision without interfering with the night vision capability of the rod cells. Furthermore, RGB content contains more information than a monochrome image. In some examples, green light (e.g., light with a wavelength of 498 nm) can be used to display content 424. In some examples, 498nm was chosen because using that wavelength for content display offered the greatest energy savings. In some examples, for content displayed only to rod cells, the retina was positioned within a 15- to 20-degree range from the gaze direction, where rod cell density reached its maximum. Spatial and temporal resolutions can be much lower than required for daytime vision, and spatial resolution can be adaptively reduced toward the periphery.

[0081] Reference Figure 13 A more comprehensive description of the gaze point night mode 408.

[0082] Based on some examples, Figure 5 Figure 502 shows the estimated rhodopsin level for average ambient light levels, and Figure 504 shows the estimated dark adaptation level for average ambient light levels. In some examples, the XR system provides a night mode in part by estimating the user's eye's dark adaptation level, as shown in Figure 504. Figure 12 A more comprehensive description.

[0083] Based on some examples, Figure 6A Table 602 includes eye response characteristics, and Figure 6B Various aspects of a night mode AR content processing method 606 are illustrated. The XR system provides a night mode in part based on the physiological characteristics of the user's eyes adapting to darkness. In the night mode AR content processing method 606, the input content 604 is desaturated while maintaining contrast 610, instead of simply performing initial desaturation 608, as shown in the reference... Figure 14 A more comprehensive description.

[0084] Night mode

[0085] Based on some examples, Figure 7 This is a collaborative diagram of the components of the XR system 706 with night mode, and Figure 8 This is a flowchart of the night mode method 800 of the XR system 706.

[0086] Although the night mode method 800 describes a specific sequence of operations, this sequence can be changed without departing from the scope of this disclosure. For example, some of the described operations can be performed in parallel, in a different order, or by different components of the XR system without substantially affecting the functionality of the method.

[0087] The night mode method 800 is used by the XR system 706 to provide an XR user interface 716 to the user 704. The XR system 706 includes XR user devices, such as, but not limited to, a head-mounted wearable device 100. The XR system 706 uses the XR user device to provide the XR user interface 716 to the user 704. The XR user interface 716 is generated by an XR application 702 of the XR system 706, which uses the services of the night mode component 728 to provide a night mode for an XR experience that includes a real-world scene 718 comprising one or more physical objects 722. The XR application 702 can be a utility application, such as an interactive game, maintenance guide, interactive map, interactive tour guide, tutorial, etc. The XR application 702 can also be an entertainment application, such as a video game, interactive video, etc.

[0088] The physical object 722 may include objects in the real-world scene 718 and one or more parts of the user's body 704, such as, but not limited to, the user's hand.

[0089] To optimize rendering in night mode, when the XR user interface 716 is displayed to user 704, the night mode component 728 prioritizes maximizing the amount of information from the visible content on the display of the XR system 706. The night mode component 728 can achieve this by using RGB images instead of grayscale, as RGB conveys more information.

[0090] In some examples, the night mode component 728 conserves energy by calculating and displaying what the eye can actually perceive. For calculation, the night mode component 728 renders content that the eye will be able to see based on the eye's adaptation level. For display, the night mode component 728 adjusts brightness based on the increased sensitivity of cone and rod cells in low light. The night mode component 728 can also use 498 nm light to display content for rod cells (at which rod cells are most sensitive) to further reduce the energy required relative to other wavelengths.

[0091] In some examples, the night mode component 728 maintains night vision capabilities so that users can see their surroundings. The night mode component 728 avoids illuminating the rod cells with content that is not targeted at them. Overall, the night mode component 728 attempts to adjust the brightness of the AR content to match the environment of the real-world scene 718.

[0092] In some examples, the AR content exceeds the ambient brightness, allowing it to remain visible during adaptation. The night mode component 728 analyzes the real-world scene 718 to avoid overlaying content onto environmental objects, which would increase overall brightness.

[0093] In some examples, in the case of gaze tracking, the night mode component 728 uses at least one eye-tracking sensor 730 to perform foveated rendering for cone cells and peripheral rendering for rod cells. The night mode component 728 uses RGB for cone cells and 498 nm light for rod cells. Peripheral rendering has a lower spatial resolution (1 / 4 to 1 / 8 of the fovea) and a lower frame rate (1 / 2 to 1 / 3 of the fovea). The resolution further decreases towards the periphery as the rod cell density decreases. The night mode component 728 desaturates peripheral content while maintaining contrast.

[0094] In some examples, without gaze tracking, the Night Mode component 728 offers a variety of rendering modes, such as, but not limited to, all red above 640 nm, all light at 498 nm, or dark RGB. The choice depends on the application and whether night vision capabilities that damage rod cells are acceptable.

[0095] In operation 802, the night mode component 728 uses at least one camera device 734 of the XR system 706 to capture image data 732. For example, the night mode component 728 captures image data 732 of a real-world scene by controlling at least one camera device 734 of the XR system 706. The night mode component 728 can directly access the camera device hardware and capture image frames. Alternatively, the night mode component 728 can call operating system APIs and libraries to control the camera device 734 and obtain image data 732. The night mode component 728 can also preprocess the image data by performing operations such as image enhancement, compression, resizing, and format conversion, utilizing visual libraries and frames.

[0096] In operation 804, the night mode component 728 estimates the dark adaptation level based on image data. For example, the night mode component 728 estimates the user's eye dark adaptation level based on analysis of image data captured in operation 802. In some examples, the night mode component 728 processes image data to determine ambient light levels and other environmental conditions. In some examples, the night mode component 728 utilizes computer vision techniques and eye-tracking data 726 captured using at least one eye-tracking sensor 730 to detect the user's pupil size from the eye-tracking data 726, as shown in reference. Figure 12 A more comprehensive description. In some examples, the night mode component 728 additionally utilizes physiological data collected from the user 704, such as references. Figure 9 A more comprehensive description. In some examples, the night mode component 728 uses user stimuli and recorded user responses to estimate the level of dark adaptation, as referenced. Figure 10 A more comprehensive description.

[0097] At operation 806, the night mode component 728 selects a night mode rendering configuration based on the estimated dark adaptation level. For example, the night mode component 728 selects an appropriate night mode rendering configuration for the XR display based on the estimated dark adaptation level from operation 804. In some examples, the night mode component 728 selects an eye-tracking-based night mode rendering method 1300 (see [reference]). Figure 13 (For a more comprehensive description), to select rendering settings such as resolution, frame rate, and color mode for the concave and peripheral regions. In some examples, the night mode component 728 selects a non-eye-tracking-based rendering method 1400 (see [reference]). Figure 14 (For a more comprehensive description), allowing selection between different rendering modes (such as full red, 498 nm, or dark RGB).

[0098] At operation 808, the night mode component 728 uses the selected rendering configuration to generate the XR user interface 716 of the XR application 702. For example, the night mode component 728 actively generates the XR user interface 716 of the XR application 702 using the night mode rendering configuration selected in operation 806. In some examples, the night mode component 728 utilizes graphics APIs and rendering engines to generate the visual content of the XR user interface 716 based on the selected configuration. The night mode component 728 utilizes, for example... Figure 13 Eye-tracking based rendering methods and Figure 14 The non-eye-tracking rendering method in the rendering process is used to render the XR user interface 716. In some examples, the night mode component 728 performs foveated rendering for the gaze area and peripheral rendering for the outer area. In some examples, the night mode component 728 selects between different color and brightness modes based on the application before performing rendering.

[0099] In operation 810, the night mode component 728 displays the XR user interface 716 on the display of the XR system 706. For example, the night mode component 728 first generates XR user interface graphics data 708 representing the visual content of the XR user interface 716. The night mode component 728 uses the GPU API and graphics engine to render the XR user interface graphics data 708 based on the XR user interface 716 generated in operation 808. Next, the night mode component 728 sends the XR user interface graphics data 708 to the image display driver 710 of the optical engine 714. The image display driver 710 then generates a display control signal 712 to control the optical component 700. Under the guidance of the display control signal 712, the optical component 700 displays the visual content of the XR user interface 716, including one or more virtual objects 720 of the XR user interface 716.

[0100] Figure 9 It shows the result of Figure 7 The night mode component 728 uses an example physiological dark adaptation estimation method 900. Although the example physiological dark adaptation estimation method 900 depicts a specific sequence of operations, this sequence can be changed without departing from the scope of this disclosure. For example, some of the depicted operations can be performed in parallel or in a different order that does not substantially affect the functionality of the physiological dark adaptation estimation method 900. In other examples, different components of the example device or system implementing the physiological dark adaptation estimation method 900 can perform their functions substantially simultaneously or in a specific order.

[0101] In operation 902, the night mode component 728 is... Figure 7 The XR display device in the XR system 706 flickers a weak light signal. For example, the night mode component 728... Figure 7 The XR display device of the XR system 706 shown actively flashes a low-light signal. A night mode component 728 controls the optical components and display driver of the XR system 706 to modulate the brightness of the XR display using the low-light signal through a flashing pattern. In some examples, the night mode component 728 generates multiple low-light signals flashing at one or more frequencies to determine at what light intensity and flashing frequency the user 704 begins to see the display of the low-light signal.

[0102] In operation 904, the night mode component 728 uses the XR system 706 to measure the user's visual cortex electroencephalogram / event-related potential (EEG / ERP) signal. The EEG / ERP signal indicates whether photoreceptors have detected a dim light signal. For example, the night mode component 728 actively measures the user's visual cortex EEG / ERP signal using the EEG sensor (not shown) of the XR system 706. After the dim light signal is flashed in operation 902, the night mode component 728 uses the EEG sensor to detect electrical activity in the user's visual cortex region. The EEG / ERP signal indicates whether the user's photoreceptors have detected the dim light signal flashed by the night mode component 728. By actively measuring this brain activity signal, the night mode component 728 can determine whether the user 704 has perceived the dim light signal.

[0103] In operation 906, the night mode component 728 method uses EEG / ERP signals to estimate the dark adaptation level. For example, the night mode component 728 analyzes EEG / ERP signal data to determine whether the user's photoreceptors have detected a flickering weak light signal. Based on this analysis, the night mode component 728 estimates the user's current dark adaptation level. For example, if the EEG / ERP signal indicates that a weak light flicker has been detected, the night mode component 728 estimates that the eye has achieved a target level of dark adaptation. Conversely, if the signal indicates that no flicker has been perceived, the estimated dark adaptation level is low.

[0104] Figure 10 It shows the result of Figure 7 The night mode component 728 uses an example eye-tracking-based adaptive estimation method 1000. Although the example eye-tracking-based adaptive estimation method 1000 depicts a specific sequence of operations, this sequence can be changed without departing from the scope of this disclosure. For example, some of the depicted operations may be performed in parallel or in a different order that does not substantially affect the functionality of the eye-tracking-based adaptive estimation method 1000. In other examples, different components of the example device or system implementing the eye-tracking-based adaptive estimation method 1000 may perform functions substantially simultaneously or in a specific order.

[0105] In operation 1002, Figure 7 Night mode component 728 Figure 7The XR display of the XR system 706 flashes a low-light signal. For example, the night mode component 728 controls the image display driver 710 and the optical components 700 of the XR system 706 to modulate the brightness of the XR display using a low-light signal through a flashing pattern. This flashing of the low-light signal is performed by the night mode component 728 as part of estimating the user's dark adaptation level based on eye tracking. By flashing the low-light signal in the XR display, the night mode component 728 provides visual stimulation to the user's eyes to determine whether it can be perceived.

[0106] In operation 1004, the night mode component 728 uses at least one eye-tracking sensor 730 to capture eye-tracking data 726 of at least one eye 724 of the user 704. For example, the eye-tracking sensor 730 actively tracks the gaze direction and movement of the user's eye 724 to generate eye-tracking data 726. This is done in response to a dim light signal flickering in the XR display by the night mode component 728 in operation 1002. By tracking the eye 724, the eye-tracking sensor 730 can detect whether the user's gaze changes in response to the perceived flickering of the dim light signal. The eye-tracking data 726 generated by the eye-tracking sensor 730 provides detection of whether the user 704 has noticed the dim light flickering stimulus.

[0107] In operation 1006, the night mode component 728 analyzes eye-tracking data 726 to detect whether at least one eye 724 of the user 704 has noticed the flickering dim light signal. For example, the night mode component 728 examines gaze patterns and movements contained in the eye-tracking data 726. It looks for changes that indicate a response of the user's eye 724 to the flickering stimulus provided in operation 1002. For example, a rapid shift in gaze direction could indicate that the user 704 has perceived the flicker. By thoroughly analyzing the eye-tracking data 726, the night mode component 728 can determine whether the user's eyes have noticed the dim light flicker.

[0108] In operation 1008, the night mode component 728 estimates the dark adaptation level of at least one eye 724 of the user 704 based on whether flicker is noticed. For example, the night mode component 728 makes this estimate based on an analysis completed in operation 1006 of whether the user's eye 724 notices a dim light flickering stimulus. If the analysis determines that the eye 724 notices the flicker, the night mode component 728 estimates that the eye 724 has achieved a high level of dark adaptation. This is because the eye's photoreceptors are capable of detecting dim light signals. Conversely, if the analysis in 1006 finds that flicker is not noticed, the night mode component 728 estimates that the eye 724 has a low level of dark adaptation. By utilizing eye-tracking analysis, the night mode component 728 determines the current dark adaptation level of the user's eye.

[0109] Figure 11 It shows the result of Figure 7 The night mode component 728 uses an example behavior-based dark adaptation estimation method 1100. Although the example behavior-based dark adaptation estimation method 1100 depicts a specific sequence of operations, this sequence can be changed without departing from the scope of this disclosure. For example, some of the depicted operations may be performed in parallel or in different sequences that do not substantially affect the functionality of the behavior-based dark adaptation estimation method 1100. In other examples, different components of the example device or system implementing the behavior-based dark adaptation estimation method 1100 may perform functions substantially simultaneously or in a specific order.

[0110] In operation 1102, the night mode component 728 is in Figure 7 The XR system 706 uses low-intensity, low-contrast text to display interactive instructions in its display. A night mode component 728 displays the interactive instructions at varying contrast levels. For example, the night mode component 728 controls the display driver and optics of the XR system 706 to render text prompts guiding the user to perform specific actions. The text is displayed at different contrast levels, ranging from high to low contrast. As the user's eyes gradually adapt to the darkness over time, more of the previously invisible low-contrast text becomes visible. By utilizing this effect and displaying instructions with progressively lower contrast, the night mode component 728 can determine when certain contrast levels become detectable. This allows for estimation of the current level of dark adaptation. The use of interactive instructions enables behavior-based methods to estimate the level of dark adaptation. In some examples, the night mode component 728 instructs the user 704 to make a specific gesture with their hand, such as, but not limited to, giving a thumbs-up gesture if they can see the instruction.

[0111] In operation 1104, the night mode component 728 detects that the user has executed a displayed interactive instruction. For example, as the user's eyes darken over time and the low-contrast text prompts become more visible, the user will be able to follow the instructions and perform the specified actions. The night mode component 728 utilizes the sensors and input devices of the XR system 706 to detect the user's movements. For example, a camera device can visually detect posture, while an inertial sensor can detect head movement. By utilizing various sensors and inputs, the night mode component 728 is able to detect when the user has executed the prompted instruction, indicating that the prompted instruction has become visible to the user 704. Detecting which instructions the user can perceive allows for the estimation of the level of dark adaptation.

[0112] In operation 1106, the night mode component 728 estimates the user's eye dark adaptation level based on the contrast level of the displayed instructions that the user can detect.

[0113] Figure 12It shows the result of Figure 7 The night mode component 728 uses an example dark adaptation estimation method 1200 based on a machine learning model. Although the example dark adaptation estimation method 1200 based on a machine learning model depicts a specific order of operations, this order can be changed without departing from the scope of this disclosure. For example, some of the depicted operations may be performed in parallel or in a different order that does not substantially affect the functionality of the dark adaptation estimation method 1200 based on a machine learning model. In other examples, different components of an example device or system implementing the dark adaptation estimation method 1200 based on a machine learning model may perform functions substantially simultaneously or in a specific order.

[0114] In operation 1202, the night mode component 728 senses the ambient light level of the surrounding environment. For example, the night mode component 728 uses a light sensor (such as a photodiode, photoresistor, or one or more camera pixels) to measure the intensity of ambient light around the user. This provides an objective estimate of the overall lighting conditions to which the user's eyes have adapted. The sensor converts photons into electrical signals representing the brightness of the surrounding environment. Lower light levels indicate dark-vision or meso-vision adaptation. By quantifying ambient light, the night mode component 728 collects useful data to combine with other physiological measurements to estimate the level of dark adaptation.

[0115] In operation 1204, the night mode component 728 measures the pupil size of the user's eye. For example, the night mode component 728 utilizes at least one eye-tracking sensor 730 to determine the pupil size of the user's eye. As the eye adapts to lower light levels, the pupil dilates to allow more light to reach the retina. By measuring the pupil diameter, the night mode component 728 can estimate the level of adaptation. A larger pupil size is associated with a more dilated eye adapting to lower light. Along with the ambient light level, pupil size provides another data point for estimating the level of dark adaptation using a machine learning model.

[0116] In operation 1206, the night mode component 728 inputs ambient light level and pupil size data over a period of time into a machine learning model. For example, the night mode component 728 feeds light level data from operation 1202 and pupil size data from operation 1204 into the machine learning model, for example... Figure 7 The dark adaptation model 736. The machine learning model has previously been trained on labeled pairs of data regarding light level, pupil size, and corresponding dark adaptation levels. By inputting the current measurement as a feature vector, the model can estimate the user's current dark adaptation level based on the correlations it has learned. For example, low light levels and large pupils over a period of time (e.g., but not limited to 10 minutes) will map to a high dark adaptation level. The machine learning approach allows for accurate inference of the nonlinear relationship between the signal and the adaptation level.

[0117] In some examples, the time period is in the range of 25 to 35 minutes.

[0118] In some examples, the night mode component 728 collects real-time dark adaptation level data and uses the dark adaptation level data to continuously retrain the dark adaptation model 736 in real time. For example, the night mode component 728 uses methods (e.g., but not limited to) Figure 9 Physiological dark adaptation estimation method 900 Figure 10 An eye-tracking-based adaptation estimation method (e.g., 1000) was used to collect dark adaptation level data.

[0119] At operation 1208, the night mode component 728 estimates the user's eye dark adaptation level based on the output of a machine learning model. For example, after inputting light level and pupil size data into a trained machine learning model in operation 1206, the model outputs a predicted dark adaptation level. This can be a continuous value representing the estimated adaptation level, or a discrete classification (e.g., "low," "medium," or "high"). The night mode component 728 takes this model output and uses it as a current estimate of the user's dark adaptation level. This data can then be used to optimize rendering and display configurations to match what the user's eyes can perceive in their adapted state. The machine learning method provides a data-driven estimate of the dark adaptation level.

[0120] Figure 13 It shows the result of Figure 7 The night mode component 728 uses an example eye-tracking-based night mode rendering method 1300. Although the example eye-tracking-based night mode rendering method 1300 depicts a specific order of operations, this order can be changed without departing from the scope of this disclosure. For example, some of the depicted operations may be performed in parallel or in a different order that does not substantially affect the functionality of the eye-tracking-based night mode rendering method 1300. In other examples, different components of the example device or system implementing the eye-tracking-based night mode rendering method 1300 may perform their functions substantially simultaneously or in a specific order.

[0121] In operation 1302, the night mode component 728 captures eye-tracking data. For example, the night mode component 728 utilizes at least one eye-tracking sensor 730 to track the user's gaze direction and focus. Figure 7The eye-tracking data 726 can include the coordinates of the user's gaze point relative to the display, as well as measurements of pupil size and eye movement. By continuously capturing this data in real time, the night mode component 728 can identify which area of ​​the display the user is viewing—the foveal region or the peripheral region. The eye-tracking data allows for optimized rendering based on which area the cone and rod cells are viewing. This enables night vision capability to be maintained in the peripheral region while displaying detailed content only in the foveal gaze area.

[0122] In operation 1304, the night mode component 728 performs foveal-point rendering for the foveal gaze region identified from eye-tracking data. For example, using the gaze direction and coordinates obtained from the eye-tracking sensor, the night mode component 728 identifies the user's foveal gaze region. This is the central area of ​​focus. The night mode component 728 then performs foveal-point rendering, rendering the content in the foveal gaze region at full resolution while reducing the resolution of the content in the peripheral regions outside the foveal region. This allows concentrated display processing power to render high-resolution content only where the user is directly viewing. Content outside the foveal gaze region can be rendered at a lower resolution to save processing power because peripheral rod and cone cells cannot perceive fine details. Eye-tracking data enables the isolation of the foveal region for high-fidelity rendering.

[0123] In operation 1306, the night mode component 728 uses RGB colors for the content displayed in the foveal fixation area. For example, based on the foveal region identified from eye-tracking data in operation 1304, the night mode component 728 renders content in full RGB colors in that region. This provides the highest color fidelity, allowing the cone cells, concentrated in the fovea, to perceive the full color spectrum. The night mode component 728 leverages the fact that cone cells are concentrated in the center of the fovea while rod cells dominate in the periphery. By limiting RGB colors only to the foveal fixation area, the night mode component 728 can display colored content to the cone cells while avoiding illuminating the peripheral rod cells with colors that might disrupt their dark adaptation. This approach optimizes the information content for the cone cells while protecting the night vision capabilities of the rod cells.

[0124] In operation 1308, the night mode component 728 uses light with a wavelength of 498 nm to display content in the peripheral region outside the foveal fixation area. For example, based on the peripheral region identified from eye-tracking data, the night mode component 728 uses light centered at a wavelength of 498 nm to display content in that region. This wavelength corresponds to the peak photosensitivity of the rod cell photoreceptors concentrated in the peripheral region. Using 498 nm light allows the display to utilize the increased sensitivity of the rod cells at this wavelength, meaning less light energy is required compared to other wavelengths. This helps save power.

[0125] In operation 1310, the night mode component 728 renders the peripheral region at a lower spatial resolution compared to the foveal region. For example, based on the peripheral region identified from eye-tracking data, the night mode component 728 displays content in that region at a reduced spatial resolution compared to the foveal gaze region. This is because the visual acuity provided by the peripheral rod cell photoreceptors is lower than that of the cone cells concentrated in the fovea. By reducing the spatial resolution of the peripheral content, less processing power is required while still providing an image that the rod cells can perceive. The degree of resolution reduction can be customized based on the estimated level of scotopic adaptation. In some examples, the peripheral resolution can be 1 / 4 to 1 / 8 of the foveal resolution. And within the periphery, the resolution can even be further reduced towards the outer boundary of the decrease in rod cell density. Therefore, this operation allows for efficient and perceptually optimized rendering by utilizing the difference between foveal and peripheral vision.

[0126] In operation 1312, the night mode component 728 renders the peripheral region at a lower temporal resolution compared to the foveal region. For example, the night mode component 728 renders the peripheral region at a lower temporal resolution compared to the foveal region. Based on the peripheral region identified from eye-tracking data, the night mode component 728 displays content in that region at a reduced frame rate or refresh rate compared to the foveal fixation region. This utilizes the difference in temporal response between rod cell photoreceptors and cone cell photoreceptors. Rod cells respond more slowly to changes in light intensity. Therefore, in the peripheral region, the night mode component 728 can render content at a lower frame rate (e.g., 1 / 2 to 1 / 3 of the foveal rate) while remaining perceptible to the rod cells. This saves processing power. The degree of reduction in frame rate can be customized based on the estimated level of dark vision adaptation. Therefore, this operation optimizes the refresh rate based on the ability of peripheral night vision, thereby reducing power consumption.

[0127] In operation 1314, the night mode component 728 desaturates the content in the peripheral region while preserving contrast. For example, for content rendered in the peripheral region outside the foveal gaze area, the night mode component 728 converts RGB colors to grayscale. The night mode component 728 applies a desaturation algorithm that maintains or enhances the relative contrast between different grayscale levels. Therefore, while color information is removed, the contrast and dynamic range of the original content are preserved as much as possible. This allows maximum information to be delivered to the rod cells within their perceptual limits while avoiding overstimulation that could interfere with night vision adaptation. Thus, desaturation balances information preservation and night vision protection.

[0128] Figure 14 It shows the result of Figure 7The night mode component 728 uses an example of a non-eye-tracking rendering method 1400. Although the example of the non-eye-tracking rendering method 1400 depicts a specific order of operations, this order can be changed without departing from the scope of this disclosure. For example, some of the depicted operations may be performed in parallel or in a different order that does not substantially affect the functionality of the non-eye-tracking rendering method 1400. In other examples, different components of the example device or system implementing the non-eye-tracking rendering method 1400 may perform functions substantially simultaneously or in a specific order.

[0129] In operation 1402, the night mode component 728 provides at least one rendering mode, such as, but not limited to: 1) a mode using wavelengths above 640 nm; 2) a mode using light with a wavelength of 498 nm; and 3) a dark RGB color mode. For example, the night mode component 728 makes these three rendering modes available for selection based on the application and whether night vision that damages rod cells is acceptable.

[0130] Using only wavelengths above 640 nm avoids stimulating rod cells, thus fully preserving dark adaptation. This mode may be optimal when night vision must be protected.

[0131] The 498 nm mode utilizes the wavelength most sensitive to rod cells, allowing some peripheral content to simultaneously reduce power. However, the dark adaptation of rod cells may be partially disrupted.

[0132] The dark RGB mode provides the foveal content visible to the cone cells, but the rod cells can also be illuminated, thus reducing their night vision capabilities. This mode conveys the most information but has the greatest impact on dark adaptation.

[0133] Therefore, the Night Mode component 728 provides this range of options that balance information, power, and night vision capabilities based on context, usage, and user preferences.

[0134] In operation 1404, the night mode component 728 selects the rendering mode based on the application. For example, the selection of night mode rendering can depend on the application context and priority:

[0135] Maintaining full night vision capability is crucial for applications such as military and astronomy, which may require strict use of >640nm mode to avoid any rod cell stimulation.

[0136] Applications that enhance low-light real-world environments (such as nighttime hiking) can select the 498 nm mode to overlay some surrounding content without completely disrupting dark adaptation.

[0137] Applications that display detailed color content (such as games and videos) can utilize the Dark RGB mode to provide rich visual effects, although it may affect the night vision capabilities of rod cells.

[0138] Safety-oriented applications (such as night driving) can choose 498 nm or >640 nm modes to avoid excessive distraction from surrounding content.

[0139] Power-constrained applications (such as those with low battery) can use the 498 nm mode to save energy.

[0140] Applications intended to briefly disrupt night vision (such as notifications) can briefly use dark RGB mode.

[0141] In operation 1406, for night modes 1 and 2, the night mode component 728 desaturates the AR content while maintaining contrast in either a 640 nm or 498 nm wavelength mode. For example, in the mode using only wavelengths above 640 nm, since this red light is invisible to rod cells, the content can be displayed in full color to stimulate the L-cones in the fovea. However, to save power, the night mode component 728 converts this colored foveal content to grayscale, removing the hue.

[0142] Similarly, for the 498 nm mode, the content is converted to grayscale to avoid excessive rod cell stimulation that could disrupt dark adaptation. However, the night mode component 728 does not simply convert to grayscale (which would reduce contrast), but instead applies a desaturation algorithm. This preserves the relative contrast between different grayscale levels, maintaining the dynamic range and visual distinctiveness of the original content as much as possible. For example, the image is converted to the CIELAB color space, where the L channel represents lightness, and the A and B channels represent color-complementary values. The image is then desaturated by setting the A and B channels to zero while preserving the L channel. This removes color information while maintaining lightness contrast.

[0143] In some examples, an alternative color model is used, such as, but not limited to, YUV, where Y contains luminance or lightness information. The U and V chroma channels are discarded for desaturation, while the luminance channel Y is retained to maintain contrast.

[0144] In some examples, color histogram matching is used to desaturate a reference grayscale image with the desired contrast and lighting characteristics.

[0145] Therefore, for both of these peripheral-sparing night modes, the night mode component 728 utilizes desaturation to balance the impact on night vision and the preservation of information.

[0146] In operation 1408, the night mode component 728 adjusts the brightness of the AR content to be similar to that of objects in the environment. For example, the night mode component 728 analyzes image data captured by the camera device to estimate the ambient brightness level of the physical environment. Based on this analysis, the night mode component 728 configures the XR display brightness to match the scene.

[0147] For example, if the environment is dark, the virtual content is rendered darker to avoid large differences that could interfere with dark adaptation. In some cases, AR brightness can be slightly higher than the environment to remain visible while the user's eyes are adapting. But generally, the goal is to make the virtual brightness as consistent as possible with the real brightness.

[0148] This synchronization of real and virtual brightness helps avoid overstimulating rod or cone cells. It also provides a more natural and seamless viewing experience during night mode, as AR content blends into the environment. This operation aims to balance AR visibility, adaptive disruption, and realism by matching the brightness of the real-world scene as closely as possible.

[0149] In operation 1410, the night mode component 728 places AR content onto the darker areas of the real-world scene 718 environment. For example, the night mode component 728 performs analysis of captured image data to detect darker and brighter areas in the environment. For example, it can detect surfaces, objects, or portions of objects with low measured brightness levels.

[0150] When determining where to overlay AR content, the Night Mode component 728 prioritizes detected dark areas. For example, it places virtual text or graphics atop a dark wall, or a virtual character in a dimly lit corner. This avoids exacerbating the overall brightness of the area by overlaying virtual content. If AR content is simply overlaid arbitrarily, it can significantly increase local brightness if placed on an already bright surface or object. This overstimulation can interfere with dark adaptation. By intelligently prioritizing the placement of AR content in dark areas, overall brightness is more likely to remain close to the original real-world environment. In some examples, the determination of where to place XR content is based on what the user can see at their currently estimated dark adaptation level, rather than what the camera can detect (e.g., dark objects invisible to the user's eye are identified as part of the background, even if these dark objects can be detected by the camera). This helps maintain a natural scene appearance during Night Mode while supporting the user's transition to either scotopic or mesoscopic vision.

[0151] Machine Learning Pipeline

[0152] Figure 16 This is a flowchart depicting a machine learning pipeline 1600 based on some examples. Machine learning pipeline 1600 can be used to generate trained machine learning models 1602 (e.g., ...). Figure 7 The dark adaptation model 736 is used to perform operations associated with providing a night mode.

[0153] Overview

[0154] In a broad sense, machine learning can involve using computer algorithms to automatically learn patterns and relationships in data, potentially without explicit programming. Machine learning algorithms can be divided into three main categories: supervised learning, unsupervised learning, and reinforcement learning.

[0155] Supervised learning involves training a model using labeled data to predict outputs for new, unseen inputs. Examples of supervised learning algorithms include linear regression, decision trees, and neural networks.

[0156] Unsupervised learning involves training a model on unlabeled data to find hidden patterns and relationships within the data. Examples of unsupervised learning algorithms include clustering, principal component analysis, and generative models such as autoencoders.

[0157] Reinforcement learning involves training a model to make decisions in dynamic environments by receiving feedback in the form of rewards or penalties. Examples of reinforcement learning algorithms include Q-learning and policy gradient methods.

[0158] Examples of specific machine learning algorithms that can be deployed include logistic regression, a type of supervised learning algorithm for binary classification tasks. Logistic regression models the probability of a binary response variable based on one or more predictor variables. Another example type of machine learning algorithm is Naive Bayes, another supervised learning algorithm for classification tasks. Naive Bayes is based on Bayes' theorem and assumes that the predictor variables are independent of each other. Random forests are another type of supervised learning algorithm used for classification, regression, and other tasks. Random forests build an ensemble of decision trees and combine their outputs to make predictions. Further examples include neural networks, which consist of interconnected layers of nodes (or neurons) that process information based on input data and make predictions. Matrix factorization is another type of machine learning algorithm used for recommendation systems and other tasks. Matrix factorization decomposes a matrix into two or more matrices to discover hidden patterns or relationships in the data. Support Vector Machines (SVMs) are a type of supervised learning algorithm used for classification, regression, and other tasks. SVMs find hyperplanes that separate different classes in the data. Other types of machine learning algorithms include decision trees, k-nearest neighbors, clustering algorithms, and deep learning algorithms such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and Transformer models. The choice of algorithm depends on the nature of the data, the complexity of the problem, and the performance requirements of the application.

[0159] The performance of a machine learning model is typically evaluated on a separate test dataset that was not used during training to ensure that the model can generalize to new, unseen data.

[0160] While this article discusses several specific examples of machine learning algorithms, the principles discussed can be applied to other machine learning algorithms as well. Deep learning algorithms, such as convolutional neural networks, recurrent neural networks, and transformers, as well as more traditional machine learning algorithms, such as decision trees, random forests, and gradient boosting, can be used in a variety of machine learning applications.

[0161] In machine learning, there are three types of example problems: classification, regression, and generation. Classification problems (also known as categorization problems) aim to classify an item into one of several category values ​​(e.g., is the object an apple or an orange?). Regression algorithms aim to quantify some items (e.g., by providing values ​​as real numbers). Generation algorithms aim to produce new examples similar to those provided for training. For example, a text generation algorithm is trained on many text documents and configured to generate new, coherent text with similar statistical properties to the training data.

[0162] Training phase

[0163] Generating a trained machine learning model 1602 may include multiple stages forming part of a machine learning pipeline 1600, said multiple stages including, for example... Figure 15 The following stages are shown:

[0164] Data Collection and Preprocessing 1502: This stage may include acquiring and cleaning data to ensure it is suitable for use in machine learning models. This stage may also include removing duplicates, handling missing values, and converting the data to a suitable format.

[0165] Feature engineering 1504: This stage may include selecting and transforming training data 1606 to create features useful for predicting the target variable. Feature engineering may include (1) receiving features 1608 (e.g., as structured or labeled data in supervised learning) and / or (2) identifying features 1608 in training data 1606 (e.g., unstructured or unlabeled data for unsupervised learning).

[0166] Model selection and training 1506: This stage may include selecting an appropriate machine learning algorithm and training it on preprocessed data. This stage may also involve splitting the data into training and test sets, using cross-validation to evaluate the model, and tuning hyperparameters to improve performance.

[0167] Model Evaluation 1508: This phase may include evaluating the performance of the trained model (e.g., a trained machine learning model 1602) on a separate test dataset. This phase can help determine whether the model is overfitting or underfitting, and whether the model is suitable for deployment.

[0168] Prediction 1510: This stage involves using a trained model (e.g., a trained machine learning model 1602) to generate predictions for new, unseen data.

[0169] Validation, refinement, or retraining 1512: This stage may include updating the model based on feedback generated from the prediction stage, such as new data or user feedback.

[0170] Deployment 1514: This phase may include integrating the trained model (e.g., a trained machine learning model 1602) into a wider system or application (e.g., a web service, mobile application, or IoT device). This phase may involve setting up an API, building a user interface, and ensuring that the model is scalable and can handle large amounts of data.

[0171] Figure 16Further details of two example phases are shown: the training phase 1604 (e.g., part of model selection and training 1506) and the prediction phase 1610 (part of prediction 1510). Prior to the training phase 1604, feature engineering 1504 is used to identify features 1608. This can include identifying informative, distinctive, and independent features for effectively operating the trained machine learning model 1602 in pattern recognition, classification, and regression. In some examples, the training data 1606 includes labeled data known for the pre-identified features 1608 and one or more outcomes. Each of the features 1608 can be a variable or attribute, such as a measurable characteristic of a process, item, system, or phenomenon represented by the dataset (e.g., training data 1606). By way of example only, features 1608 can also be of different types, such as numerical features, strings, and graphs, and can include one or more of content 1612, concepts 1614, attributes 1616, historical data 1618, and / or user data 1620.

[0172] In the training phase 1604, the machine learning pipeline 1600 uses the training data 1606 to find the correlations between features 1608 that affect the prediction results or the prediction / inference data 1622.

[0173] Using training data 1606 and identified features 1608, the trained machine learning model 1602 is trained during the training phase 1604 of the machine learning program training 1624. The machine learning program training 1624 evaluates the values ​​of features 1608 as they relate to the training data 1606. The result of the training is the trained machine learning model 1602 (e.g., a trained or learned model).

[0174] Furthermore, the training phase 1604 may involve machine learning, where the training data 1606 is structured (e.g., labeled during preprocessing). The trained machine learning model 1602 implements a neural network 1626 capable of performing operations such as classification and clustering. In other examples, the training phase 1604 may involve deep learning, where the training data 1606 is unstructured, and the trained machine learning model 1602 implements a deep neural network 1626 capable of performing both feature extraction and classification / clustering operations.

[0175] In some examples, neural network 1626 may be generated during training phase 1604 and implemented within a trained machine learning model 1602. Neural network 1626 includes a hierarchical (e.g., layered) organization of neurons, where each layer consists of multiple neurons or nodes. Neurons in the input layer receive input data, while neurons in the output layer produce the network's final output. Between the input and output layers, there may be one or more hidden layers, each consisting of multiple neurons.

[0176] In a neural network 1626, each neuron can operationally compute a function, such as an activation function, which takes as input a weighted sum of the outputs of neurons in the previous layer and a bias term. The output of this function is then passed as input to neurons in the next layer. If the output of the activation function exceeds a certain threshold, the output is passed from that neuron (e.g., a firing neuron) to connected neurons (e.g., receiving neurons) in the next layer. Connections between neurons have associated weights that define the influence of the input from the firing neuron to the receiving neuron. During the training phase, these weights are adjusted by a learning algorithm to optimize the network's performance. Different types of neural networks can use different activation functions and learning algorithms, thus affecting their performance on different tasks. The hierarchical organization of neurons and the use of activation functions and weights enable neural networks to model complex relationships between inputs and outputs and generalize to new inputs not seen during training.

[0177] In some examples, and by way of example only, neural network 1626 can also be one of several different types of neural networks, such as a single-layer feedforward network, a multilayer perceptron (MLP), an artificial neural network (ANN), a recurrent neural network (RNN), a long short-term memory network (LSTM), a bidirectional neural network, a symmetric connection neural network, a deep belief network (DBN), a convolutional neural network (CNN), a generative adversarial network (GAN), an autoencoder neural network (AE), a restricted Boltzmann machine (RBM), a Hopfield network, a self-organizing map (SOM), a radial basis function network (RBFN), a spiking neural network (SNN), a liquid state machine (LSM), an echo state network (ESN), a neural Turing machine (NTM), or a transformer network.

[0178] In addition to the training phase 1604, a validation phase can be performed on a separate dataset called the validation dataset. The validation dataset is used to tune the model's hyperparameters, such as the learning rate and regularization parameters. Tuning these hyperparameters improves the model's performance on the validation dataset.

[0179] Once the model is fully trained and validated, the testing phase allows it to be tested on a new dataset. The test dataset is used to evaluate the model's performance and ensure that it has not overfitted the training data.

[0180] In the prediction phase 1610, the trained machine learning model 1602 uses features 1608 to analyze the query data 1628 to generate inferences, results, or predictions, as examples of the prediction / inference data 1622. For example, during the prediction phase 1610, the trained machine learning model 1602 generates outputs. In response to receiving the query data 1628, the query data 1628 is provided as input to the trained machine learning model 1602, and the trained machine learning model 1602 generates the prediction / inference data 1622 as output.

[0181] In some examples, the trained machine learning model 1602 can be a generative AI model. Generative AI is a term that can refer to any type of artificial intelligence that can create new content from training data 1606. For example, generative AI can produce text, images, videos, audio, code, or synthetic data that are similar to but not identical to the original data.

[0182] Some techniques that can be used in generative AI are:

[0183] Convolutional Neural Networks (CNNs): CNNs can be used for image recognition and computer vision tasks. For example, a CNN can be designed to extract features from an image by using filters or kernels that scan the input image and highlight important patterns.

[0184] Recurrent Neural Networks (RNNs): For example, RNNs can be used to process sequential data such as speech, text, and time series data. RNNs employ feedback loops, allowing them to capture temporal dependencies and remember past inputs.

[0185] Generative Adversarial Networks (GANs): A GAN can consist of two neural networks: a generator and a discriminator. The generator network attempts to create realistic content that can "fool" the discriminator network, while the discriminator network tries to distinguish between real and fake content. The generator and discriminator networks compete with each other and improve over time.

[0186] Variational Autoencoders (VAEs): VAEs encode input data into a latent space (e.g., a compressed representation) and then decode it back to output data. The latent space can be manipulated to generate new variations of the output data. VAEs can use self-attention mechanisms to process input data, allowing them to handle long sequences of text and capture complex dependencies.

[0187] Transformer models: Transformer models use attention mechanisms to learn relationships between different parts of input data (such as words or pixels) and generate output data based on these relationships. Transformer models can handle sequential data such as text or speech, as well as non-sequential data such as images or code.

[0188] In the generative AI example, query data 1628 may include text, audio, images, videos, digital or media content cues, and output prediction / inference data 1622 may include text, images, videos, audio, code or synthetic data.

[0189] In some examples, the night mode component 728 collects real-time dark adaptation level data and uses the dark adaptation level data to continuously retrain the dark adaptation model 736 in real time. For example, the night mode component 728 uses methods (e.g., but not limited to) Figure 9 Physiological dark adaptation estimation method 900 Figure 10 An eye-tracking-based adaptation estimation method (e.g., 1000) was used to collect dark adaptation level data.

[0190] system

[0191] Figure 17 A system 1700 including a head-mounted wearable device 100 is shown according to some examples. Figure 17 This is a high-level functional block diagram of an example head-mounted wearable device 100 that is communicatively coupled to mobile devices 1714 and various server systems 1704 (e.g., interactive server system 1812) via various networks 1810.

[0192] The head-mounted wearable device 100 includes one or more camera devices, each of which may be, for example, one or more camera devices 1708, light emitter 1710, and one or more broadband camera devices 1712.

[0193] Mobile device 1714 connects to head-mounted wearable device 100 using both low-power wireless connection 1716 and high-speed wireless connection 1718. Mobile device 1714 also connects to server system 1704 and network 1706.

[0194] The head-mounted wearable device 100 also includes two image displays in the image display 1720 of the optical components. The two image displays 1720 of the optical components include one image display associated with the left lateral side of the head-mounted wearable device 100 and one image display associated with the right lateral side of the head-mounted wearable device 100. The head-mounted wearable device 100 also includes an image display driver 1722 and a GPU 1724. The image display 1720, image display driver 1722, and GPU 1724 of the optical components constitute the optical engine of the head-mounted wearable device 100. The image display 1720 of the optical components is used to present images and videos to the user of the head-mounted wearable device 100, including images that may include a graphical user interface (GUI).

[0195] The image display driver 1722 commands and controls the image display 1720 of the optical components. The image display driver 1722 can directly deliver image data to the image display 1720 of the optical components for presentation, or it can convert the image data into a signal or data format suitable for delivery to the image display device. For example, the image data can be video data formatted according to compression formats such as H.264 (MPEG-4 Part 10), HEVC, Theora, Dirac, RealVideo RV40, VP8, VP9, ​​etc., and still image data can be formatted according to compression formats such as Portable Network Group (PNG), Joint Photographic Experts Group (JPEG), Tag Image File Format (TIFF), or Exchangeable Image File Format (EXIF), etc.

[0196] The head-wearable device 100 includes a frame and a pole (or temple) extending laterally from the frame. The head-wearable device 100 also includes a user input device 1730 (e.g., a touch sensor or a press button), comprising an input surface on the head-wearable device 100. The user input device 1730 (e.g., a touch sensor or a press button) is used to receive input selections from the user to manipulate a presented image's GUI.

[0197] Figure 17 The components shown for the head-wearable device 100 are located on one or more circuit boards, such as PCBs or flexible PCBs, in the frame or temples. Alternatively or additionally, the depicted components may be located in blocks, frames, hinges, or nose bridges of the head-wearable device 100. The left and right camera devices 1708 may include digital camera elements, such as complementary metal-oxide-semiconductor (CMOS) image sensors, charge-coupled devices, camera lenses, or any other corresponding visible light or light-capturing elements that can be used to capture data, including images of scenes with unknown objects.

[0198] The head-mounted wearable device 100 includes a memory 1702 that stores instructions for performing a subset or all of the functions described herein. The memory 1702 may also include a storage device.

[0199] like Figure 17 As shown, the high-speed circuit system 1728 includes a high-speed processor 1732, a memory 1702, and a high-speed wireless circuit system 1734. In some examples, an image display driver 1722 is coupled to the high-speed circuit system 1728 and operated by the high-speed processor 1732 to drive the left and right image displays of the image display 1720 of the optical components. The high-speed processor 1732 can be any processor capable of managing the high-speed communication and operation of any general-purpose computing system required by the head-worn device 100. The high-speed processor 1732 includes the processing resources required to manage high-speed data transmission over the high-speed wireless connection 1718 to a wireless local area network (WLAN) using the high-speed wireless circuit system 1734. In some examples, the high-speed processor 1732 executes the operating system of the head-worn device 100, such as the LINUX operating system or another such operating system, and this operating system is stored in the memory 1702 for execution. Among other duties, the high-speed processor 1732, which executes the software architecture for the head-worn device 100, manages data transmission with the high-speed wireless circuit system 1734. In some examples, the high-speed wireless circuit system 1734 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, also referred to herein as WiFi. In some examples, the high-speed wireless circuit system 1734 may implement other high-speed communication standards.

[0200] The low-power wireless circuit system 1736 and high-speed wireless circuit system 1734 of the head-worn wearable device 100 may include a short-range transceiver (Bluetooth™) and a wireless wide area network, local area network, or wide area network (WAN) transceiver (e.g., cellular or WiFi). The mobile device 1714 (including transceivers communicating via low-power wireless connection 1716 and high-speed wireless connection 1718) may be implemented using details of the architecture of the head-worn wearable device 100 or other elements of the network 1706.

[0201] Memory 1702 includes any storage device capable of storing various data and applications, including camera data generated by the left and right camera devices 1708, the broadband camera device 1712, and the GPU 1724, as well as images generated by the image display driver 1722 for display on the image display 1720 of the optical components. While memory 1702 is shown as integrated with the high-speed circuitry 1728, in some examples, memory 1702 may be a separate, independent component of the head-mounted wearable device 100. In some such examples, electrical wiring may provide a connection from the GPU 1724 or the low-power processor 1738 to memory 1702 via a chip including a high-speed processor 1732. In some examples, the high-speed processor 1732 may manage addressing of memory 1702 such that the low-power processor 1738 will activate the high-speed processor 1732 whenever a read or write operation involving memory 1702 is required.

[0202] like Figure 17 As shown, the low-power processor 1738 or high-speed processor 1732 of the head-mounted wearable device 100 may be coupled to a camera device (camera device 1708, light emitter 1710 or broadband camera device 1712), an image display driver 1722, a user input device 1730 (e.g., a touch sensor or a press button), and a memory 1702.

[0203] The head-mounted wearable device 100 is connected to a host computer. For example, the head-mounted wearable device 100 is paired with a mobile device 1714 via a high-speed wireless connection 1718 or connected to a server system 1704 via a network 1706. The server system 1704 may be one or more computing devices as part of a service or network computing system, and for example, it includes a processor, memory, and network communication interfaces to communicate with the mobile device 1714 and the head-mounted wearable device 100 via the network 1706.

[0204] Mobile device 1714 includes a processor and a network communication interface coupled to the processor. The network communication interface allows communication via network 1706, low-power wireless connection 1716, or high-speed wireless connection 1718. Mobile device 1714 may also store at least a portion of the instructions for generating dual-channel audio content in the memory of mobile device 1714 to implement the functions described herein.

[0205] The output components of the head-worn wearable device 100 include visual components, such as displays (e.g., LCD, PDP, LED displays, projectors, or waveguides). The image display of the optical components is driven by an image display driver 1722. The output components of the head-worn wearable device 100 also include acoustic components (e.g., speakers), haptic components (e.g., vibration motors), other signal generators, etc. Input components of the head-worn wearable device 100, mobile device 1714, and server system 1704, such as user input device 1730, may include alphanumeric input components (e.g., keyboards, touchscreens configured to receive alphanumeric input, photoelectric keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touchpads, trackballs, joysticks, motion sensors, or other pointing instruments), haptic input components (e.g., physical buttons, touchscreens that provide positioning and force for touch or touch gestures, or other haptic input components), audio input components (e.g., microphones), etc.

[0206] The head-mounted wearable device 100 may also include additional peripheral device elements. Such peripheral device elements may include biometric sensors, additional sensors, or display elements integrated with the head-mounted wearable device 100. For example, peripheral device elements may include any I / O components, including output components, motion components, position components, or any other such elements described herein.

[0207] For example, biometric components include those for detecting expressions (e.g., hand gestures, facial expressions, voice expressions, body posture, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), and identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition). Motion components include accelerometer components (e.g., accelerometers), gravity sensor components, rotation sensor components (e.g., gyroscopes), etc. Position components include positioning sensor components (e.g., GPS receiver components) for generating positioning coordinates, Wi-Fi or Bluetooth™ transceivers for generating positioning system coordinates, altitude sensor components (e.g., altimeters or barometers that detect air pressure, from which altitude can be obtained), orientation sensor components (e.g., magnetometers), etc. Such positioning system coordinates can also be received from mobile device 1714 via low-power wireless connection 1716 and high-speed wireless connection 1718, through low-power wireless circuit system 1736 or high-speed wireless circuit system 1734.

[0208] Networked computing environment

[0209] Figure 18This is a block diagram illustrating an example interactive system 1800 for facilitating interactions on a network, such as exchanging text messages, making text-to-audio and video calls, or playing games. Interactive system 1800 includes one or more XR systems, such as XR computing system 1802, each hosting multiple applications including interactive client 1806 and other applications 1808. Each interactive client 1806 is communicatively coupled to other instances of interactive client 1806 (e.g., hosted on corresponding other computing systems, such as computing system 1804), interactive server system 1812, and third-party server 1814 via one or more communication networks, including network 1810 (e.g., the Internet). Interactive client 1806 can also communicate with locally hosted applications 1808 using an application programming interface (API).

[0210] Each XR system 1802 may include one or more user devices, such as mobile devices 1714, head-mounted wearable devices 100, and computer client devices 1816, which are communicatively connected to exchange data and messages.

[0211] Interactive client 1806 interacts with other interactive clients 1806 and with interactive server system 1812 via network 1810. The data exchanged between interactive clients 1806 (e.g., interaction 1818) and between interactive client 1806 and interactive server system 1812 includes functions (e.g., commands for activating functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0212] Interactive server system 1812 provides server-side functionality to interactive client 1806 via network 1810. While some functions of interactive system 1800 are described herein as being performed by interactive client 1806 or interactive server system 1812, the location of certain functions—whether within interactive client 1806 or interactive server system 1812—may be a design choice. For example, technically it might be preferred that specific technologies and functions are initially deployed within interactive server system 1812, but later migrated to interactive client 1806 of XR system 1802 with sufficient processing power.

[0213] The interactive server system 1812 supports various services and operations provided to the interactive client 1806. Such operations include sending data to and receiving data from the interactive client 1806, and processing data generated by the interactive client 1806. This data may include message content, client device information, geolocation information, media enhancements and overlays, message content persistence conditions, social network information, and live event information. Data exchange within the interactive system 1800 is activated and controlled via functions available through the user interface (UI) of the interactive client 1806.

[0214] The focus now shifts to the interactive server system 1812. The API server 1820 is coupled to the interactive server 1822 and provides it with a programming interface, making the functionality of the interactive server 1822 accessible to the interactive client 1806, other applications 1808, and third-party servers 1814. The interactive server 1822 is communicatively coupled to the database server 1824, thereby facilitating access to the database 1826, which stores data associated with the interactions processed by the interactive server 1822. Similarly, the web server 1828 is coupled to the interactive server 1822 and provides a web-based interface to it. To this end, the web server 1828 handles incoming network requests via HTTP and several other related protocols.

[0215] API server 1820 receives and sends interactive data (e.g., command and message payloads) between interactive server 1822 and XR system 1802 (as well as interactive client 1806 and other applications 1808) and third-party server 1814. Specifically, API server 1820 provides a set of interfaces (e.g., routines and protocols) that interactive client 1806 and other applications 1808 can call or query to activate the functionality of interactive server 1822. API server 1820 exposes various functions supported by interactive server 1822, including account registration; login functionality; sending interactive data from one interactive client 1806 to another interactive client 1806 via interactive server 1822; transferring media files (e.g., images or videos) from interactive client 1806 to interactive server 1822; setting up media data sets (e.g., stories); retrieving the friend list of users of XR system 1802; retrieving information and content; adding and deleting entities (e.g., friends) in an entity graph (e.g., a social graph); locating friends within a social graph; and opening application events (e.g., related to interactive client 1806).

[0216] Interactive server 1822 hosts multiple systems and subsystems, as shown below. Figure 20The following description is provided. Returning to interactive client 1806, the features and functionalities of an external resource (e.g., a linked application 1808 or applet) are available to the user via the interface of interactive client 1806. In this context, "external" refers to the fact that application 1808 or applet is outside of interactive client 1806. While external resources are typically provided by third parties, they can also be provided by the creator or provider of interactive client 1806. Interactive client 1806 receives user selections regarding options for launching or accessing the features of such external resources. External resources can be application 1808 (e.g., a "native app") installed on XR system 1802, or a smaller version (e.g., an "app") of an application hosted on XR system 1802 or located remotely on XR system 1802 (e.g., on a third-party server 1814). A smaller version of an application includes a subset of the features and functionalities of the application (e.g., a full-scale, local version of the application) and is implemented using markup language documentation. In some examples, a smaller version of the application (e.g., a "mini-program") is a web-based markup language version of the application and is embedded in the interactive client 1806. This is in addition to using markup language documents (e.g., ...). In addition to ml files, mini-programs can include scripting languages ​​(e.g., ...). .js files or .json files) and stylesheets (e.g., ...). (SS file).

[0217] In response to receiving a user selection of an option for launching or accessing an external resource, the interactive client 1806 determines whether the selected external resource is a web-based external resource or a locally installed application 1808. In some cases, the application 1808, locally installed on the XR system 1802, can be launched independently of and separately from the interactive client 1806, for example, by selecting the icon corresponding to the application 1808 on the home screen of the XR system 1802. A smaller version of such an application can be launched or accessed via the interactive client 1806, and in some examples, no part of the smaller application can be accessed outside the interactive client 1806, or only a limited portion of the smaller application can be accessed outside the interactive client 1806. The smaller application can be launched by receiving, for example, a markup language document associated with the smaller application from a third-party server 1814 and processing such a document.

[0218] In response to determining that the external resource is a locally installed application 1808, the interactive client 1806 instructs the XR system 1802 to launch the external resource by executing locally stored code corresponding to the external resource. In response to determining that the external resource is a web-based resource, the interactive client 1806 communicates with, for example, a third-party server 1814 to obtain a markup language document corresponding to the selected external resource. The interactive client 1806 then processes the obtained markup language document to present the web-based external resource within the user interface of the interactive client 1806.

[0219] Interactive client 1806 can notify users of XR system 1802 or other users (e.g., "friends") associated with such users of one or more external resources of ongoing activity. For example, interactive client 1806 can provide participants in a conversation (e.g., a chat session) within interactive client 1806 with notifications related to the current or recent use of external resources by one or more members of a group of users. One or more users can be invited to join a valid external resource or to activate a recently used but currently inactive external resource (within the friends group). External resources can provide participants in the conversation, each using their respective interactive client 1806, with the ability to share items, conditions, states, or locations within the external resource with one or more members of a group of users during the chat session. Shared items can be interactive chat cards that chat members can use to interact, for example, to activate the corresponding external resource, view specific information within the external resource, or take chat members to a specific location or state within the external resource. Within a given external resource, response messages can be sent to users on interactive client 1806. External resources can selectively include different media items in the response based on the current context of the external resource.

[0220] Interactive client 1806 can present users with a list of available external resources (e.g., application 1808 or a mini-program) to launch or access a given external resource. This list can be presented as a context-sensitive menu. For example, the icons representing different applications (or mini-programs) of application 1808 (or a mini-program) can change based on how the user launches the menu (e.g., from a conversational interface or a non-conversational interface).

[0221] Data Architecture

[0222] Figure 19 This is a schematic diagram illustrating a data structure 1900 that can be stored in a database 1904 of an interactive server system 1812, according to certain examples. Although the contents of the database 1904 are shown as including multiple tables, it will be appreciated that the data can be stored in other types of data structures (e.g., as an object-oriented database).

[0223] Database 1904 includes message data stored in message table 1906. For any given message, this message data includes at least message sender data, message receiver (or recipient) data, and a payload. See below for reference. Figure 19 Further details are provided regarding information that can be included in the message and is contained within the message data stored in message table 1906.

[0224] Entity table 1908 stores entity data and (for example, links to entity diagram 1910 and profile data 1902). Entities whose records are maintained in entity table 1908 can include individuals, company entities, organizations, objects, locations, events, etc. Regardless of the entity type, any entity whose data is stored in interactive server system 1812 can be an identifiable entity. Each entity is assigned a unique identifier and an entity type identifier (not shown).

[0225] Entity Graph 1910 stores information about relationships and associations between entities. For example, such relationships can be social or professional relationships based on interests or activities (e.g., working in the same company or organization). Some relationships between entities can be unidirectional, such as an individual user subscribing to digital content (e.g., a newspaper or other digital media channel or brand) for a business or publishing user. Other relationships can be bidirectional, such as the "friendship" relationships between various users of Interactive System 1800.

[0226] Certain licenses and relationships can be attached to each relationship, and also to each direction of the relationship. For example, a two-way relationship (e.g., a friend relationship between individual users) may include authorization for the public disclosure of digital content items between the individual users, but certain restrictions or filters may be imposed on the public disclosure of such digital content items (e.g., based on content characteristics, location data, or time of day data). Similarly, a subscription relationship between an individual user and a business user may impose varying degrees of restrictions on the public disclosure of digital content from the business user to the individual user, and may greatly limit or prevent the public disclosure of digital content from the individual user to the business user. As an example of an entity, a specific user may (e.g., through privacy settings) record certain restrictions in the record for that entity within entity table 1908. Such privacy settings may be applied to all types of relationships in the context of interaction system 1800, or selectively applied to only certain types of relationships.

[0227] Profile data 1902 stores various types of profile data about a specific entity. Based on privacy settings specified by the specific entity, profile data 1902 can be selectively used and presented to other users of the interaction system 1800. In the case of an individual, profile data 1902 includes, for example, a username, phone number, address, settings (e.g., notification and privacy settings), and an avatar representation (or a set of such avatar representations) selected by the user. The specific user can then selectively include one or more of these avatar representations within the content of messages transmitted via the interaction system 1800 and on a map interface displayed to other users by the interaction client 1806. The set of avatar representations may include “status avatars,” which present a graphical representation of a status or activity that the user can choose to transmit at a specific time.

[0228] In the case that the entity is a group, in addition to the group name, members and various settings for the associated group (e.g., notifications), the profile data for the group 1902 may similarly include one or more avatar representations associated with the group.

[0229] Database 1904 also stores enhancement data, such as overlays or filters, in enhancement table 1912. Enhancement data is associated with and applied to videos (video data is stored in video table 1914) and images (image data is stored in image table 1916).

[0230] In some examples, filters are displayed as overlays on images or videos during presentation to the message recipient. Filters can be of various types, including user-selected filters from a set of filters presented to the message sender by the interactive client 1806 while the message sender is composing a message. Other types of filters include geolocation filters (also known as geographic filters), which can be presented to the message sender based on geographic location. For example, based on geographic location information determined by the GPS unit of the XR system 1802, a geolocation filter specific to the vicinity or a particular location can be presented by the interactive client 1806 within the user interface.

[0231] Another type of filter is a data filter, which can be selectively presented to the message sender by the interactive client 1806 based on other inputs or information collected by the XR system 1802 during the message creation process. Examples of data filters include the current temperature at a specific location, the current speed of the message sender, the battery life of the XR system 1802, or the current time.

[0232] Other enhancement data that can be stored in image table 1916 includes XR content items (e.g., corresponding to applying an XR experience to a live or previously captured image or video). XR content items can be live special effects and sounds that can be added to an image or video.

[0233] As described above, augmented data includes XR, VR, and XR content items, overlays, image transformations, images, and modifications that can be applied to image data (e.g., video or images). This includes real-time modifications, which modify images using modifications as the device sensors (e.g., one or more camera devices) of the XR system 1802 capture images and then display the images on the screen of the XR system 1802. This also includes modifications to stored content (e.g., video clips that can be modified in a collection or group). For example, in an XR system 1802 that accesses multiple XR content items, a user can use a single video clip with multiple XR content items to see how different XR reality content items will modify the stored clip. Similarly, real-time video capture can be modified to show how the video image currently captured by the sensors of the XR system 1802 will modify the captured data. Such data may simply be displayed on the screen without being stored in memory, or the content captured by the device sensors may be recorded and stored in memory with or without modification (or both). In some systems, the preview feature can simultaneously show how different XR content items will look in different windows on the monitor. For example, this allows multiple windows with different pseudo-random animations to be viewed on the monitor at the same time.

[0234] Therefore, using XR content items' data and various systems, or other such transformation systems that use that data to modify the content, can involve: the detection of objects (e.g., faces, hands, bodies, cats, dogs, surfaces, objects, etc.) in video frames; tracking such objects as they leave, enter, and move around the field of view; and modifying or transforming such objects while tracking them. Different methods can be used to implement such transformations in various examples. Some examples may involve: generating 3D mesh models of one or more objects; and using transformations of the models and animated textures within the video to implement the transformations. In some examples, tracking points on the objects can be used to place images or textures (which can be 2D or 3D) at the tracked locations. In yet another example, neural network analysis of video frames can be used to place images, models, or textures within the content (e.g., images or video frames). Thus, XR content items involve both images, models, and textures used to create transformations within the content, and the additional modeling and analysis information required to implement such transformations using object detection, tracking, and placement.

[0235] Real-time video processing can be performed using any type of video data (e.g., video streams, video files, etc.) stored in the memory of any type of computerized system. For example, a user can load video files and store them in the device's memory, or the device's sensors can be used to generate video streams. Furthermore, computer-animated models can be used to process any object, such as a human face and parts of the human body, animals, or inanimate objects (e.g., chairs, cars, or other objects).

[0236] In some examples, when a specific modification is selected along with the content to be transformed, the element to be transformed is identified by the computing device and then detected and tracked if the element to be transformed exists in a frame of the video. The elements of the object are modified according to the modification request, thereby transforming the frames of the video stream. Different methods can be used to transform the frames of the video stream for different kinds of transformations. For example, for frame transformations that primarily involve changing the form of elements of an object, feature points for each element of the object are calculated (e.g., using an Active Shape Model (ASM) or other known methods). A feature-point-based mesh is then generated for each element of the object. This mesh is used in subsequent stages of tracking the elements of the object in the video stream. During tracking, the mesh for each element is aligned with the position of each element. Additional points are then generated on the mesh.

[0237] In some examples, transforming certain regions of an object using its elements can be performed by computing feature points for each element of the object and generating a mesh based on those calculated feature points. Points are generated on the mesh, and then various regions are generated based on these points. The elements of the object are then tracked by aligning the regions of each element with the positions of at least one of the elements, and the properties of the regions can be modified based on modification requests, thereby transforming frames of the video stream. Depending on the specific modification request, the properties of the mentioned regions can be transformed in different ways. Such modifications can involve: changing the color of the region; removing portions of the region from frames of the video stream; including new objects in regions based on modification requests; and modifying or distorting elements of the region or object. Any combination of such modifications or other similar modifications can be used in various examples. For certain models to be animated, some feature points can be selected as control points to determine the entire state space of options for model animation.

[0238] In some examples of computer animation models that use face detection to transform image data, a specific face detection algorithm (e.g., Viola-Jones) is used to detect faces in the image. The ASM algorithm is then applied to the facial regions of the image to detect facial feature reference points.

[0239] Other methods and algorithms suitable for face detection can be used. For example, in some examples, landmarks are used to locate virtual features, which represent distinguishable points present in most of the images considered. For example, for facial landmarks, the location of the left pupil can be used. If the initial landmark is unrecognizable (e.g., in the case of a person wearing an eye patch), secondary landmarks can be used. Such landmark recognition processes can be used for any such object. In some examples, a set of landmarks forms a shape. The shape can be represented as a vector using the coordinates of the points in the shape. One shape is aligned with another shape using a similarity transformation (allowing translation, scaling, and rotation) that minimizes the average Euclidean distance between the points of the shapes. The average shape is the average of the aligned training shapes.

[0240] The transformation system can capture image or video streams on a client device (e.g., XR system 1802) and perform complex image manipulations locally on the XR system 1802 while maintaining an appropriate user experience, computation time, and power consumption. Complex image manipulations can include size and shape changes, mood shifts (e.g., changing a face from frowning to smiling), state shifts (e.g., aging a subject, reducing apparent age, or changing gender), style shifts, application of graphic elements, and any other suitable image or video manipulations implemented by a convolutional neural network that has been configured to execute efficiently on the XR system 1802.

[0241] In some examples, a computer animation model for transforming image data can be used by a system in which a user can capture an image or video stream (e.g., a selfie) using an XR system 1802 that has a neural network operating as part of an interactive client 1806 operating on the XR system 1802. A transformation system operating within the interactive client 1806 determines the presence of a face within the image or video stream and provides a modification icon associated with the computer animation model to transform the image data, or the computer animation model may exist in association with the interface described herein. This modification icon will be included as part of the modification operation, serving as the basis for modifying the user's face within the image or video stream. Once a modification icon is selected, the transformation system initiates processing to transform the user's image to reflect the selected modification icon (e.g., generating a smiley face on the user). Once the image or video stream is captured and the specified modification is selected, the modified image or video stream can be presented in a GUI displayed on the XR system 1802. The transformation system may implement a complex convolutional neural network on a portion of the image or video stream to generate and apply the selected modification. In other words, users can capture image or video streams, and once an edit icon is selected, the modified result can be presented to the user in real-time or near real-time. Furthermore, while the video stream is being captured, the modifications can be persistent, and the selected edit icon continues to be toggled. Machine-trained neural networks can be used to achieve such modifications.

[0242] The GUI presenting the modifications performed by the transformation system can offer users additional interactive options. Such options can be based on the interface used to initiate the selection and content capture of a specific computer animation model (e.g., initiated from a content creator user interface). In various examples, the modifications can be persistent after the initial selection of the modification icon. Users can toggle the modification on or off by tapping or otherwise selecting the face modified by the transformation system, and save it for later viewing or browsing to other areas of the imaging application. In cases where multiple faces are modified by the transformation system, users can globally toggle the modification on or off by tapping or selecting a single face modified and displayed within the GUI. In some examples, individual faces within a set of multiple faces can be modified separately, or such modifications can be toggled individually by tapping or selecting a single face or a series of individual faces displayed within the GUI.

[0243] Story table 1918 stores data about collections of messages and associated image, video, or audio data, compiled into collections (e.g., stories or galleries). The creation of a specific collection can be initiated by a specific user (e.g., each user for whom records are maintained in entity table 1908). A user can create a "personal story" in the form of a collection of content that has been created and sent / broadcast by that user. For this purpose, the user interface of interactive client 1806 can include user-selectable icons that allow message senders to add specific content to their personal stories.

[0244] Collections can also constitute "live stories" as a collection of content from multiple users, created manually, automatically, or using a combination of manual and automatic techniques. For example, a "live story" can constitute a curated stream of user-submitted content from various locations and events. Users whose client devices have location services enabled and who are at a co-located event at a specific time can be presented with the option to contribute content to a specific live story, for example, via the user interface of interactive client 1806. Live stories can be identified to a user by interactive client 1806 based on their location. The end result is a "live story" told from a collective perspective.

[0245] Another type of content collection is called a "location story," which allows users of the XR system 1802 located in a specific geographic location (e.g., on a college or university campus) to contribute to a specific collection. In some examples, contributing to a location story may require secondary authentication to verify that the end user belongs to a specific organization or other entity (e.g., a student on a university campus).

[0246] As mentioned above, video table 1914 stores video data, which in some examples is associated with messages whose records are maintained within message table 1906. Similarly, image table 1916 stores image data associated with messages whose message data is stored in entity table 1908. Entity table 1908 can associate various enhancements from enhancement table 1912 with various images and videos stored in image table 1916 and video table 1914.

[0247] Database 1904 also includes social network information collected by the social network system 2022.

[0248] System Architecture

[0249] Figure 20This is a block diagram illustrating further details of the interactive system 1800 according to some examples. Specifically, the interactive system 1800 is shown as including an interactive client 1806 and an interactive server 1822. The interactive system 1800 comprises multiple subsystems, which are supported on the client side by the interactive client 1806 and on the server side by the interactive server 1822. Example subsystems are discussed below.

[0250] Image processing system 2002 provides various functions that enable users to capture and enhance (e.g., enhance or otherwise modify or edit) media content associated with a message.

[0251] The camera device system 2004 includes (e.g., in a camera device application) control software that (e.g., directly or via an operating system) interacts with and controls the hardware camera device hardware of the XR system 1802 to modify and enhance real-time images captured and displayed via the interactive client 1806.

[0252] Enhancement system 2006 provides functionality related to the generation and distribution of enhancements (e.g., media overlays) for images captured in real-time by the camera device of XR system 1802 or retrieved from the memory of XR system 1802. For example, enhancement system 2006 is operable to select, present, and display media overlays (e.g., image filters or image lenses) for interactive client 1806 to enhance real-time images received via camera device system 2004 or stored images retrieved from memory 1702 of XR system 1802. These enhancements are selected by enhancement system 2006 and presented to the user of interactive client 1806 based on some inputs and data, such as:

[0253] The geolocation of the XR system 1802; and

[0254] Social network information of users of XR system 1802.

[0255] Enhancements may include audio and visual content and visual effects. Examples of audio and visual content include images, text, logos, animations, and sound effects. Examples of visual effects include color overlays. Audio and visual content or visual effects may be applied to media content items (e.g., photos or videos) at XR system 1802 for transmission in messages, or to video content such as video content streams or feeds sent from interactive client 1806. Therefore, image processing system 2002 can interact with and support various subsystems of communication system 2008, such as messaging system 2010 and video communication system 2012.

[0256] Media overlays may include text or image data that can be superimposed on photographs taken by the XR system 1802 or video streams produced by the XR system 1802. In some examples, media overlays may be location overlays (e.g., Venice Beach), names of live events, or names of businesses (e.g., Beach Cafe). In other examples, the image processing system 2002 uses the geolocation of the XR system 1802 to identify media overlays that include the name of a business at the geolocation of the XR system 1802. Media overlays may include additional tags associated with the business. Media overlays may be stored in a database 1826 and accessed through a database server 1824.

[0257] Image processing system 2002 provides a user-based publishing platform that allows users to select geographic locations on a map and upload content associated with those locations. Users can also specify the environment in which particular media overlays should be provided to other users. Image processing system 2002 generates a media overlay that includes the uploaded content and associates it with the selected geographic location.

[0258] Enhanced Creation System 2014 supports the XR developer platform and includes applications that enhance (e.g., XR experiences) the creation and publishing of interactive clients 1806 for content creators (e.g., artists and developers). Enhanced Creation System 2014 provides content creators with a library of built-in features and tools, including, for example, custom shaders, tracking technologies, and templates.

[0259] In some examples, Enhancement Creation System 2014 provides a merchant-based publishing platform that allows merchants to select specific enhancements associated with geolocation through a bidding process. For instance, Enhancement Creation System 2014 associates the media overlay of the highest-bidder merchant with a corresponding geolocation for a predefined amount of time.

[0260] Communication system 2008 is responsible for enabling and processing various forms of communication and interaction within interactive system 1800, and includes messaging system 2010, audio communication system 2016, and video communication system 2012. Messaging system 2010 is responsible for enabling temporary or time-limited access to content by interactive client 1806. Messaging system 2010 includes multiple timers (e.g., within short-lived timer system 2018) that selectively enable access (e.g., for presentation and display) of messages and associated content via interactive client 1806 based on duration and display parameters associated with a message or set of messages (e.g., a story). Further details regarding the operation of short-lived timer system 2018 are provided below. Audio communication system 2016 enables and supports audio communication (e.g., real-time audio chat) between multiple interactive clients 1806. Similarly, video communication system 2012 enables and supports video communication (e.g., real-time video chat) between multiple interactive clients 1806.

[0261] The User Management System 2020 is operationally responsible for managing user data and profiles, and includes a Social Network System 2022, which maintains social network information about the relationships between users of the Interaction System 1800.

[0262] The Collection Management System 2024 is operationally responsible for managing collections or sets of media (e.g., collections of text, images, video, and audio data). Collections of content (e.g., messages, including images, videos, text, and audio) can be organized into "event galleries" or "event stories." Such collections can be made available for a specified time period (e.g., the duration of the event the content relates to). For example, content related to a concert can be made available as a "story" for the duration of the concert. The Collection Management System 2024 can also be responsible for publishing icons that notify the user interface of the interactive client 1806 of the availability of specific collections. The Collection Management System 2024 includes curatorial functions that enable collection managers to manage and curate specific content collections. For example, the curation interface allows event organizers to curate collections of content related to a specific event (e.g., removing inappropriate content or redundant messages). Additionally, the Collection Management System 2024 employs machine vision (or image recognition technology) and content rules to automatically curate content collections. In some examples, users may be compensated for including user-generated content in a collection. In such cases, the collection management system 2024 operates to automatically pay such users for access to its content.

[0263] Map system 2026 provides various geolocation functions and supports the presentation of map-based media content and messages by interactive client 1806. For example, map system 2026 enables the display (e.g., stored in profile data 1902) of user icons or avatars on a map to indicate the current or past locations of the user's "friends" and media content generated by these friends (e.g., a collection of messages including photos and videos) within the map context of interactive client 1806. For example, on the map interface of interactive client 1806, messages posted by a user from a specific geolocation to interactive system 1800 can be displayed to the specific user's "friends" within the context of that specific location on the map. Users can also share their location and status information with other users of interactive system 1800 via interactive client 1806 (e.g., using an appropriate status avatar), where the location and status information is similarly displayed to selected users within the context of the map interface of interactive client 1806.

[0264] Game System 2028 provides various game functions within the context of Interactive Client 1806. Interactive Client 1806 provides a game interface that offers a list of available games that a user can launch within the context of Interactive Client 1806 and play with other users of Interactive System 1800. Interactive System 1800 also enables specific users to invite other users to participate in specific games by sending invitations from Interactive Client 1806. Interactive Client 1806 also supports sending and receiving audio, video, and text messages (e.g., chat) within the context of playing the game, provides leaderboards for the game, and also supports providing in-game rewards (e.g., game currency and items).

[0265] External resource system 2030 provides interactive client 1806 with an interface to communicate with remote servers (e.g., third-party server 1814) to launch or access external resources (i.e., applications or applets). Each third-party server 1814 hosts applications or smaller versions of applications (e.g., game applications, utility applications, payment applications, or ride-sharing applications) based on markup languages ​​(e.g., HTML5). Interactive client 1806 can launch web-based resources (e.g., applications) by accessing HTML5 files from third-party server 1814 associated with the web-based resource. Applications hosted by third-party server 1814 are programmed in JavaScript using a software development kit (SDK) provided by interactive server 1822. The SDK includes APIs with functionality that can be called or activated by the web-based application. Interactive server 1822 hosts a JavaScript library that provides access to a given external resource for specific user data of interactive client 1806. HTML5 is an example of a technology used for programming games, but applications and resources programmed using other technologies can be used.

[0266] To integrate the SDK's functionality into the web-based resource, the third-party server 1814 downloads the SDK from the interactive server 1822, or the third-party server 1814 otherwise receives the SDK. Once downloaded or received, the SDK is included as part of the application code of the web-based external resource. The code of the web-based resource can then call or activate certain functions of the SDK to integrate the features of the interactive client 1806 into the web-based resource.

[0267] The SDK stored on the interactive server system 1812 effectively bridges external resources (e.g., application 1808 or applet) with the interactive client 1806. This provides users with a seamless experience communicating with other users on the interactive client 1806 while preserving the look and feel of the interactive client 1806. To bridge communication between external resources and the interactive client 1806, the SDK facilitates communication between a third-party server 1814 and the interactive client 1806. The WebViewJavaScriptBridge running on the XR system 1802 establishes two unidirectional communication channels between the external resources and the interactive client 1806. Messages are sent asynchronously between the external resources and the interactive client 1806 via these communication channels. Each SDK function activation is sent as a message and callback. Each SDK function is implemented by constructing a unique callback identifier and sending a message with that callback identifier.

[0268] By using the SDK, not all information from the interactive client 1806 is shared with the third-party server 1814. The SDK limits which information is shared based on the needs of the external resources. Each third-party server 1814 provides the interactive server 1822 with an HTML5 file corresponding to the web-based external resource. The interactive server 1822 can add a visual representation (e.g., box design or other graphics) of the web-based external resource to the interactive client 1806. Once the user selects the visual representation or instructs the interactive client 1806 via its GUI to access a feature of the web-based external resource, the interactive client 1806 obtains the HTML5 file and instantiates the resource used to access the feature of the web-based external resource.

[0269] Interactive client 1806 presents a GUI (e.g., a login page or title screen) for an external resource. During, before, or after presenting the login page or title screen, interactive client 1806 determines whether the initiated external resource has previously been authorized to access the user data of interactive client 1806. In response to determining that the initiated external resource has previously been authorized to access the user data of interactive client 1806, interactive client 1806 presents another GUI for the external resource, including its functionality and characteristics. In response to determining that the initiated external resource has not previously been authorized to access the user data of interactive client 1806, after displaying the login page or title screen of the external resource for a threshold time period (e.g., 3 seconds), interactive client 1806 slides up a menu (e.g., animates the menu to appear from the bottom of the screen to the middle of the screen or other parts) to authorize the external resource to access user data. This menu identifies the type of user data that the external resource will be authorized to use. In response to receiving a user selection of the accept option, interactive client 1806 adds the external resource to the list of authorized external resources and allows the external resource to access user data from interactive client 1806. External resources are authorized by interactive client 1806 to access user data under the OAuth 2 framework.

[0270] Interactive client 1806 controls the type of user data shared with external resources based on the type of authorized external resource. For example, it provides access to a first type of user data (e.g., 2D avatars of users with or without different avatar characteristics) to external resources including full-scale applications (e.g., application 1808). As another example, it provides access to a second type of user data (e.g., payment information, 2D avatars of users, 3D avatars of users, and avatars with various avatar characteristics) to external resources including smaller versions of applications (e.g., web-based versions of applications). Avatar characteristics include different ways of customizing the appearance and feel of an avatar (e.g., different poses, facial features, clothing, etc.).

[0271] The advertising system 2032 operates by enabling third parties to purchase advertisements to be presented to end users via the interactive client 1806, and also handles the delivery and presentation of these advertisements.

[0272] Software Architecture

[0273] Figure 21 This is a block diagram 2100 illustrating a software architecture 2102 that can be installed on any or more of the devices described herein. The software architecture 2102 is supported by hardware such as a machine 2104, which includes a processor 2106, memory 2108, and I / O components 2110. In this example, the software architecture 2102 can be conceptualized as a stack of layers, where each layer provides a specific function. The software architecture 2102 includes layers such as an operating system 2112, libraries 2114, frameworks 2116, and applications 2118. Operationally, application 2118 activates API calls 2120 via the software stack and receives messages 2122 in response to API calls 2120.

[0274] Operating system 2112 manages hardware resources and provides public services. Operating system 2112 includes, for example, a kernel 2124, services 2126, and drivers 2128. Kernel 2124 serves as an abstraction layer between hardware and other software layers. For example, kernel 2124 provides memory management, processor management (e.g., scheduling), component management, networking and security settings, and other functions. Services 2126 can provide other public services to other software layers. Drivers 2128 are responsible for controlling or interfacing with the underlying hardware. For example, drivers 2128 may include display drivers, camera drivers, Bluetooth® or Bluetooth® Low Energy drivers, flash memory drivers, serial communication drivers (e.g., USB drivers), Wi-Fi® drivers, audio drivers, power management drivers, etc.

[0275] Library 2114 provides common low-level infrastructure used by application 2118. Library 2114 may include system library 2130 (e.g., the C standard library), which provides functions such as memory allocation, string manipulation, and mathematical functions. Furthermore, library 2114 may include API library 2132, such as media libraries (e.g., libraries for supporting the rendering and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Picture Experts Group (JPEG or JPG), or Portable Web Graphics (PNG)), graphics libraries (e.g., the OpenGL framework for rendering graphic content in 2D and 3D on a display), database libraries (e.g., SQLite, which provides various relational database functions), web libraries (e.g., WebKit, which provides web browsing capabilities), and so on. Library 2114 may also include various other libraries 2134 to provide many other APIs to application 2118.

[0276] Framework 2116 provides common high-level infrastructure for use by application 2118. For example, framework 2116 provides various GUI functions, advanced resource management, and advanced location services. Framework 2116 can provide a wide range of other APIs that can be used by application 2118, some of which may be specific to a particular operating system or platform.

[0277] In the example, application 2118 may include home application 2136, contact application 2138, browser application 2140, book reader application 2142, location application 2144, media application 2146, messaging application 2148, game application 2150, and a wide variety of other applications such as third-party application 2152. Application 2118 is a program that performs the functions defined in the program. One or more applications 2118 can be created using various programming languages, such as object-oriented programming languages ​​(e.g., Objective-C, Java, or C++) or procedural programming languages ​​(e.g., C or assembly language). In a particular example, third-party application 2152 (e.g., an application developed by an entity other than a platform-specific vendor using the Android™ or iOS™ SDK) may be mobile software running on a mobile operating system such as iOS™, Android™, Windows® Phone, or another mobile operating system. In this example, third-party application 2152 may activate API calls 2120 provided by operating system 2112 to facilitate the functions described herein.

[0278] in conclusion

[0279] Changes and modifications may be made to the disclosed examples without departing from the scope of this disclosure. Such and other changes or modifications are intended to be included within the scope of this disclosure as set forth in the appended claims.

[0280] Glossary

[0281] "Carrier signal" refers to any intangible medium capable of storing, encoding, or carrying instructions to be executed by a machine, and includes digital or analog communication signals or other intangible media to facilitate the communication of such instructions. Instructions can be sent or received over a network using a transmission medium via a network interface device.

[0282] "Client device" refers to any machine that interfaces with a communication network to obtain resources from one or more server systems or other client devices. Client devices can be, but are not limited to, mobile phones, desktop computers, laptop computers, PDAs, smartphones, tablet computers, ultrabooks, netbooks, laptops, multiprocessor systems, microprocessor-based or programmable consumer electronics, game consoles, set-top boxes, or any other communication device that a user can use to access the network.

[0283] "Communications network" refers to one or more parts of a network, which can be an ad hoc network, intranet, extranet, virtual private network (VPN), local area network (LAN), wireless LAN (WLAN), WAN, wireless WAN (WWAN), metropolitan area network (MAN), the Internet, a part of the Internet, a part of the Public Switched Telephone Network (PSTN), a Common Old-Style Telephone Service (POTS) network, a cellular telephone network, a wireless network, a Wi-Fi® network, other types of networks, or a combination of two or more such networks. For example, a network or part of a network may include a wireless network or a cellular network, and the coupling may be a Code Division Multiple Access (CDMA) connection, a Global System for Mobile Communications (GSM) connection, or other types of cellular or wireless coupling. In this example, coupling can enable any data transmission technology of various types, such as single-carrier radio transmission technology (1xRTT), evolved data optimization (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rate GSM evolution (EDGE) technology, the 3rd Generation Partnership Project (3GPP) including 3G, fourth-generation wireless (4G) networks, Universal Mobile Telecommunications System (UMTS), High-Speed ​​Packet Access (HSPA), Global Microwave Access Interoperability (WiMAX), Long Term Evolution (LTE) standards, other data transmission technologies defined by various standards setting organizations, other long-distance protocols, or other data transmission technologies.

[0284] A “component” is a logical, device, or physical entity having boundaries defined by function or subroutine calls, branch points, APIs, or other technologies provided for partitioning or modularizing specific processing or control functions. Components can be combined with other components via their interfaces to perform machine processing. A component can be an encapsulated functional hardware unit designed for use with other components and can be part of a program that typically performs a specific function within a related function. Components can constitute software components (e.g., code implemented on a machine-readable medium) or hardware components. A “hardware component” is a tangible unit capable of performing certain operations and which can be configured or arranged in some physical manner. In various examples, one or more hardware components (e.g., standalone computer systems, client computer systems, or server computer systems) or one or more hardware components (e.g., processors or processor groups) of a computer system can be configured by software (e.g., an application or application portion) to operate to perform certain operations as described herein. Hardware components can also be implemented mechanically, electronically, or in any suitable combination thereof. For example, a hardware component can include a dedicated circuit system or logic permanently configured to perform certain operations. Hardware components can be dedicated processors, such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). Hardware components can also include programmable logic or circuit systems that are temporarily configured by software to perform certain operations. For example, a hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a particular machine (or a specific part of a machine), which is uniquely tailored to perform the configured function and is no longer a general-purpose processor. It will be appreciated that a decision may be made, for cost and time reasons, whether to implement a hardware component mechanically in a dedicated and permanently configured circuit system or in a temporarily configured (e.g., configured by software) circuit system. Therefore, the phrase “hardware component” (or “hardware-implemented component”) should be understood to include tangible entities, i.e., entities that are physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a certain way or perform certain operations described herein. Consider the example of a hardware component being temporarily configured (e.g., programmed), without needing to configure or instantiate each hardware component at any given time. For example, in cases where the hardware components include a general-purpose processor that can be configured by software to become a dedicated processor, the general-purpose processor can be configured as different dedicated processors (e.g., including different hardware components) at different times. The software accordingly configures one or more specific processors to constitute a specific hardware component at one time and different hardware components at different times. The hardware components can provide information to and receive information from other hardware components.Therefore, the described hardware components can be considered communicatively coupled. In the presence of multiple hardware components, communication can be achieved through signal transmission between or among two or more hardware components (e.g., via appropriate circuitry and buses). In examples where multiple hardware components are configured or instantiated at different times, such communication between hardware components can be achieved, for example, by storing information in a memory structure accessible to the multiple hardware components and retrieving information from the memory structure. For example, a hardware component can perform an operation and store the output of that operation in a memory device to which it is communicatively coupled. Another hardware component can then access the memory device at a subsequent time to retrieve and process the stored output. Hardware components can also initiate communication with input or output devices and can operate on resources (e.g., collections of information). The various operations of the example methods described herein can be performed, at least in part, by one or more processors configured, either temporarily (e.g., via software) or permanently, to perform the relevant operations. Whether temporarily or permanently configured, such processors can constitute processor-implemented components that operate to perform one or more operations or functions described herein. As used herein, "processor-implemented component" refers to a hardware component implemented using one or more processors. Similarly, the methods described herein can be implemented at least in part by processors, where a particular processor or one or more processors are examples of hardware. For example, at least some of the operations of the methods can be executed by one or more processors or processor-implemented components. Furthermore, one or more processors can operate to support the execution of related operations in a “cloud computing” environment or as “Software as a Service” (SaaS). For example, at least some of the operations can be executed by a group of computers (as an example of a machine including processors), where these operations are accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., APIs). The execution of some operations can be distributed among processors, not residing within a single machine, but deployed across multiple machines. In some examples, the processor or processor-implemented component may reside in a single geographic location (e.g., in a home environment, office environment, or server cluster). In other examples, the processor or processor-implemented component may be distributed across multiple geographic locations.

[0285] "Machine-readable storage medium" refers to both machine storage media and transmission media. Therefore, these terms encompass both storage devices / media and carrier / modulated data signals. The terms "computer-readable medium," "machine-readable medium," and "device-readable medium" refer to the same thing and can be used interchangeably in this disclosure.

[0286] "Machine storage medium" refers to one or more storage devices and media (e.g., centralized or distributed databases, and associated caches and servers) that store executable instructions, routines, and data. Therefore, this term should be considered to include, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGAs, and flash memory devices; disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage medium," "device storage medium," and "computer storage medium" mean the same thing and may be used interchangeably in this disclosure. The terms "machine storage medium," "computer storage medium," and "device storage medium" expressly exclude carrier waves, modulated data signals, and other such media, at least some of which are covered by the term "signal medium."

[0287] "Non-transitory machine-readable storage medium" refers to a tangible medium capable of storing, encoding, or carrying instructions that can be executed by a machine.

[0288] "Signal medium" means any intangible medium capable of storing, encoding, or carrying instructions executable by a machine, and includes digital or analog communication signals or other intangible media to facilitate the communication of software or data. The term "signal medium" should be considered to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal whose characteristics are set or altered in a manner that encodes information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

Claims

1. A machine-implemented method, comprising: Image data is captured using a camera device that utilizes an extended reality (XR) system; Based on the image data, estimate the dark adaptation level of at least one user among the users; The night mode rendering configuration is selected based on the estimated dark adaptation level. Use the selected rendering configuration to generate the XR display; as well as The XR display is shown to the user.

2. The machine-implemented method according to claim 1, wherein, Estimating the dark adaptation level includes estimating the photoreceptor response of rod cells.

3. The machine-implemented method according to claim 1, wherein, Estimating the dark adaptation level involves analyzing eye-tracking data.

4. The machine-implemented method according to claim 1, wherein, Selecting the rendering configuration includes choosing peripheral rendering settings and central concave rendering settings.

5. The machine-implemented method according to claim 1, wherein, The rendering configuration includes a lower spatial resolution in the peripheral region compared to the central recessed region.

6. The machine-implemented method according to claim 1, wherein, The rendering configuration includes a lower temporal resolution in the peripheral region compared to the central recessed region.

7. The machine-implemented method according to claim 1, wherein, The rendering configuration includes light of a longer wavelength in the peripheral region compared to the central recessed region.

8. A machine comprising: One or more processors; as well as The memory stores instructions that, when executed by the one or more processors, cause the machine to perform operations, including: Image data is captured using a camera device that utilizes an extended reality (XR) system; Based on the image data, estimate the dark adaptation level of at least one user among the users; The night mode rendering configuration is selected based on the estimated dark adaptation level. Use the selected rendering configuration to generate the XR display; and The XR display is shown to the user.

9. The machine according to claim 8, wherein, Estimating the dark adaptation level includes estimating the photoreceptor response of rod cells.

10. The machine according to claim 8, wherein, Estimating the dark adaptation level involves analyzing eye-tracking data.

11. The machine according to claim 8, wherein, Selecting the rendering configuration includes choosing peripheral rendering settings and central concave rendering settings.

12. The machine according to claim 8, wherein, The rendering configuration includes a lower spatial resolution in the peripheral region compared to the central recessed region.

13. The machine according to claim 8, wherein, The rendering configuration includes a lower temporal resolution in the peripheral region compared to the central recessed region.

14. The machine according to claim 8, wherein, The rendering configuration includes light of a longer wavelength in the peripheral region compared to the central recessed region.

15. A machine storage medium including instructions that, when executed by a machine, cause the machine to perform operations, the operations including: Image data is captured using a camera device that utilizes an extended reality (XR) system; Based on the image data, estimate the dark adaptation level of at least one user among the users; The night mode rendering configuration is selected based on the estimated dark adaptation level. Use the selected rendering configuration to generate the XR display; as well as The XR display is shown to the user.

16. The machine storage medium according to claim 15, wherein, Estimating the dark adaptation level includes estimating the photoreceptor response of rod cells.

17. The machine storage medium according to claim 15, wherein, Estimating the dark adaptation level involves analyzing eye-tracking data.

18. The machine storage medium according to claim 15, wherein, Selecting the rendering configuration includes choosing peripheral rendering settings and central concave rendering settings.

19. The machine storage medium according to claim 15, wherein, The rendering configuration includes a lower spatial resolution in the peripheral region compared to the central recessed region.

20. The machine storage medium according to claim 15, wherein, The rendering configuration includes a lower temporal resolution in the peripheral region compared to the central recessed region.

Images