Adaptive brightness for augmented reality displays

Abstract

Systems and methods for adaptively adjusting the brightness of a wearable device projection system are disclosed. The systems and methods perform operations including: causing projection elements of an AR wearable device to project an image; receiving a measurement of ambient light from an ambient light sensor; adjusting one or more hardware parameters of the projection elements of the AR wearable device based on the measurement of ambient light; modifying one or more color values ​​of an image displayed by the projection elements of the AR wearable device based on the measurement of ambient light; and projecting an image with modified color values ​​using the projection elements of the AR wearable device having one or more adjusted hardware parameters.

Description

Technology Field

[0001] Cross-reference regarding related applications

[0002] This application claims the benefit of priority to U.S. Provisional Application No. 63 / 208,810 filed June 9, 2021 and U.S. Patent Application No. 17 / 582,633 filed January 24, 2022. The contents of these prior applications are deemed to be part of this application, and are thus incorporated herein by reference in their entirety.

[0003] Technology field

[0004] The present disclosure generally relates to wearable devices having display systems. Specifically, the present disclosure deals with systems and methods for using digital light projectors for augmented reality wearable devices. Background Technology

[0005] Wearable devices can be implemented with a transparent or translucent display that allows the user of the wearable device to see the surrounding environment. These devices allow the user to see the surrounding environment through the transparent or translucent display, and also allow the user to see objects (e.g., virtual objects such as images, videos, text, etc.) that are created so that the display appears as part of the surrounding environment and / or appears or is overlaid on the surrounding environment. Brief explanation of the drawing

[0006] To facilitate the identification of the discussion of any specific element or act, the top digit or numbers in the reference number indicate the drawing number where the element is first introduced. FIG. 1 is a perspective view of a wearable device according to some examples. FIG. 2 is a block diagram illustrating a network environment for operating an Augmented Reality (AR) wearable device according to one example. FIG. 3 is a block diagram illustrating an AR wearable device according to one example. FIG. 4 is a block diagram illustrating a DLP projector of an AR wearable device according to one example. Figure 5 is a block diagram illustrating a DLP controller according to one example. FIG. 6 illustrates an AR wearable device according to some examples. FIG. 7 is a flowchart illustrating exemplary operations of a DLP controller according to some examples. FIG. 8 is a schematic representation of a machine in the form of a computer system in which a set of instructions can be executed to enable the machine to perform any one or more of the methodologies discussed in this specification, according to some examples. FIG. 9 is a block diagram illustrating a software architecture in which examples can be implemented. Specific details for implementing the invention

[0007] The following description describes systems, methods, techniques, instruction sequences, and computing machine program products that exemplify examples of the subject matter. In the following description, for the purposes of explanation, many specific details are presented to provide an understanding of the various examples of the subject matter. However, it will be apparent to those skilled in the art that examples of the subject matter may be practiced with some or all of these specific details. Examples are merely representative of possible variations. Unless explicitly stated otherwise, structures (structural components, such as modules) may be optional, combined, or subdivided, and operations (e.g., in procedures, algorithms, or other functions) may be ordered, combined, or subdivided.

[0008] AR wearable devices implemented with transparent or translucent displays allow users to view their surrounding environment through the transparent or translucent displays. Additionally, AR wearable devices can allow users to view objects rendered on the display of the AR wearable device (e.g., virtual objects such as images, videos, text, etc.) that appear as part of the surrounding environment and / or appear as an overlay on the surrounding environment. Such AR wearable devices can provide users with an augmented reality experience.

[0009] The rendering of virtual objects can be based on the position of the wearable device relative to a reference frame (outside the wearable device) or relative to a physical object, so that the virtual object appears accurately on the display. The virtual object appears aligned with the physical object as perceived by the user of the AR wearable device. Graphics (e.g., graphic elements including instructions and guides) appear to be attached to the physical object of interest. To do this, the AR wearable device detects the physical object and tracks the posture of the AR wearable device relative to the position of the physical object. The posture identifies the position and orientation of the object relative to the reference frame or another object.

[0010] In one example, an AR wearable device includes a projector (e.g., a Digital Light Projector (DLP)) that displays virtual objects on the display of the AR wearable device (e.g., the lenses of the AR wearable device). DLP projectors operate by projecting light from a light source (e.g., one or more LEDs, such as red, green, and blue LEDs) through a color wheel toward a Digital Micromirror Device (DMD). The DMD controls whether to reflect the colored light toward the display of the AR wearable device. DLP projectors generate colors for the human eye by cycling through (R)ed, (G)reen, and (B)lue bit planes at very high rates (e.g., 10 kHz). The sum of all bit planes generates the impression of color for the human eye. The order in which the bit planes are displayed is optimized individually for each DLP projector (with respect to power and colors). As such, different DLP projectors will have different color cycle arrangements. Furthermore, depending on the frame rate of the DLP projector, the DLP projector fills the frame time by repeating a bit plane sequence (e.g., cycled). In this way, each DLP projector is typically configured to optimize the bit plane sequence (for power saving, color correction, and reduction of rainbow artifacts (for wall projectors)).

[0011] In some implementations, AR wearable devices using DLP projectors struggle to operate in environments with significant variations in ambient brightness, ranging from dim indoor lighting to bright outdoor lighting (i.e., outdoor parks or dimly lit restaurants). In these cases, the AR wearable device displays objects with a perceived brightness (effective brightness value) that matches the ambient lighting conditions of the operating environment, while also reducing the amount of power used and maintaining a high contrast ratio. Some AR wearable devices can modify the brightness of a projected image by repeating the projected RGB bit planes, such as projecting an RGBRGB bit plane (e.g., repeating the same RGB pixel values ​​twice)—referred to as 0% dark time. This can result in a higher effective brightness but can cause image ghosting and color breakup. To avoid image ghosting and color breakup, AR wearable devices can introduce short blanking periods, such as projecting a dark (all black - no light) bit plane after the RGB bit plane—this is referred to as 50% dark time. While this approach can resolve image ghosting and color breakup effects, it cannot reach the same high level of brightness, even if it can achieve a level of brightness lower than 0% dark time.

[0012] The disclosed systems and methods include operations that configure the operation of a DLP projector for use in an AR wearable device based on measured ambient light received from an ambient light sensor of the AR wearable device. The disclosed systems can dynamically switch between 0% dark time and 50% dark time and can modify pixel values ​​in software to achieve a desired effective brightness level (maximum brightness level available using 0% dark time or minimum brightness level available using 50% dark time) without introducing image ghosting or color breakup. By detecting ambient brightness (measurement of ambient light) and controlling a combination of LED current, projector dark time, and RGB composition of the projection, the AR wearable device can adapt to a wide range of ambient light in different real-world environments while maintaining the desired perceived brightness and power savings.

[0013] FIG. 1 is a perspective view of an AR wearable device (e.g., AR eyeglasses or AR glasses) (100) according to some examples. The AR glasses (100) may include a frame (132) made of any suitable material, such as plastic or metal, including any suitable shape memory alloy. In one or more examples, the frame (132) includes a front piece (134) comprising a first optical left optical element holder (114) (e.g., a display or lens holder) and a second or right optical element holder (120) connected by a bridge (118). The front piece (134) further includes a left end portion (110) and a right end portion (124). The first or left optical element (116) and the second or right optical element (122) may be provided within their respective left optical element holder (114) and right optical element holder (120). The optical elements (116 and 122) may be tinted (where a tinting layer is placed on top of the lens or glass constituting the optical elements (116 and 122)) or transparent (where no tinting layer is placed). Each of the right optical element (122) and the left optical element (116) may be a lens, a display, a display assembly, or a combination thereof. Any of the display assemblies disclosed herein may be provided in the AR glasses (100).

[0014] The frame (132) further comprises a left arm or temple piece (136) and a right arm or temple piece (138) that are attached to the respective left end (110) and right end (124) of the front piece (134) by any suitable means, such as a hinge (not shown), to be attached to the front piece (134) or to be firmly or fixedly attached to the front piece (134) so ​​as to be integral with the front piece (134). In one or more embodiments, each of the temple piece (136) and the temple piece (138) comprises a first part (108) that is attached to the respective left end (110) or right end (124) of the front piece (134) and any suitable second part (126) for attaching to the user's ear. In one example, the front piece (134) may be formed from a single piece of material to have a single or integral structure. In one example as illustrated in FIG. 1, the entire frame (132) may be formed from a single piece of material to have a single or integral structure.

[0015] AR glasses (100) may include a computing device, such as a computer (128), which may be of any suitable type to be mounted on a frame (132) and, in one or more examples, may be of a suitable size and shape to be placed at least partially on one of a temple piece (136) and a temple piece (138). In one or more examples, as illustrated in FIG. 1, the computer (128) has a size and shape similar to that of one temple piece (138) (e.g., or temple piece (136)), and accordingly, is placed almost entirely, if not completely, within the structure and boundaries of such temple piece (138). In one or more examples, the computer (128) is placed on both the temple piece (136) and the temple piece (138). The computer (128) may include one or more processors having memory, wireless communication circuitry, and power. As discussed below, the computer (128) includes low-power circuits, high-speed circuits, and a display processor. Various other examples may include these elements in different configurations or integrate them together in different ways. Further details of the embodiments of the computer (128) may be implemented as exemplified by the wearable device (210) discussed below. In some embodiments, the computer (128) implements the DLP controller discussed below.

[0016] The computer (128) additionally includes a battery (106) or other suitable portable power source. In one example, the battery (106) is placed in either the temple piece (136) or the temple piece (138). In the AR glasses (100) illustrated in FIG. 1, the battery (106) is shown placed in the left temple piece (136) and electrically coupled to the rest of the computer (128) placed in the right temple piece (138) using a connection (130). The AR glasses (100) may include a connector or port (not shown), a wireless receiver, a transmitter, or a transceiver (not shown) or a combination of these devices suitable for charging the battery (106) accessible from the outside of the frame (132).

[0017] In one or more implementations, the AR glasses (100) include cameras (102). Although two cameras are depicted, other examples consider the use of a single or additional (i.e., more than two) cameras. In one or more examples, in addition to the cameras (102), the glasses (100) include any number of input sensors (e.g., one or more ambient light sensors) or peripheral devices. The front piece (134) is provided with an outward-facing, forward-facing, or front or outward surface (112) facing forward or outward from the user when the glasses (100) are mounted on the user's face, and an inward-facing, rearward-facing, or rear or inward surface (104) facing the opposite side of the user's face when the glasses (100) are mounted on the user's face. These sensors may include inward-facing video sensors or digital imaging modules, such as cameras (102), which may be mounted or provided on the inner surface (104) of the front piece (134) or elsewhere on the frame (132) facing the user, and outward-facing video sensors or digital imaging modules, such as cameras (102), which may be mounted or provided on the outer surface (112) of the front piece (134) or elsewhere on the frame (132) facing away from the user. These sensors, peripheral devices, or peripherals may additionally include biometric sensors, position sensors, ambient light sensors, thermal temperature sensors, or any other such sensors.

[0018] FIG. 2 is a network diagram illustrating a network environment (200) suitable for operating an AR wearable device (210) according to some examples. The network environment (200) includes an AR wearable device (210), a client device (211), and a server (212) that are coupled to communicate with each other directly or through a network (204). The AR wearable device (210) and the server (212) may each be implemented in a computer system, wholly or partially, as described below with respect to FIG. 8 and FIG. 9. The server (212) may be part of a network-based system. For example, the network-based system may be or may include a cloud-based server system that provides additional information to the AR wearable device (210), such as virtual content (e.g., images of three-dimensional models of virtual objects).

[0019] The client device (211) may be a smartphone, tablet, phablet, laptop computer, access point, or any other such device capable of connecting to the wearable device (210) using both low-power wireless and high-speed wireless connections. The client device (211) is connected to a server (212) and a network (204). The network (204) may include any combination of wired and wireless connections. The server (212) may be one or more computing devices as part of a service or network computing system. Any elements of the client device (211), the server (212), and the network (204) may be implemented using the details of the software architecture (704) or machine (800) described in FIGS. 8 and 9. The client device (211) may provide one or more images for display on the AR wearable device (210). The client device (211) may receive input from a user to adjust the brightness setting of the AR wearable device (210) (e.g., by moving a slider between a minimum brightness setting position and a maximum brightness setting position). In response, the client device (211) transmits a brightness value set by the user to the AR wearable device (210) based on the received input. In these cases, the AR wearable device (210) sets an effective brightness value to the setting indicated by the client device (211) independently of any ambient light measurements received from an ambient light sensor.

[0020] A user (206) operates an AR wearable device (210). The user (206) may be a human user (e.g., a human), a machine user (e.g., a computer configured by a software program to interact with the AR wearable device (210)), or any suitable combination thereof (e.g., a human assisted by a machine or a machine supervised by a human). The user (206) is not part of the network environment (200) but is associated with the AR wearable device (210).

[0021] The AR wearable device (210) may be a computing device having a display, such as a smartphone, a tablet computer, or a wearable computing device (e.g., AR glasses or a head-mounted display device). The computing device may be handheld or detachably mounted on the head of the user (206). In one example, the display may be a screen that displays what is captured by the camera of the AR wearable device (210). In another example, the display of the device may be transparent, such as in the lenses of wearable computing glasses. In yet another example, the display of the AR wearable device (210) may display an image received from the client device (211), such as by using one or more projection elements operated by a DLP controller.

[0022] A user (206) operates an application on an AR wearable device (210). The application may include an AR application configured to provide the user (206) with an experience triggered by a physical object (208), such as a two-dimensional physical object (e.g., a photograph), a three-dimensional physical object (e.g., a statue), a location (e.g., within a facility), or any references in a real-world physical environment (e.g., perceived corners of a wall or furniture). For example, the user (206) may point the camera of the AR wearable device (210) to capture an image of the physical object (208). The image is tracked and recognized locally on the AR wearable device (210) using a local context-aware dataset module of the AR application on the AR wearable device (210). The local context-aware dataset module may include a library of virtual objects associated with real-world physical objects or references. Afterward, the AR application generates additional information corresponding to the image (e.g., a 3D model) and, in response to identifying the recognized image, presents this additional information as one or more images on the display of the AR wearable device (210). If the captured image is not recognized locally by the AR wearable device (210), the AR wearable device (210) downloads additional information corresponding to the captured image (e.g., a 3D model) from the database of the server (212) or from the client device (211) via the network (204).

[0023] In one example, the server (212) may be used to detect and identify a physical object (208) based on sensor data (e.g., image and depth data) from an AR wearable device (210), and to determine the posture of the AR wearable device (210) and the physical object (208) based on the sensor data. The server (212) may also create a virtual object based on the posture of the AR wearable device (210) and the physical object (208). The server (212) communicates the virtual object to the AR wearable device (210). Object recognition, tracking, and AR rendering may be performed on the AR wearable device (210), the server (212), or a combination of the AR wearable device (210) and the server (212).

[0024] Any of the machines, databases, or devices illustrated in FIG. 1 may be implemented on a general-purpose computer modified (e.g., configured or programmed) by software to become a special-purpose computer to perform one or more of the functions described herein for that machine, database, or device. As used herein, "database" is a data storage resource and may store structured data as a text file, table, spreadsheet, relational database (e.g., object-relational database), triple store, hierarchical data store, or any suitable combination thereof. Furthermore, any two or more of the machines, databases, or devices illustrated in FIG. 1 may be combined into a single machine, and the functions described herein for any single machine, database, or device may be subdivided among multiple machines, databases, or devices.

[0025] The network (204) may be any network that enables communication between or among machines (e.g., servers (212)), databases, and devices (e.g., AR wearable devices (210)). Accordingly, the network (204) may be a wired network, a wireless network (e.g., mobile, Bluetooth, short-range network, or long-range network, or cellular network), or any suitable combination thereof. The network (204) may include one or more parts constituting a private network, a public network (e.g., the Internet), or any suitable combination thereof.

[0026] FIG. 3 is a block diagram illustrating modules (e.g., components) of an AR wearable device (210) according to some examples. The AR wearable device (210) includes sensors (302), a display system (304), a processor (308), and a storage device (306). Examples of the AR wearable device (210) include a wearable computing device, AR glasses (AR glasses or AR eyeglasses), a desktop computer, a vehicle computer, a tablet computer, a navigational device, a portable media device, or a smartphone.

[0027] Sensors (302) include, for example, optical sensors (316) (e.g., cameras such as color cameras, thermal cameras, depth sensors, and one or more grayscale, global shutter tracking cameras) and inertial sensors (318) (e.g., gyroscopes, accelerometers). Other examples of sensors (302) include proximity or position sensors (e.g., proximity communication, GPS, Bluetooth, Wi-Fi), ambient light sensors, thermal temperature sensors, audio sensors (e.g., microphones), or any suitable combination thereof. It should be noted that the sensors (302) described herein are for illustrative purposes only and are therefore not limited to those described above.

[0028] The display system (304) includes a screen (324) and a DLP projector (326). The DLP projector (326) includes one or more projection elements (LEDs) that project an image of a virtual object onto the screen (324). In one example, the screen (324) may be transparent or semi-opaque so that a user (206) can view it through the screen (324) (in an AR use case). The DLP projector (326) is configured to operate with a predictable color sequence, a single RGB color cycle per frame, and shorter pixel persistence. The DLP projector (326) is described in more detail below in relation to FIG. 4.

[0029] The processor (308) includes an AR application (310), a tracking system (312), and a DLP controller (314). The AR application (310) uses computer vision to detect and identify a physical environment or a physical object (208). The AR application (310) (which may be partially implemented by a client device (211)) searches for a virtual object (e.g., a 3D object model) based on the identified physical object (208) or physical environment. The AR application (310) renders the virtual object on a display system (304). For the AR application, the AR application (310) includes a local rendering engine that generates a visualization of the virtual object that is overlaid (e.g., superimposed on it, or otherwise displayed alongside it) on an image of the physical object (208) captured by an optical sensor (316). Visualization of a virtual object can be manipulated by adjusting the position of a physical object (208) relative to an optical sensor (316) (e.g., its physical position, orientation, or both). Similarly, visualization of a virtual object can be manipulated by adjusting the posture of an AR wearable device (210) relative to a physical object (208).

[0030] In one example, the AR application (310) includes a brightness control application. The brightness control application receives a measurement of ambient light from an ambient light sensor (e.g., sensors (302)) and adjusts one or more hardware parameters of projection elements of an AR wearable device (e.g., a DLP projector (326)) based on the measurement of ambient light. The AR application (310) modifies one or more color values ​​of an image displayed by projection elements of the AR wearable device based on the measurement of ambient light and projects the image with the modified color values ​​using projection elements of the AR wearable device having one or more adjusted hardware parameters. In one example, the AR application (310) calculates an effective brightness value based on the measurement of ambient light used to control adjustments to one or more hardware parameters and one or more color values. In some examples, as part of calculating the effective brightness value, the AR application (310) selects between operating the projection elements of the DLP projector (326) at 0% Dark Time (DT) or 50% DT.

[0031] The tracking system (312) uses optical sensors (e.g., depth-enabled 3D camera, image camera), inertial sensors (e.g., gyroscope, accelerometer), wireless sensors (Bluetooth, Wi-Fi), GPS sensors, and / or audio sensors to track the posture (e.g., position and orientation) of the AR wearable device (210) relative to the real-world environment (202) to determine the location of the AR wearable device (210) within the real-world environment (202). The tracking system (312) includes, for example, accessing inertial sensor data from the inertial sensor (318) and optical sensor data from the optical sensor (316), and determining the posture based on the combined inertial sensor data and optical sensor data. In another example, the tracking system (312) determines the posture (e.g., position, orientation) of the AR wearable device (210) relative to a reference frame (e.g., real-world environment (202)). In another example, the tracking system (312) includes a visual odometry system that estimates the posture of the AR wearable device (210) based on 3D maps of feature points from inertial sensor data and optical sensor data.

[0032] The DLP controller (314) communicates data signals to the DLP projector (326) to project virtual content (including one or more images) onto a screen (324) (e.g., a transparent display). The DLP controller (314) includes hardware that converts signals from an AR application (310) into display signals for the DLP projector (326). In one example, the DLP controller (314) is part of the processor (308). In another example, the DLP controller (314) is part of the DLP projector (326).

[0033] In one example, the DLP controller (314) configures the DLP projector (326) to operate with a predictable color sequence, a single RGB color cycle per frame, and shorter pixel persistence. For example, the DLP controller (314) determines or identifies the color sequence pattern of the DLP projector (326). The DLP controller (314) instructs the light source (or color filter system) of the DLP projector (326) to produce a single color cycle per frame. The DLP controller (314) also instructs the Digital Micro-mirror Device (DMD) of the DLP projector (326) to generate shorter pixel persistence. The DLP controller (314) is described in more detail below in relation to FIG. 5.

[0034] A storage device (306) stores virtual object content (320) and DLP configuration settings (322). The virtual object content (320) includes, for example, a database of visual references (e.g., images) and corresponding experiences (e.g., 3D virtual objects, interactive features of 3D virtual objects). In one example, the storage device (306) includes a main content dataset, a contextual content dataset, and a visualization content dataset. The main content dataset includes, for example, a first set of images and corresponding experiences (e.g., interactions with 3D virtual object models). For example, images may be associated with one or more virtual object models. The main content dataset may include a core set of images. The core set of images may include a limited number of images identified by the server (212). For example, the core image set may include images depicting covers of the 10 most viewed physical objects and their corresponding experiences (e.g., virtual objects representing the 10 most viewed physical objects). In another example, the server (212) may generate a first image set based on the most popular or frequently scanned images received from the server (212). Thus, the main content dataset does not rely on physical objects or images acquired by the optical sensor (316).

[0035] DLP configuration settings (322) include effective brightness values ​​and settings determined by, for example, a DLP projector (326) and / or a DLP controller (314). Examples of settings include an RGB bit plane cycle rate, a frame rate, a color sequence, brightness values, and a pixel persistence time.

[0036] Any one or more of the modules described herein may be implemented using hardware (e.g., a processor of a machine) or a combination of hardware and software. For example, any module described herein may be configured to have a processor that performs the operations described herein for that module. Furthermore, any two or more of these modules may be combined into a single module, and the functions described herein for the single module may be subdivided among the multiple modules. Furthermore, according to various examples, the modules described herein as being implemented within a single machine, database, or device may be distributed across multiple machines, databases, or devices.

[0037] FIG. 4 is a block diagram illustrating a DLP projector (326) according to one example. The DLP controller (314) includes a light source (402) (also referred to as a light source component or projection element), a condensing lens (404), a shaping lens (406), a DMD (408), and a projection lens (410). The light source (402) includes, for example, a pressurized bulb, a laser, or a high-power LED. In one example, the light source (402) includes three color LEDs, namely a blue LED (412), a red LED (414), and a green LED (416). Each color LED emits color light from its corresponding collimating lens (e.g., collimating lens (418), collimating lens (420), collimating lens (422)) according to its corresponding hardware parameters.

[0038] The DLP controller (314) interfaces with the light source (402) of the DLP projector (326) and controls the light source (402) to generate a single RGB repetition per frame. In one example, the DLP controller (314) interfaces with the light source (402) and identifies the color sequence of the light source (402). For example, the DLP controller (314) queries the DLP projector (326) and identifies the model of the DLP projector (326). The DLP controller (314) identifies the color sequence of the light source (402) based on the model of the DLP projector (326).

[0039] In another example, the light source (402) includes, for example, a white light source (not shown) and a color wheel (not shown) divided into primary colors (red, green, and blue). The color wheel rotates at a high speed (e.g., 7200 RPM). The DLP controller (314) synchronizes the rotation motion of the color wheel so that the green component is displayed on the DMD when the green section of the color wheel is in front of the lamp. The same applies to the red, blue, and other sections. The colors are displayed sequentially at a sufficiently high rate so that the observer sees a composite (full-color) image. Black color is produced by directing unused light away from the light source (402). For example, the unused light is scattered so as to be reflected and dissipated on the projection lens (410) or the inner walls of the DMD (408). The DLP controller (314) operates the light source (402) so that the color wheel rotates in one RGB cycle per frame.

[0040] The condensing lens (404) focuses light from the light source (402) onto the forming lens (406). The forming lens (406) diffuses light from the light source (402) to the DMD (408). The DMD (408) contains hundreds of individual micromirrors. Digital signals representing 0 and 1 drive these micromirrors to rotate at selected angles to reflect unwanted light and to direct the required light toward the projection lens (410). Through visual persistence, lights of different colors are combined to form a color image for the human eye. In one example, a DLP controller (314) controls the DMD (408) to reduce the persistence of each pixel. Persistence may be referred to as the time each pixel remains lit. High persistence (e.g., 8.3 ms at 120 Hz) causes blurring and smearing of the images. The DLP controller (314) reduces the duration of each pixel to, for example, less than 3ms.

[0041] FIG. 5 illustrates a DLP controller (314) according to one example. The DLP controller (314) includes a motion color artifact compensation module (506) and a low persistence module (508). The motion color artifact compensation module (506) reduces color artifacts produced by motion of the AR wearable device (210). For example, as the user (206) moves their head (while the AR wearable device (210) is mounted on their head), the displayed virtual content will break up into its primary colors (RGB), or more precisely, a color sequence will become visible.

[0042] DLP projectors utilizing a mechanical spinning color wheel exhibit this color breakup, also known as the "rainbow effect." This is best described as short flashes of perceived red, blue, and green "shadows," most frequently observed when projected content features high-contrast areas of moving bright or white objects against a predominantly dark or black background. These brief visible separations of colors can also be evident when an observer rapidly moves their eyes across the projected image. Typically, the faster the user moves their eyes / head, the further the colors appear.

[0043] The motion color artifact compensation module (506) reduces or eliminates rainbow effects by compensating for color artifacts based on predictable data. In other words, the motion color artifact compensation module (506) predicts where and how colors will be rendered under user motion and compensates for motion-to-photon latency on a per-color basis. In one example, the motion color artifact compensation module (506) includes a color cycle module (502) and a color sequence module (504).

[0044] The color cycle module (502) configures the light source (402) to generate only one single repetition of the basic colors (RGB) per image frame. For example, a conventional light source (402) produces four color RGB-RGB-RGB-RGB per frame (e.g., at about 60 Hz). Multiple color cycles result in stuttering effects because the picture is shown four times at different positions. These stuttering effects are particularly exacerbated during head motion of the AR wearable device (210) while virtual content is being displayed.

[0045] The color sequence module (504) identifies or determines the color sequence of the light source (402). As previously mentioned, in a conventional DLP projector, as the user moves their head, the displayed virtual content will break up into its primary colors, or more precisely, the color sequence will become visible. For example, a simple RGB sequence will bleed its three colors. The faster the user moves their head, the further the colors will appear apart. High-frequency color sequences can be used to counteract the color bleeding. However, high frequencies can cause motion blur and illegible text. The color sequence module (504) identifies the color sequence (R, G, and B) of the light source (402) and counteracts the effect of color breakup based on the predicted color sequence for each frame.

[0046] The low persistence module (508) reduces the persistence of each pixel by controlling the DMD (408) to direct light from the light source (402) away from the projection lens (410). In one example, the low persistence module (508) reduces the persistence of each pixel, for example, to less than 3ms. In another example, the DLP controller (314) causes shifting of individual color planes by controlling the DMD (408) to display black (directing light away from the projection lens (410)) for 50% of the frame time.

[0047] Additionally, the DLP controller (314) can control the current for each LED of the light source (402) to provide a range of possible brightness. However, the provided current range may limit the upper or lower limit of the LED brightness. In some examples, a current range is provided to provide maximum LED brightness to achieve perceive brightness corresponding to outdoor ambient brightness. This can lead to a lower perceived brightness limit that is still greater than indoor ambient brightness, resulting in projections that are too bright for the user to use comfortably in an indoor setting. Therefore, a process of turning off the projector for a certain period (i.e., dark time) may be used to achieve lower perceived brightness in an indoor setting while maintaining the ability to achieve maximum perceived brightness in an outdoor setting (i.e., through maximum current and 0% dark time). Still, other methods may be used. For example, an RGB synthesizer may change the hue and color spectrum for any given color to be displayed to increase or decrease the perceived brightness of the color. Each of these methods, either individually or in combination, allows AR wearable devices to operate in a wide range of ambient light while maintaining low power consumption and extending battery life.

[0048] In one example, the DLP controller (314) can operate the LEDs of the light source (402) in 0% dark time (DT) mode or 50% DT mode. In 0% DT mode, the DLP controller (314) performs two repetitions for each desired color (e.g., RGBRGB). Additionally, the DLP controller (314) can drive the LEDs at a maximum current amount in 0% DT mode. This results in a maximum achievable effective brightness setting. In this implementation, 0% DT mode also consumes maximum power. To reduce power consumption and brightness, the DLP projector (326) can operate the LEDs of the light source (402) in 50% DT mode. In 50% DT mode, the DLP projector (326) performs one repetition for each desired color (e.g., RGB, dark, dark, dark). That is, in this mode, the LEDs are driven at half the number of repetitions as in 0% DT mode. To achieve the same level of effective or perceived brightness as in 0% DT mode in 50% DT mode, the current used to drive the LEDs needs to be doubled. However, certain projection elements of the light source (402) cannot be driven with such a high current without damaging the components.

[0049] One advantage of operating the LEDs of the light source (402) in 50% DT mode is that a lower level of minimum perceived brightness can be achieved than in 0% DT mode. That is, driving the LEDs with the minimum operating current in 50% DT mode results in a lower perceived effective brightness than when the LEDs are driven by the same minimum operating current in 0% DT mode. According to some implementations, the DLP projector (326) may choose between operating the LEDs of the light source (402) in 0% DT mode or 50% DT mode depending on the desired or calculated effective brightness value for a given real-world environment. Additionally, the DLP projector (326) may also modify the color values ​​of the pixels of the image received for display in the AR wearable device (210) to further decrease or increase the perceived effective brightness value. By using a combination of modifying software pixel values ​​based on current ambient light measurements (e.g., changing white color in an image to gray color pixels, or scaling or multiplying red, green, and blue pixel values ​​in an image to a specified value) and modifying hardware parameters of LEDs (e.g., changing the DT operation mode and modifying the current used to drive the LEDs), the effective brightness perceived by the user through the AR wearable device (210) can be more effectively set for different real-world environments, including bright outdoor environments and darker indoor environments. This allows the AR wearable device (210) to be used in a wider range of environments without negatively affecting the overall user experience.

[0050] In some examples, the DLP controller (314) receives a measurement of ambient light from an ambient light sensor of the AR wearable device (210). The measurement of ambient light may be received as a measurement in units of Lux (lx) used as one component of the effective brightness setting calculated by the DLP controller (314). In one example, the DLP controller (314) may automatically change or control the effective brightness setting of the AR wearable device (210). In one example, the DLP controller (314) may change or control the effective brightness setting of the AR wearable device (210) based on manual input control (e.g., based on inputs received directly from a user on the AR wearable device (210) or from an application implemented on a client device (211) communicating with the AR wearable device (210). The effective brightness setting may be specified between a range of 0 (minimum brightness value) and 255 (maximum brightness value). In certain implementations, the maximum brightness value may be reduced based on the current thermal temperature of the AR wearable device when the current thermal temperature is higher than or exceeds a specified threshold (e.g., from 255 to 204, which is 80% of the maximum brightness value). When the current thermal temperature returns below the specified threshold, the maximum brightness value returns to a default value, such as 255.

[0051] In one example, the DLP controller (314) determines that the measurement of ambient light is less than a first threshold value (e.g., 10 lx). In this case, the DLP controller (314) sets the effective brightness setting to a first value which is 30% of the maximum brightness setting. The DLP controller (314) may determine that the measurement of ambient light is in a first range (e.g., 10 lx to 200 lx). In this case, the DLP controller (314) may calculate the effective brightness setting as an interpolation between the first value (e.g., a value which is 30% of the maximum brightness setting) and the second value (e.g., a value which is 100% of the maximum brightness setting). For example, the DLP controller (314) may calculate the difference between the current brightness measurement and the first threshold value. Then, the DLP controller (314) may calculate the effective brightness value by linearly interpolating between the first value and the second value based on the calculated difference. For example, the DLP controller (314) may determine that the current measurement of ambient light is 50 lx, and in these cases, the DLP controller (314) may calculate the effective brightness value as 39% of the maximum brightness setting (linear interpolation from 30% to 100% based on the difference representing an increase to the first threshold of 40 lx). The DLP controller (314) may determine that the measurement of ambient light exceeds the maximum value of the first range (e.g., greater than 200 lx). In these cases, the DLP controller (314) may set the effective brightness setting to a second value (e.g., 100% of the maximum brightness setting).

[0052] In some examples, the DLP controller (314) may calculate an effective brightness value based on whether the lenses of the AR wearable device (210) are tinted or transparent. Specifically, the DLP controller (314) may determine that the lenses are tinted. In these cases, the DLP controller (314) may determine that the measurement of ambient light is less than a first threshold value (e.g., 10 lx). In response, the DLP controller (314) sets the effective brightness setting to a specific value, such as 0% of the maximum brightness setting. The DLP controller (314) may determine that the measurement of ambient light is in a second range (e.g., 25 lx to 300 lx). In these cases, the DLP controller (314) may calculate the effective brightness setting as an interpolation between a specific value (e.g., a value that is 0% of the maximum brightness setting) and a second value (e.g., a value that is 100% of the maximum brightness setting). For example, the DLP controller (314) can calculate the difference between the current brightness measurement and a first threshold value. Then, the DLP controller (314) can calculate an effective brightness value by linearly interpolating between a first value and a second value based on the calculated difference. For example, the DLP controller (314) can determine that the current measurement of ambient light is 50 lx, and in these cases, the DLP controller (314) can calculate the effective brightness value as 39% of the maximum brightness setting (linear interpolation from 30% to 100% based on the difference representing an increase to the first threshold value of 40 lx). The DLP controller (314) can determine that the measurement of ambient light exceeds the maximum value of a second range (e.g., greater than 200 lx or 300 lx). In these cases, the DLP controller (314) can set the effective brightness setting to the second value (e.g., 100% of the maximum brightness setting).

[0053] As another example, the DLP controller (314) may determine that the lenses are transparent. In these cases, the DLP controller (314) may determine that the ambient light measurement is less than a second threshold (e.g., 25 lx) which can be set greater than the first threshold used when the lenses are determined to be colored. That is, the DLP controller (314) may select between a first threshold (associated with colored lenses) and a second threshold (associated with transparent lenses) from a plurality of thresholds used to control effective brightness based on ambient light measurements. In response to selecting the second threshold, the DLP controller (314) sets the effective brightness setting to a specific value, such as 0% of the maximum brightness setting, when the ambient light measurement is less than the second threshold. The DLP controller (314) may determine that the ambient light measurement is in a third range (e.g., 10 lx to 200 lx). In these cases, the DLP controller (314) can calculate the effective brightness setting as an interpolation between a specific value (e.g., a value that is 0% of the maximum brightness setting) and a second value (e.g., a value that is 100% of the maximum brightness setting). For example, the DLP controller (314) can calculate the difference between the current brightness measurement and the second threshold value. Then, the DLP controller (314) can calculate the effective brightness value by linearly interpolating between the first value and the second value based on the calculated difference. For example, the DLP controller (314) can determine that the current measurement of ambient light is 50 lx, and in these cases, the DLP controller (314) can calculate the effective brightness value as 39% of the maximum brightness setting (linear interpolation from 30% to 100% based on the difference representing an increase to the second threshold value of 40 lx). The DLP controller (314) can determine that the measurement of ambient light exceeds the maximum value of the second range (e.g., greater than 200 lx or 300 lx).In these cases, the DLP controller (314) can set the effective brightness setting to a second value (e.g., 100% of the maximum brightness setting).

[0054] In some implementations, the DLP controller (314) may calculate an effective brightness setting for the AR wearable device (210) by obtaining a minimum value between the screen brightness offset by an animation scale (which may be set between 0 and 1) and the thermal limit of the AR wearable device (210). That is, if the screen brightness calculated in the manner described above (e.g., based on user input or based on measurements of ambient light received from an ambient light sensor) is lower than the maximum brightness associated with the thermal limit, the DLP controller (314) sets the brightness setting to the brightness value calculated based on user input or based on measurements of ambient light received from an ambient light sensor. If the screen brightness calculated in the manner described above is greater than or exceeds the maximum brightness associated with the thermal limit, the DLP controller (314) sets the brightness setting to the thermal limit value (e.g., 80% of the maximum brightness value or any number between 80% and 100%).

[0055] In some implementations, the DLP controller (314) changes the composition of the projector LED current, operating mode (e.g., 50% DT (50-DT) or 0% DT (0-DT)), and image RGB values ​​based on the calculated effective brightness setting. An exemplary way in which the DLP controller (314) changes these parameters is summarized in the table below:

[0056]

[0057] Table 1: Hardware and Software Modifications for Brightness Control

[0058] Specifically, the DLP controller (314) may access ambient temperature measurements to calculate an effective brightness value or receive input from a user setting an effective brightness value. In response to determining that the effective brightness value is set to a maximum brightness setting (e.g., 100%), the DLP controller (314) may drive the projection elements of the projector (e.g., LEDs) with a maximum current and set the operating mode to 0% DT. In this case, the DLP controller (314) may output an image on the AR wearable device (210) with the received RGB values ​​(e.g., without scaling the RGB values). In some cases, the DLP controller (314) may determine that the effective brightness value is set to a brightness amount between 80% and 100%. In these cases, the DLP controller (314) may drive the projection elements of the projector (e.g., LEDs) with a current that is linearly scaled with respect to the maximum current. That is, the lower the effective brightness level calculated to be between 80% and 100%, the lower the current is scaled linearly (for example, if the effective brightness level decreases from 100% to 95%—a 5% drop—the DLP controller (314) reduces the current by 5%). The DLP controller (314) can set the operation mode to 50% DT and output an image on the AR wearable device (210) with the received RGB values ​​(for example, without scaling the RGB values).

[0059] In some cases, the DLP controller (314) may determine that the effective brightness value is set to an amount of brightness between 80% and 0% (or a value slightly greater than 0%). In these cases, the DLP controller (314) may drive the projection elements (e.g., LEDs) of the projector with minimum current and set the operating mode to 50% DT. The DLP controller (314) may also linearly scale the RGB values ​​of the image on the AR wearable device (210). That is, the DLP controller (314) may modify the values ​​of the received image before displaying the image on the display of the AR wearable device (120) by scaling the RGB values ​​based on the amount by which the effective brightness is lowered compared to 80%. For example, RGB values ​​can be linearly scaled to a first value when the effective brightness is calculated to be 60%, and to a second value (lower than the first value) when the effective brightness is calculated to be 40%. The lowest brightness is achieved by driving the LEDs without current, setting the operation mode to 50-DT, and scaling the RGB values ​​of the image to 0%.

[0060] FIG. 6 illustrates a wearable device (e.g., glasses (600)) according to some examples. The glasses (600) may be any suitable type including glasses (100), and similar reference numbers have been used to describe similar components of the glasses (600) and glasses (100). For simplification, a portion of the glasses (600) is shown in FIG. 6. The glasses (600) include optical lenses (602) fixed within a left optical element holder (114) and a right optical element holder (120), respectively. Each of the optical lenses (602) has its own front (not shown) and opposite rear (604). The left end (110) and right end (124) of the front piece (134) may include their respective left frame extension (608) and right frame extension (614) extending rearward from their respective left end (110) and right end (124). As discussed above in relation to FIG. 1, a left temple piece (136) (not shown in FIG. 6) and a right temple piece (138) may be provided and fixedly fixed to their respective frame extensions (608) and frame extensions (614), or detachably attached to their respective frame extensions (608) and frame extensions (614). In one or more examples, connector mechanisms (616) are provided to fix the temple piece (136) and the temple piece (138) to their respective frame extensions (608) and frame extensions (614).

[0061] The glasses (600) include a computer (620) (e.g., a computing system) that may be similar to a computer (128) or a machine (800). In the example of FIG. 6, the computer (620) is powered by a suitable rechargeable battery (not shown) that may be similar to a battery (106). The computer (620) may be implemented using one or more of the processor elements of the AR wearable device (210), including a processor (308), and / or a storage device (306).

[0062] The glasses (600) additionally include sensors (618) (e.g., one or more cameras, ambient light sensors), which may resemble optical sensors (316) and / or other digital sensors, and may face inward (e.g., toward the user) and / or outward (e.g., away from the user). Data feeds from the sensors (618) are provided to a computer (620). In the example of FIG. 6, the computer (620) is placed within the first part (108) of the right temple piece (138), but the computer (620) may be placed elsewhere in alternative examples. In the example of FIG. 6, the right temple piece (138) includes a removable cover section (606) for access to connector mechanisms (616) or other electronic components of the glasses (600).

[0063] The glasses (600) include an optical assembly (622) for displaying images to the user. In the example of FIG. 6, one optical assembly (622) is shown, but in other examples, optical assemblies may be provided for both eyes of the user (e.g., for both the temple piece (136) and the temple piece (138). The optical assembly (622) includes an optical source, such as a projector (610), placed on one of the arms or temples of the glasses (e.g., the right temple piece (136) of the glasses (600)). In one or more examples, the projector (610) is a three-color laser projector using a scanning mirror or a galvanometer. A computer (620) is connected to the projector (610).

[0064] FIG. 7 is a flowchart of a process (700) according to some examples. The flowchart may describe the operations as a sequential process, but many of the operations may be performed in parallel or simultaneously. Furthermore, the order of the operations may be rearranged. The process terminates when its operations are completed. The process may correspond to a method, procedure, etc. The steps of the methods may be performed wholly or partially, and may be performed together with some or all of the steps in other methods, and may be performed by any number of different systems, or any part thereof, such as a processor included in any of the systems.

[0065] In operation 701, the DLP controller (314) commands the projection elements of the AR wearable device to project an image, as discussed above.

[0066] In operation 702, the DLP controller (314) receives a measurement of ambient light from an ambient light sensor, as discussed above.

[0067] In operation 703, the DLP controller (314) adjusts one or more hardware parameters of the projection elements of the AR wearable device based on measurements of ambient light, as discussed above.

[0068] In operation 704, the DLP controller (314) modifies one or more color values ​​of an image displayed by projection elements of an AR wearable device based on measurements of ambient light, as discussed above.

[0069] In operation 705, the DLP controller (314) projects an image with modified color values ​​using projection elements of an AR wearable device having one or more adjusted hardware parameters as discussed above.

[0070] Machine Architecture

[0071] FIG. 8 is a schematic representation of a machine (800) on which instructions (808) (e.g., software, program, application, applet, app, or other executable code) can be executed to cause the machine (800) to perform any one or more of the methodologies discussed herein. For example, instructions (808) may cause the machine (800) to perform any one or more of the methods described herein. Instructions (808) transform a general unprogrammed machine (800) into a specific machine (800) programmed to perform the described and illustrated functions in the described manner. The machine (800) may operate as a standalone device or may be coupled to other machines (e.g., networked). In a networked deployment, the machine (800) may operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine (800) may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smartphone, a mobile device, a wearable device (e.g., a smartwatch), a smart home device (e.g., a smart appliance), other smart devices, a web appliance, a network router, a network switch, a network bridge, or any machine capable of executing commands (808) that specify actions to be taken by the machine (800) sequentially or otherwise. Furthermore, although only a single machine (800) is exemplified, the term “machine” should also be considered to include a collection of machines that execute commands (808) individually or jointly to perform any one or more of the methodologies discussed herein.For example, the machine (800) may include any one of a plurality of server devices forming part of a client device (102) or a messaging server system (108). In some examples, the machine (800) may also include both client and server systems, and specific operations of a specific method or algorithm are performed on the server side and specific operations of a specific method or algorithm are performed on the client side.

[0072] The machine (800) may include processors (802), memory (804), and I / O (input / output) components (838) that can be configured to communicate with each other via a bus (840). In one example, the processors (802) (e.g., a CPU (Central Processing Unit), a RISC (Reduced Instruction Set Computing) processor, a CISC (Complex Instruction Set Computing) processor, a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an RFIC (Radio-Frequency Integrated Circuit), other processors, or any suitable combination thereof) may include, for example, a processor (806) and a processor (810) that execute instructions (808). The term "processor" is intended to include multi-core processors that may include two or more independent processors (sometimes referred to as "cores") capable of executing instructions simultaneously. FIG. 8 illustrates multiple processors (802), but the machine (800) may include a single processor having a single core, a single processor having multiple cores (e.g., a multi-core processor), multiple processors having a single core, multiple processors having multiple cores, or any combination thereof.

[0073] Memory (804) includes main memory (812), static memory (814), and storage unit (816), all of which are accessible to processors (802) via a bus (840). The main memory (804), static memory (814), and storage unit (816) store instructions (808) that implement any one or more of the methodologies or functions described herein. The instructions (808) may also exist, wholly or partially, during the execution by the machine (800), in the main memory (812), in the static memory (814), in a machine-readable medium within the storage unit (816), in at least one of the processors (802) (e.g., in the processor's cache memory), or any suitable combination thereof.

[0074] The I / O components (838) may include a wide variety of components for receiving inputs, providing outputs, generating outputs, transmitting information, exchanging information, capturing measurements, etc. The specific I / O components (838) included in a particular machine will depend on the type of machine. For example, portable machines such as mobile phones may include touch input devices or other such input mechanisms, while headless server machines are unlikely to include such touch input devices. It will be acknowledged that the I / O components (838) may include many other components not shown in FIG. 8. In various examples, the I / O components (838) may include user output components (824) and user input components (826). User output components (824) may include visual components (e.g., a display such as a plasma display panel (PDP), a light-emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), acoustic components (e.g., a speaker), haptic components (e.g., a vibration motor, a resistance mechanism), other signal generators, etc. User input components (826) may include alphanumeric input components (e.g., a keyboard, a touch screen configured to receive alphanumeric input, a photo-optical keyboard, or other alphanumeric input components), point-based input components (e.g., a mouse, a touchpad, a trackball, a joystick, a motion sensor, or other pointing mechanism), tactile input components (e.g., a physical button, a touch screen providing the location and force of a touch or touch gesture, or other tactile input components), audio input components (e.g., a microphone), etc.

[0075] In additional examples, I / O components (838) may include, among various other components, biometric components (828), motion components (830), environment components (832), or position components (834). For example, biometric components (828) include components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body gestures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), identifying a person (e.g., voice identification, retinal identification, face identification, fingerprint identification, or EEG-based identification), and doing similar things. Motion components (830) include acceleration sensor components (e.g., accelerometers), gravity sensor components, and rotation sensor components (e.g., gyroscopes).

[0076] Environmental components (832) may include, for example, one or more cameras (having still image / photo and video capabilities), light sensor components (e.g., photometers), temperature sensor components (e.g., one or more thermometers for detecting ambient temperature), humidity sensor components, pressure sensor components (e.g., barometers), acoustic sensor components (e.g., one or more microphones for detecting background noise), proximity sensor components (e.g., infrared sensors for detecting nearby objects), gas sensors (e.g., gas detection sensors for detecting concentrations of hazardous gases for safety or measuring contaminants in the atmosphere), or other components capable of providing indications, measurements, or signals corresponding to the surrounding physical environment.

[0077] With respect to the cameras, the client device (102) may have a camera system including, for example, front cameras located on the front of the client device (102) and rear cameras located on the rear of the client device (102). The front cameras may be used, for example, to capture still images and videos (e.g., "selfies") of the user of the client device (102), which may then be augmented with the augmentation data (e.g., filters) described above. The rear cameras may be used, for example, to capture still images and videos in a more traditional camera mode, and these images are similarly augmented with augmentation data. In addition to the front and rear cameras, the client device (102) may also include a 360° camera for capturing 360° photos and videos.

[0078] Additionally, the camera system of the client device (102) may include dual rear cameras (e.g., a depth-sensing camera as well as a main camera) on the front and rear sides of the client device (102), or even triple, quadruple, or quintuple rear camera configurations. These multiple camera systems may include, for example, a wide camera, an ultra-wide camera, a telephoto camera, a macro camera, and a depth sensor.

[0079] Position components (834) include position sensor components (e.g., GPS receiver components), altitude sensor components (e.g., altimeters or barometers that detect air pressure from which altitude can be derived), orientation sensor components (e.g., magnetometers), and similar ones.

[0080] Communication can be implemented using a wide variety of technologies. The I / O components (838) further include communication components (836) operable to connect the machine (800) to the network (820) or devices (822) through their respective combinations or connections. For example, the communication components (836) may include a network interface component, or other suitable device for interfacing with the network (820). In additional examples, the communication components (836) may include wired communication components, wireless communication components, cellular communication components, near-field communication (NFC) components, Bluetooth ® Components (e.g., Bluetooth) ® Low Energy), Wi-Fi ® It may include other communication components that provide communication through components and other aspects. The devices (822) may be other machines or any various peripheral devices (e.g., peripheral devices connected via USB).

[0081] Furthermore, the communication components (836) may include components capable of detecting identifiers or operable to detect identifiers. For example, the communication components (836) may include Radio Frequency Identification (RFID) tag reader components, NFC smart tag detection components, optical reader components (e.g., optical sensors for detecting one-dimensional barcodes such as Universal Product Code (UPC) barcodes, multi-dimensional barcodes such as Quick Response (QR) codes, Aztec codes, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCC RSS-2D barcodes, and other optical codes), or acoustic detection components (e.g., microphones for identifying tagged audio signals). In addition, various information, such as location via Internet Protocol (IP) geolocation, location via Wi-Fi® signal triangulation, and location via detection of NFC beacon signals that can indicate a specific location, can be derived through the communication components (836).

[0082] Various memories (e.g., main memory (812), static memory (814), and memory of the processors (802)) and storage unit (816) may store one or more sets of instructions and data structures (e.g., software) that implement any one or more of the methodologies or functions described herein. These instructions (e.g., instructions (808)) enable various operations to implement the disclosed examples when executed by the processors (802).

[0083] Commands (808) may be transmitted or received over a network (820) using a transmission medium through a network interface device (e.g., a network interface component included in the communication components (836)) and using any one of several well-known transmission protocols (e.g., HTTP (hypertext transfer protocol)). Similarly, commands (808) may be transmitted or received using a transmission medium through a connection to the devices (822) (e.g., peer-to-peer connection).

[0084] Software Architecture

[0085] FIG. 9 is a block diagram (900) illustrating a software architecture (904) that may be installed in any one or more of the devices described herein. The software architecture (904) is supported by hardware such as a machine (902) that includes processors (920), memory (926), and I / O components (938). In this example, the software architecture (904) may be conceptualized as a stack of layers, each layer providing a specific function. The software architecture (904) includes layers such as an operating system (912), libraries (910), frameworks (908), and applications (906). Operationally, applications (906) invoke API calls (950) through the software stack and receive messages (952) in response to the API calls (950).

[0086] The operating system (912) manages hardware resources and provides common services. The operating system (912) includes, for example, a kernel (914), services (916), and drivers (922). The kernel (914) acts as an abstraction layer between the hardware and other software layers. For example, the kernel (914) provides, among other functions, memory management, processor management (e.g., scheduling), component management, networking, and security settings. Services (916) may provide other common services for other software layers. Drivers (922) are responsible for controlling or interfacing with the underlying hardware. For example, drivers (922) may include display drivers, camera drivers, BLUETOOTH® or BLUETOOTH® Low Energy drivers, flash memory drivers, serial communication drivers (e.g., USB drivers), WI-FI® drivers, audio drivers, power management drivers, etc.

[0087] Libraries (910) provide common low-level infrastructure used by applications (906). Libraries (910) may include system libraries (918) (e.g., C standard library) that provide functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. Additionally, the libraries (910) may include API libraries (924) such as media libraries (e.g., libraries that support the presentation and manipulation of various media formats such as MPEG4 (Moving Picture Experts Group-4), Advanced Video Coding (H.264 or AVC), MP3 (Moving Picture Experts Group Layer-3), AAC (Advanced Audio Coding), AMR (Adaptive Multi-Rate) audio codecs, Joint Photographic Experts Group (JPEG or JPG), or PNG (Portable Network Graphics), graphics libraries (e.g., OpenGL frameworks used to render graphic content on a display in two dimensions (2D) and three dimensions (3D)), database libraries (e.g., SQLite providing various relational database functions), web libraries (e.g., WebKit providing web browsing functionality). The libraries (910) may also include a wide variety of other libraries (928) that provide many different APIs to applications (906).

[0088] Frameworks (908) provide common high-level infrastructure used by applications (906). For example, frameworks (908) provide various graphical user interface (GUI) functions, high-level resource management, and high-level location services. Frameworks (908) may provide a wide range of other APIs that can be used by applications (906), some of which may be specific to a particular operating system or platform.

[0089] In one example, applications (906) may include a wide range of other applications such as a home application (936), a contacts application (930), a browser application (932), a book reader application (934), a location application (942), a media application (944), a messaging application (946), a game application (948), and an external application (940). Applications (906) are programs that execute functions defined in programs. Various programming languages ​​may be used to create one or more of the applications (906), which are structured in various ways, such as object-oriented programming languages ​​(e.g., Objective-C, Java, or C++) or procedural programming languages ​​(e.g., C or assembly language). In a specific example, an external application (940) (e.g., an application developed using an ANDROID™ or IOS™ software development kit (SDK) by an entity other than the vendor of a specific platform) may be mobile software running on a mobile operating system such as IOS™, ANDROID™, WINDOWS® Phone, or other mobile operating systems. In this example, the external application (940) may invoke API calls (950) provided by the operating system (912) to facilitate the functions described herein.

[0090] Glossary

[0091] "Carrier signal" refers to any intangible medium capable of storing, encoding, or carrying instructions for execution by a machine, and includes digital or analog communication signals or other intangible media to facilitate the communication of such instructions. Instructions may be transmitted or received over a network using a transmission medium through a network interface device.

[0092] "Client device" refers to any machine that interfaces with a communication network to obtain resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop, PDAs (portable digital assistants), smartphones, tablets, ultrabooks, netbooks, laptops, multi-processor systems, microprocessor-based or programmable consumer electronics, game consoles, set-top boxes, or any other communication device that a user can use to access the network.

[0093] "Communication network" refers to one or more parts of a network that may be an ad-hoc network, intranet, extranet, VPN (virtual private network), LAN (local area network), wireless LAN (WLAN), WAN (wide area network), wireless WAN (WWAN), MAN (metropolitan area network), the Internet, part of the Internet, part of the PSTN (Public Switched Telephone Network), POTS (plain old telephone service) network, cellular telephone network, wireless network, Wi-Fi® network, other types of networks, or a combination of two or more of these networks. For example, a network or part of a network may include a wireless or cellular network, and a coupling may be a CDMA (Code Division Multiple Access) connection, a GSM (Global System for Mobile communications) connection, or other types of cellular or wireless coupling.In this example, the combination can implement any of the various types of data transmission technologies, such as 1xRTT (Single Carrier Radio Transmission Technology), EVDO (Evolution-Data Optimized) technology, GPRS (General Packet Radio Service) technology, EDGE (Enhanced Data rates for GSM Evolution) technology, 3GPP (third Generation Partnership Project) including 3G, 4th generation wireless (4G) networks, UMTS (Universal Mobile Telecommunications System), HSPA (High Speed ​​Packet Access), WiMAX (Worldwide Interoperability for Microwave Access), LTE (Long Term Evolution) standards, other things defined by various standard-setting organizations, other long-range protocols, or other data transmission technologies.

[0094] "Component" refers to a device, physical entity, or logic having boundaries defined by function or subroutine calls, branch points, APIs, or other technologies that provide the division or modularization of specific processing or control functions. Components can be combined with other components through their interfaces to complete machine processes. A component may be a packaged functional hardware unit designed to be used with other components and, typically, a part of a program that performs a specific function among the related functions.

[0095] Components may comprise software components (e.g., code embodied on a machine-readable medium) or hardware components. A "hardware component" is a tangible unit capable of performing specific operations and may be configured or arranged in a specific physical manner. In various examples, one or more computer systems (e.g., a standalone computer system, a client computer system, or a server computer system) or one or more hardware components of a computer system (e.g., a processor or a group of processors) may be configured by software (e.g., an application or part of an application) as hardware components that operate to perform specific operations as described herein.

[0096] Hardware components may also be implemented mechanically, electronically, or any suitable combination thereof. For example, a hardware component may include dedicated circuits or logic that are permanently configured to perform specific operations. A hardware component may be a special-purpose processor, such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). A hardware component may also include programmable logic or circuits that are temporarily configured by software to perform specific operations. For example, a hardware component may include software executed by a general-purpose processor or another programmable processor. Once configured by such software, the hardware components become specific machines (or specific components of machines) uniquely customized to perform the configured functions and are no longer general-purpose processors. It will be recognized that the decision to implement a hardware component mechanically, in dedicated, permanently configured circuits, or in temporarily configured circuits (e.g., configured by software) may be driven by cost and time considerations. Accordingly, the phrase “hardware component” (or “hardware-implemented component”) should be understood to encompass tangible entities as long as they are physically configured, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a particular way or to perform the specific operations described herein.

[0097] When considering examples where hardware components are configured (e.g., programmed) temporarily, it is not necessary for each hardware component to be configured or instantiated at any single time instance. For example, if a general-purpose processor is configured by software to be a special-purpose processor, the general-purpose processor may be configured as different special-purpose processors at different times (e.g., including different hardware components). Thus, software configures a specific processor or processors to configure a specific hardware component at one time instance and a different hardware component at a different time instance.

[0098] Hardware components can provide information to other hardware components and receive information from them. Thus, the described hardware components can be considered as being communicably coupled. When multiple hardware components exist simultaneously, communication can be achieved through the transmission of signals between two or more of the hardware components or between two or more (e.g., via appropriate circuits and buses). In examples where multiple hardware components are configured or instantiated at different times, communication between such hardware components can be achieved, for example, through the storage and retrieval of information within memory structures accessible to multiple hardware components. For example, one hardware component may perform an operation and store the output of that operation in a memory device to which it is communicably coupled. Subsequently, additional hardware components may access the memory device to retrieve and process the stored output. Hardware components may also initiate communication with input or output devices and operate on resources (e.g., a collection of information).

[0099] Various operations of the exemplary methods described herein may be performed at least partially by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors may constitute processor-implemented components that operate to perform one or more operations or functions described herein. As used herein, "processor-implemented component" refers to a hardware component implemented using one or more processors. Similarly, the methods described herein may be implemented at least partially by a processor, and specific processors or processors are examples of hardware. For example, at least some of the operations of the methods may be performed by one or more processors (802) or processor-implemented components. Furthermore, one or more processors may also operate to support the performance of relevant operations in a "cloud computing" environment or as "software as a service" (SaaS). For example, at least some of the operations may be performed by a group of computers (as examples of machines including processors), and these operations may be accessible via a network (e.g., the Internet) and through one or more appropriate interfaces (e.g., APIs). The performance of specific operations may not only reside within a single machine but may also be distributed among processors deployed across multiple machines. In some examples, processors or components implemented by processors may be located in a single geographical location (e.g., a home environment, an office environment, or within a server farm).In other examples, processors or components implemented by processors may be distributed across multiple geographical locations.

[0100] "Computer-readable storage medium" refers to both machine storage media and transmission media. Accordingly, the terms include both storage devices / medias and carriers / modulated data signals. The terms "machine-readable medium," "computer-readable medium," and "device-readable medium" mean the same thing and may be used interchangeably in this disclosure.

[0101] An "ephemeral message" refers to a message accessible for a time-limited duration. Ephemeral messages can be text, images, videos, etc. The access time for an ephemeral message can be set by the message sender. Alternatively, the access time can be a default setting or a setting specified by the recipient. Regardless of the setting method, the message is transitory.

[0102] "Machine storage medium" refers to a single or multiple storage devices and media (e.g., centralized or distributed databases, and associated caches and servers) that store executable instructions, routines, and data. Accordingly, the term should be considered to include, but not be limited to, solid-state memories including memory inside or outside processors, and optical and magnetic media. Specific examples of machine storage medium, computer storage medium, and device storage medium include, by example, non-volatile memory including semiconductor memory devices, e.g., EPROM (erasable programmable read-only memory), EEPROM (electrically erasable programmable read-only memory), FPGAs, and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage medium," "device storage medium," and "computer storage medium" mean the same thing and may be used interchangeably in this disclosure. The terms "machine storage medium," "computer storage medium," and "device storage medium" specifically exclude carrier waves, modulated data signals, and other such media, at least some of which are included under the term "signal medium."

[0103] "Non-transient computer-readable storage medium" refers to a type of medium capable of storing, encoding, or carrying instructions for execution by a machine.

[0104] "Signal medium" refers to any intangible medium capable of storing, encoding, or carrying instructions for execution by a machine, and includes digital or analog communication signals or other intangible media for facilitating the communication of software or data. The term "signal medium" should be considered to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal in which one or more of its characteristics are set or altered, such as in the matter of encoding information within the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

[0105] Changes and modifications to the disclosed examples may be made without departing from the scope of the present disclosure. These and other changes or modifications are intended to be included within the scope of the present disclosure as expressed in the following claims.

Claims

Claim 1 A method comprising: a step of causing projection elements of an augmented reality (AR) wearable device to project an image by means of one or more processors of the AR wearable device; a step of receiving a measure of ambient light from an ambient light sensor by means of one or more processors of the AR wearable device; a step of adjusting one or more hardware parameters of the projection elements of the AR wearable device based on the measure of ambient light; a step of selecting between operating the projection elements at 0% Dark Time (DT) or 50% DT based on an effective brightness value, wherein the effective brightness value is calculated based on the measure of ambient light; a step of modifying one or more color values ​​of the image projected by the projection elements of the AR wearable device based on the measure of ambient light; and a step of causing the image to project with one or more modified color values ​​using the projection elements of the AR wearable device having the one or more adjusted hardware parameters. Claim 2 In claim 1, the method wherein the AR wearable device comprises electronic glasses. Claim 3 A method according to claim 1 or 2, wherein one or more hardware parameters are adjusted based on the effective brightness value and one or more color values ​​are modified. Claim 4 In paragraph 3, the step of calculating the effective brightness value comprises: receiving input from a user for adjusting a brightness setting on an application implemented on a mobile device associated with the AR wearable device, wherein the effective brightness is calculated independently of the ambient light in response to receiving input from the user. Claim 5 A method according to claim 1 or 2, wherein the step of calculating the effective brightness value comprises: determining that the lenses of the AR wearable device include a tinted lens or a clear lens; selecting a first threshold value from a plurality of threshold values ​​to set the screen brightness value based on determining that the lenses of the AR wearable device include the tinted lens; and selecting a second threshold value from a plurality of threshold values ​​to set the screen brightness value based on determining that the lenses of the AR wearable device include the clear lens. Claim 6 A method according to claim 5, further comprising, in response to determining that the lenses of the AR wearable device include the tinted lenses: a step of comparing the measurement of the ambient light with the first threshold value among the plurality of threshold values; and a step of setting the screen brightness value to a given value based on comparing the measurement of the ambient light with the first threshold value. Claim 7 A method according to claim 6, further comprising: a step of setting the screen brightness value to 0 in response to determining that the measurement of ambient light is smaller than the first threshold value; a step of setting the screen brightness value to an interpolation between the minimum value and the maximum value in response to determining that the measurement of ambient light is larger than the first threshold value and within a value range; and a step of setting the screen brightness value to the maximum value in response to determining that the measurement of ambient light is larger than the third threshold value. Claim 8 In claim 5, the step of calculating the effective brightness value comprises: in response to determining that the lenses of the AR wearable device include the transparent lens; comparing the measurement of the ambient light with the second threshold value among the plurality of threshold values; and setting the screen brightness value to a given value based on comparing the measurement of the ambient light with the second threshold value. Claim 9 A method according to claim 1 or 2, further comprising: a step of determining that the thermal limit of the AR wearable device has been reached; and a step of preventing the effective brightness value from exceeding the maximum brightness value associated with the thermal limit. Claim 10 In claim 9, the method wherein the maximum brightness value comprises 80% or less. Claim 11 A method according to claim 9, wherein the effective brightness value is calculated as the minimum value between the screen brightness value calculated based on the measurement of the ambient light and the maximum brightness value associated with the thermal limit. Claim 12 A method according to claim 1 or 2, further comprising: a step of determining that the effective brightness value is within a first value range; and a step of linearly varying the light-emitting diode (LED) current values ​​of the projection elements between a maximum current and a minimum current in response to determining that the effective brightness value is within the first value range. Claim 13 A method according to claim 12, further comprising: a step of determining that the effective brightness value is less than the first value range; and a step of multiplying the red, green, and blue pixel values ​​of the image by a specified value in response to determining that the effective brightness value is less than the first value range. Claim 14 delete Claim 15 A method according to claim 1, wherein the projection elements are operated at 0% DT in response to determining that the effective brightness value is the maximum effective brightness value, and current is driven to the projection elements at the maximum value. Claim 16 A method according to claim 15, wherein the projection elements are operated at the 50% DT in response to determining that the effective brightness value is less than the maximum effective brightness value and greater than the second value, current is driven in the projection elements in a linear range between the minimum and maximum values, and the red, green, and blue pixel values ​​of the image are linearly scaled down. Claim 17 A method according to claim 16, wherein the projection elements are operated at the 50% DT in response to determining that the effective brightness value corresponds to the second value, the current driven in the projection elements in a linear range corresponds to the minimum value, and the red, green, and blue pixel values ​​of the image are multiplied by 0. Claim 18 A system comprising one or more processors of an augmented reality (AR) wearable device configured to perform operations, wherein the operations include: causing projection elements of the AR wearable device to project an image; receiving a measurement of ambient light from an ambient light sensor; adjusting one or more hardware parameters of the projection elements of the AR wearable device based on the measurement of ambient light; selecting between operating the projection elements at 0% Dark Time (DT) or 50% DT based on an effective brightness value, wherein the effective brightness value is calculated based on the measurement of ambient light; modifying one or more color values ​​of the image projected by the projection elements of the AR wearable device based on the measurement of ambient light; and causing the image to project with the modified one or more color values ​​using the projection elements of the AR wearable device having the one or more adjusted hardware parameters. Claim 19 In paragraph 18, the AR wearable device comprises a system including electronic glasses. Claim 20 A non-transient computer-readable medium comprising instructions, wherein the instructions configure the AR wearable device to perform operations when executed by one or more processors of the augmented reality (AR) wearable device, said operations comprising: causing projection elements of the AR wearable device to project an image; receiving a measurement of ambient light from an ambient light sensor; adjusting one or more hardware parameters of said projection elements of the AR wearable device based on said measurement of ambient light; selecting between operating said projection elements at 0% Dark Time (DT) or 50% DT based on an effective brightness value—said effective brightness value is calculated based on said measurement of ambient light—; and modifying one or more color values ​​of said image projected by said projection elements of the AR wearable device based on said measurement of ambient light. A non-transient computer-readable medium comprising projecting the image with the modified one or more color values ​​using the projection elements of the AR wearable device having the one or more adjusted hardware parameters.

Images