Lock screen for head-worn augmented reality devices

Abstract

A method for activating a head-worn device is disclosed. The method includes the steps of activating the head-worn device and performing a partial startup. When a command to perform a function is received, it is determined whether the command allows for partial startup execution. The command is executed based on the command that allows for partial startup execution, and the startup of the head-worn device is completed based on the command that does not allow for partial startup execution. The method may further include the steps of determining whether user authentication is required for command execution, and executing the command based on the fact that the command is partially startup compatible and does not require user authentication.

Description

Technology Field

[0001] preference

[0002] This application claims the benefit of priority to U.S. Patent Application No. 17 / 660,359 filed April 22, 2022, the entirety of which is incorporated herein by reference.

[0003] Technology field

[0004] The present disclosure generally relates to head-worn devices having displays for augmented or virtual reality. Background Technology

[0005] A head-worn device may be implemented as a transparent or translucent display through which the user of the head-worn device can observe the surrounding environment. These devices allow the user to view the surrounding environment through the transparent or translucent display and also to view objects (e.g., virtual objects such as 3D renderings, images, videos, text, etc.) that are created to appear as part of the surrounding environment and / or be displayed overlaid on the surrounding environment. This is typically referred to as "augmented reality."

[0006] A head-worn device may additionally completely block the user's visual field and display a virtual environment through which the user can move or be moved. This is typically referred to as "virtual reality." As used herein, the terms "augmented reality" or "AR" refer to both augmented reality and virtual reality as traditionally understood, unless the context indicates otherwise.

[0007] Users of head-worn devices can access messaging or social network applications to view content or share it with other users of the application. In some cases, users can view live or stored content and enhance or modify it. That is, images, videos, or other media for enhancement may be captured from a live camera or retrieved from a local or remote data store.

[0008] In many cases, head-worn devices can be associated with mobile devices running related applications. Mobile devices typically include a lock screen for security purposes, and the lock screen is activated when the device enters an inactive or low-activity state, for example, based on the expiration of a timer or based on the user pressing a button to turn off the mobile device's display. Current lock screens implemented on mobile devices may present difficulties when paired with devices such as head-worn devices. Brief explanation of the drawing

[0009] To easily identify the discussion of any specific element or action, the top digit or number of the reference number refers to the drawing number where the element is first introduced. FIG. 1 is a perspective view of a head-wearing device including glasses, according to some examples. FIG. 2 illustrates additional aspects of the head-wearing device of FIG. 1 according to some examples. FIG. 3 is a block diagram illustrating a networked system including details of the head-wearing device of FIG. 1 according to some examples. FIG. 4 is a block diagram illustrating the architecture of glasses in more detail, according to some examples. FIGS. 5a and 5b illustrate flowcharts for initializing the lock screen during bootup of glasses according to some examples. FIGS. 6a and FIGS. 6b illustrate flowcharts for user unlocking of glasses according to some examples. FIGS. 7a and 7b illustrate flowcharts for setting a PIN on non-security glasses according to some examples. FIG. 8 illustrates a flowchart for changing settings on safety glasses according to some examples. FIG. 9 illustrates a flowchart for erasing a PIN on security glasses according to some examples. FIGS. 10a and FIGS. 10b illustrate a flowchart for pairing a client device with glasses, according to some examples. FIGS. 11a and FIGS. 11b illustrate flowcharts regarding authorizations associated with the security and boot status of glasses according to some examples. FIG. 12 illustrates a flowchart for PIN entries on security glasses according to some examples. Figure 13 illustrates a lock screen UI used on glasses in some examples. FIG. 14 is a flowchart illustrating operations performed by glasses and a client device according to some examples. FIG. 15 is a schematic representation of a networked environment in which the present disclosure may be arranged, according to some examples. FIG. 16 is a block diagram illustrating a software architecture in which the present disclosure can be implemented, according to some examples. FIG. 17 is a schematic representation of a machine in the form of a computer system in which a set of instructions can be executed to enable the machine to perform any one or more of the methodologies discussed, according to some examples. Specific details for implementing the invention

[0010] Known head-worn devices, such as AR glasses, include a transparent or translucent display that allows a user to view through the transparent or translucent display to observe the surrounding environment. Additional information or objects (virtual objects such as 3D renderings, images, videos, text, etc.) are presented on the display and appear as part of the surrounding environment and / or overlaid on the surrounding environment to provide the user with an augmented reality experience. The display may include a waveguide that receives a light beam from, for example, a projector, but any suitable display may be used to present augmented or virtual content to the wearer.

[0011] Navigation of information or the user interface of a head-worn device may, in some cases, be provided by voice commands or by input to an associated mobile device, such as a smartphone. In some examples, a touchpad is provided on the head-worn device, which can be used to provide XY touch and tap input to the head-worn device. Sophisticated head-worn devices can now perform or initiate tasks previously associated with smartphones, such as browsing, watching, and capturing media, selecting recipients for captured media, and forwarding captured media to selected recipients. Providing appropriate and harmonized lock screen functionality to an associated smartphone can be difficult.

[0012] Head-worn devices are also resource-constrained devices, for example with regard to memory, storage, and battery power, and are sensitive to latency to ensure a satisfactory user experience. Head-worn devices need to operate in various environments in an efficient manner regarding resource consumption and battery usage. Additionally, memory usage may fluctuate, for example, when a user is capturing or forwarding media. A critical operation is the system startup of the head-worn device and associated mobile device, which can be associated with increased resource consumption and system pressure caused by operating systems starting multiple services simultaneously. This can lead to numerous undesirable side effects, such as the system terminating in an out-of-memory state or becoming unresponsive.

[0013] Due to limitations on user interactions with the head-worn device, a significant portion of the user interface runs on the paired mobile device. Additionally, multiple processes and processors exist operating on both the head-worn device and the paired mobile device. Security states need to be synchronized across the paired devices and processors. The main processor of the head-worn device also frequently shuts down, presenting additional challenges regarding startup time, security, and encryption.

[0014] To address these technical issues, in some examples, security settings can be modified on the mobile device, while enforcement of the security settings takes place on the head-worn device. The security software architecture allows for changes to security settings on the mobile device and keeps them synchronized across multiple processors. By separating data into user-sensitive and non-sensitive data, restarting the main processor of the head-worn device after periods of inactivity is simplified. All functions that do not require user-sensitive data are available with reduced latency by allowing related services to operate in this limited-function mode. Subsequently, the head-worn device appropriately transitions to full-operation mode after the combination of the head-worn device and the mobile device is unlocked by the user.

[0015] In some examples, a method is provided that is executed by one or more processors within a head-worn device comprising one or more display devices, and the method comprises receiving an activation input, performing a partial startup of the head-worn device, receiving a command to perform a function, determining whether the command allows partial startup execution, executing the command based on whether the command allows partial startup execution, and completing the startup of the head-worn device based on whether the command does not allow partial startup execution. The method may further comprise determining whether user authentication is required for command execution, and executing the command based on whether the command is partial startup compatible and does not require user authentication. Partial startup of the head-worn device may include starting a lock screen service and a camera service.

[0016] Additionally, the method may include a step of determining whether user authentication is required for command execution, and a step of executing a user authentication procedure based on a command requiring user authentication, regardless of whether the command is partially bootable compatible. The method may also include a step of verifying valid user authentication, a step of completing the startup of a head-worn device, and a step of executing a command.

[0017] Verifying valid user authentication may include receiving a password from a relevant mobile device. Based on the fact that valid user authorization is not verified using the password received from the relevant mobile device, the method may further include the step of prompting user input of the password via a head-worn device. Additionally, the authenticated state of the head-worn device may be maintained based on the proximity of the head-worn device to the relevant mobile device.

[0018] In some examples, a head-wearing device is provided that includes one or more cameras, one or more display devices, and one or more processors. The head-wearing device also includes a memory that stores instructions, and the device is configured to perform any of the aforementioned methods and limitations, which, when the instructions are executed by one or more processors, include but are not limited to the operation of receiving an activation input, the operation of performing a partial startup of the head-wearing device, the operation of receiving a command to perform a function, the operation of determining whether the command allows partial startup execution, the operation of executing the command based on whether the command allows partial startup execution, and the operation of completing the startup of the head-wearing device based on whether the command does not allow partial startup execution.

[0019] In some examples, a non-transient computer-readable storage medium is provided, and the computer-readable storage medium includes instructions that, when executed by a head-wearing device comprising one or more display devices and one or more cameras, cause the head-wearing device to perform actions according to any of the aforementioned methods and limitations, including but not limited to: performing a partial startup of the head-wearing device; receiving a command to perform a function; determining whether the command allows partial startup execution; executing the command based on whether the command allows partial startup execution; and completing the startup of the head-wearing device based on whether the command does not allow partial startup execution.

[0020] A person skilled in the art can easily see other technical features from the following drawings, descriptions, and claims.

[0021] FIG. 1 is a perspective view of a head-worn device (e.g., glasses (100)) according to some examples. The glasses (100) may include a frame (102) made of any suitable material, such as plastic or metal, including any suitable shape memory alloy. In one or more examples, the frame (102) includes a first or left optical element holder (104) (e.g., a display or lens holder) and a second or right optical element holder (106) connected by a bridge (112). The first or left optical element (108) and the second or right optical element (110) may be provided within the respective left optical element holder (104) and right optical element holder (106). Each of the right optical element (110) and the left optical element (108) may be a lens, a display, a display assembly, or a combination of the foregoing. Any suitable display assembly may be provided in the glasses (100).

[0022] The frame (102) additionally includes a left arm or temple (120) and a right arm or temple (122). In some examples, the entire frame (102) may be formed from a single piece of material to have a single or integral configuration.

[0023] The glasses (100) may include a computing device such as a computer (118), which may be of any suitable type to be held by the frame (102) and, in one or more examples, may be of a size and shape suitable for being partially placed on at least one of the temples (120) and temples (122). The computer (118) may include one or more processors having memory, wireless communication circuits, and power. As discussed below, the computer (118) includes low-power circuits, high-speed circuits, and display processors. Various examples may include these elements in different configurations or integrate them together in different ways. Further details of the embodiments of the computer (118) may be implemented as exemplified by the data processor (302) discussed below.

[0024] The computer (118) additionally includes a battery (116) or other suitable portable power supply. In some examples, the battery (116) is placed on the left temple (120) and electrically coupled to the computer (118) placed on the right temple (122). The glasses (100) may include a connector or port (not shown) suitable for charging the battery (116), a wireless receiver, a transmitter or transceiver (not shown), or a combination of these devices.

[0025] The glasses (100) include cameras (114). Although two cameras are illustrated, some examples consider the use of a single or additional (i.e., more than two) cameras. In one or more examples, the glasses (100) include any number of input sensors or other input / output devices in addition to the cameras (114). These sensors or input / output devices may further include biometric sensors, position sensors, motion sensors, etc.

[0026] The glasses (100) may also include a touchpad (124) mounted on or integrated with one or both of the left temple (120) and the right temple (122). The touchpad (124) is generally arranged vertically and, in some examples, is approximately parallel to the user's temple. As used herein, "generally arranged vertically" means that the touchpad is at least more vertical than horizontal, but preferably more vertical. In the illustrated embodiment, additional user input may be provided by one or more buttons (126) provided on the outer upper edge of the left optical element holder (104) and the right optical element holder (106). One or more touchpads (124) and buttons (126) provide a means for the glasses (100) to receive input from the user of the glasses (100).

[0027] FIG. 2 illustrates glasses (100) from the perspective of a wearer. For clarity, many of the elements shown in FIG. 1 have been omitted. As described in FIG. 1, the glasses (100) shown in FIG. 2 include a left optical element (108) and a right optical element (110) fixed respectively within a left optical element holder (104) and a right optical element holder (106).

[0028] The glasses (100) include a front optical assembly (202) including a right projector (204) and a right near-eye display (206), and a front optical assembly (210) including a left projector (212) and a near-eye display (216).

[0029] In one embodiment, near-eye displays are waveguides. A waveguide includes reflective or diffractive structures (e.g., optical elements such as gratings and / or mirrors, lenses or prisms). Light (208) emitted by a projector (204) meets the diffractive structures of a waveguide of a near-eye display (206), which directs the light toward the user's right eye to provide an image that overlays the view of the real world seen by the user on or within the right optical element (110). Similarly, light (214) emitted by a projector (212) meets the diffractive structures of a waveguide of a near-eye display (216), which directs the light toward the user's left eye to provide an image that overlays the view of the real world seen by the user on or within the left optical element (108).

[0030] However, it will be understood that other display technologies or configurations capable of displaying an image to the user in the forward field of view may be provided. For example, instead of a projector (204) and a waveguide, an LCD, LED, or other display panel or surface may be provided instead.

[0031] When in use, information, content, and various user interfaces will be presented to the wearer of the glasses (100) on near-eye displays. As described in more detail below, the user may then interact with the glasses (100) using a touchpad (124) and / or buttons (126), in addition to providing voice inputs or touch inputs on an associated device, e.g., the client device (328) illustrated in FIG. 3.

[0032] FIG. 3 is a block diagram illustrating a networked system (300) including details of glasses (100) according to some examples.

[0033] The networked system (300) includes glasses (100), a client device (328), and a server system (332). The client device (328) may be a smartphone, tablet, phablet, laptop computer, access point, or any other such device that can be connected to the glasses (100) using both a low-power wireless connection (336) and a high-speed wireless connection (334). The client device (328) is connected to the server system (332) via a network (330). The network (330) may include any combination of wireless and wired connections. The server system (332) may be one or more computing devices as part of a service or network computing system. Any elements of the client device (328), the server system (332), and the network (330) may be implemented using the details of the software architecture (1604) or machine (1700) described in FIGS. 16 and FIGS. 17.

[0034] The glasses (100) include a data processor (302), displays (310), one or more cameras (308), and additional input / output elements (316). The input / output elements (316) may include microphones, audio speakers, biometric sensors, additional sensors, or additional display elements integrated with the data processor (302). Examples of the input / output elements (316) are further discussed in relation to FIGS. 16 and FIGS. 17. For example, the input / output elements (316) may include any of the I / O components (1706), such as output components (1728) and motion components (1736). Examples of the displays (310) are discussed in FIG. 2. In the specific examples described herein, the displays (310) include displays for each of the user's left and right eyes.

[0035] The data processor (302) includes an image processor (306) (e.g., a video processor), a GPU and display driver (338), a tracking module (340), an interface (312), a low-power circuit (304), and a high-speed circuit (320). The components of the data processor (302) are interconnected by a bus (342).

[0036] The interface (312) refers to any source of user commands provided to the data processor (302). In one or more examples, the interface (312) is a physical button that transmits a user input signal from the interface (312) to the low-power processor (314) when pressed. Pressing and then immediately releasing this button may be processed by the low-power processor (314) as a request to capture a single image, or vice versa. Pressing this camera button during a first time period may be processed by the low-power processor (314) as a request to capture video data while the button is pressed and to stop video capture when the button is released, and the video captured while the button is pressed is saved as a single video file. Alternatively, pressing the button for an extended period may capture a still image. In some examples, the interface (312) may be any mechanical switch or physical interface capable of receiving user inputs associated with requests for data from the camera (308). In some examples, the interface (312) may have a software component or may be associated with a command received wirelessly from another source, such as a client device (328).

[0037] The image processor (306) includes circuitry for receiving signals from the camera (308) and processing the corresponding signals from the camera (308) into a format suitable for storing them in memory (324) or transmitting them to a client device (328). In one or more examples, the image processor (306) (e.g., video processor) includes a microprocessor integrated circuit (IC) customized to process sensor data from the camera (308), along with volatile memory used by the operating microprocessor.

[0038] The low-power circuit (304) includes a low-power processor (314) and a low-power wireless circuit (318). These elements of the low-power circuit (304) may be implemented as separate elements or may be implemented on a single IC as part of a system on a single chip. The low-power processor (314) includes logic for managing other elements of the glasses (100). As previously described, for example, the low-power processor (314) may receive user input signals from the interface (312). The low-power processor (314) may also be configured to receive input signals or command communications from a client device (328) via a low-power wireless connection (336). The low-power wireless circuit (318) includes circuit elements for implementing a low-power wireless communication system. Bluetooth™ Smart, also known as low-energy Bluetooth™, is a standard implementation of a low-power wireless communication system that can be used to implement the low-power wireless circuit (318). In some examples, other low-power communication systems may be used.

[0039] The high-speed circuit (320) includes a high-speed processor (322), memory (324), and a high-speed wireless circuit (326). The high-speed processor (322) may be any processor capable of managing high-speed communication and operation of any general-purpose computing system required by the data processor (302). The high-speed processor (322) includes processing resources required to manage high-speed data transmissions over the high-speed wireless connection (334) using the high-speed wireless circuit (326). In certain examples, the high-speed processor (322) runs an operating system such as the LINUX operating system or other such operating systems such as the operating system (1612) of FIG. 16. In addition to any other tasks, the high-speed processor (322) running the software architecture for the data processor (302) is used to manage data transmissions with the high-speed wireless circuit (326). In certain examples, the high-speed wireless circuit (326) is configured to implement the IEEE (Institute of Electrical and Electronic Engineers) 802.11 communication standards, also referred to herein as Wi-Fi. In some examples, other high-speed communication standards can be implemented by high-speed wireless circuits (326).

[0040] The memory (324) includes any storage device capable of storing camera data generated by the camera (308) and the image processor (306). Although the memory (324) is depicted as being integrated with the high-speed circuit (320), in some examples, the memory (324) may be an independent, standalone element of the data processor (302). In these specific examples, electrical wiring may provide a connection from the image processor (306) or the low-power processor (314) to the memory (324) through a chip containing the high-speed processor (322). In some examples, the high-speed processor (322) may manage the addressing of the memory (324) so ​​that the low-power processor (314) activates the high-speed processor (322) whenever a read or write operation involving the memory (324) is required.

[0041] The tracking module (340) estimates the orientation of the glasses (100). For example, the tracking module (340) uses GPS data as well as image data and corresponding inertial data from the camera (308) and position components (1740) to track the position of the glasses (100) relative to a reference frame (e.g., a real-world environment) and determine the orientation. The tracking module (340) continuously collects and uses updated sensor data describing the movements of the glasses (100) to determine updated three-dimensional orientations of the glasses (100) that represent changes in relative position and orientation with respect to physical objects in the real-world environment. The tracking module (340) enables the visual placement of virtual objects relative to physical objects by the glasses (100) within the user's field of view through displays (310).

[0042] The GPU and display driver (338) can generate frames of virtual content or other content to be presented on the displays (310) using the posture of the glasses (100) when the glasses (100) are functioning in traditional augmented reality mode. In this mode, the GPU and display driver (338) generates updated frames of virtual content based on updated three-dimensional postures of the glasses (100) that reflect changes in the user's position and orientation relative to physical objects in the user's real-world environment.

[0043] Virtual content presented on displays (310) may provide a virtual reality experience that includes or references various image processing operations corresponding to image modification, filtering, media overlays, transformations, etc., as further described herein. In some examples, these image processing operations provide an interactive experience of a real-world environment, wherein objects, surfaces, backgrounds, lighting, etc. in the real world are enhanced by computer-generated perceptual information. In this context, "augmented reality effects" include a collection of data, parameters, and other assets used to apply a selected augmented reality experience to an image or video feed. In some examples, augmented reality effects are provided by Snap, Inc. under the registered trademark LENSES.

[0044] One or more functions or operations described in this specification may also be performed by an application residing in the glasses (100), a client device (328), or a remote server. For example, one or more functions or operations described in this specification may be performed by one of the applications (1606), such as a messaging application (1646).

[0045] FIG. 4 is a block diagram illustrating in more detail the architecture of glasses (100) according to some examples. As discussed above with reference to FIG. 3, the glasses include a high-speed circuit (320) and a low-power circuit (304). The high-speed circuit (320) and the low-power circuit (304) can each be implemented, for example, as a System-On-a-Chip (SOC).

[0046] The high-speed circuit (320) runs an operating system (402) and a security OS (404). The security OS (404) provides a trusted execution environment, which is a security area of ​​the high-speed circuit (320) that ensures sensitive data is isolated and stored, processed, and protected in a trusted environment. Thus, it performs user authentication functions and provides protection against software attacks generated by or originating from the operating system (402).

[0047] The security OS (404) executes security-related functions of the key master (406) and the gatekeeper (410). The key master (406) not only stores security certificates and hashed passwords, but also generates keys and certificates used to encrypt the file system used in the memory (324) of the high-speed circuit (320). The gatekeeper (410) handles security requests from the operating system (402) and delegates related operations to the key master (406) during password setting, changing, and verification operations. The password may be a Personal Identification Number (PIN) in some examples, but other user authentication methods, such as biometric technologies, may be used.

[0048] The gatekeeper (410) also enforces user lockouts if too many login attempts fail. The number of attempts and lockout times are measured using a monotonic clock to avoid disabling the lockout enforcement by draining the battery of the glasses (100) or changing the time. After a factory reset, when a password is set on the glasses for the first time, the gatekeeper (410) encrypts the user file system using a key and associates it with the user ID. When the glasses (100) start up, the user must unlock the glasses (100) at least once to allow the operating system (402) to decrypt and mount the file system. Subsequent locking of the glasses (100) does not re-encrypt the file system until a reboot.

[0049] After booting, the glasses (100) are said to be in direct boot mode until the user enters their password and the file system is decrypted and loaded. In direct boot mode, only specific direct boot-aware applications can be executed and are allowed to be executed, and access to the encrypted data store is not provided. Thus, direct boot mode is a partial boot or partial boot mode in both terms of the applications allowed to be executed and the memory accessible by direct boot-aware applications. Direct boot mode is indicated, and the execution of direct boot-aware applications is triggered by the LOCKED_BOOT_COMPLETED system state and system state message. When the user unlocks the glasses (100) using their password, the architecture completes the remaining initialization, updates the system state, and sends the BOOT_COMPLETED status message to launch several other applications, including the home / launcher application, which is part of the glasses application (412).

[0050] The operating system (402) includes a framework having several helper classes and wrappers to communicate with the security OS (404) and cache associated values. In some examples, the operating system (402) and the associated framework are based on the Android Open Source Project (AOSP). The operating system (402) includes a lock screen service (418), a lock settings service (420), an alarm manager service (422), a properties service (424), and a camera service (428).

[0051] The lock screen service (418) is a direct-activation recognition application and is launched first to initialize the lock state (432) of the glasses (100) before any other service or application reads the lock state (432). The lock screen service (418) is also a persistent application in that the operating system (402) automatically restarts the application whenever the lock screen service (418) is terminated. The lock screen service (418) also maintains all user settings for security, such as the amount of time the glasses (100) remain locked after the glasses are removed (or "taken off") from the user's face. For additional security, all setting values ​​are stored by the lock screen service (418) as encrypted strings of key-value pairs.

[0052] The lock screen service (418) also handles all password-related communications with the lock settings service (420) via operating system (402) APIs, including handling settings, handling changes and removals of passwords from the device, verifying entered passwords and reporting successful and failed verification attempts, as well as reporting arbitrary user lockout states and times to other services. The lock screen service (418) also handles setting specific lock alarms using the alarm manager service (422) in the Android framework. When an alarm is turned off, the lock screen service (418) is notified by the alarm manager service (422) and locks the glasses by setting the lock status attribute field (426) attribute within the attribute service (424) to true. The lock screen service (418) also clears any alarms when settings or states change, such as when the user puts the glasses back on (or "wears" the glasses (100)) before the removal alarm timer expires, and sets new alarms as needed.

[0053] The lock screen service (418) also monitors the status of the wear sensor (436) and notifies the relevant services when the user puts on or takes off the glasses (100). The lock screen service (418) queries the low-power circuit (304) about the status of the wear sensor (436) through the proprietary glasses message service (430).

[0054] The alarm manager service (422) is a standard operating system (402) service that enables applications or services to set alarms that provide them with a message at a specified later time. The alarm manager service (422) is typically used to initiate periodic or delayed tasks that otherwise have inaccurate timers. This allows for precise coordination with other tasks or services for efficiency. The lock screen service (418) requires accurate alarms based on uptime to avoid problems that may otherwise arise from changed date and time settings, such as specific times, dates, or time zones.

[0055] All password and security-related communications between any other services or applications on the security OS (404) and the high-speed circuit (320) are routed through the lock setting service (420). The lock setting service (420) provides a wrapper for communication with the gatekeeper (410) and communicates with the gatekeeper (410) using an inter-process communication protocol specific to the security OS (404).

[0056] The attribute service (424) is also a direct activation recognition application. The attribute service (424) stores the current lock state (432) as a true or false (binary) value in the lock state attribute field (426).

[0057] The glasses application (412) is a proprietary application that provides a user experience for the glasses (100). It is the primary foreground activity executed by the glasses (100) and initializes and maintains an instance of an AR module (not shown) that manages and displays any selected AR content or effects provided by the user interface and the glasses (100). As illustrated in FIG. 4, the glasses application (412) also includes a lock screen UI (416) and a lock screen controller (414). Since the lock screen UI (416) and the lock screen controller (414) display the lock screen to the user, they are also direct activation recognition services that start in direct activation mode and are initialized prior to the rest of the glasses application (412).

[0058] The lock screen UI (416) displays a lock screen to the user on the displays (310) of the glasses (100) and receives user input of a PIN to unlock the glasses (100). The lock screen UI (416) also displays other security-related information, such as a lockout timer, animations, or dialog boxes, to display failed attempts and the number of failed attempts.

[0059] The lock screen controller (414) monitors the lock state (432) maintained by the attribute service (424). The lock screen controller (414) blocks user and system access to any other functionality of the glasses (100) when the glasses are locked with a lock state (432) having a true value. The lock state (432) of the glasses (100) is communicated to the user via the lock screen UI (416) as needed. The lock screen controller (414) also communicates with the lock screen service (418) via the glasses message service (430) to transmit the PIN entered by the user to the lock screen service (418). The lock screen controller (414) also initiates the main UI for the glasses (100) based on the lock state (432) changing from true to false when an unlock attempt is successful or when the device is unlocked through other means, such as a system reset or the expiration of an alarm.

[0060] The proprietary glasses message service (430) is a request-response protocol that handles remote procedure calls (RPCs) between a messaging client (408) on a client device (328), a Bluetooth controller (434) in a low-power circuit (304), and a lock screen controller (414) in a glasses application (412). All software modules proprietary to the manufacturer of the glasses (100) and the provider of the messaging client (408) communicate with each other using the glasses message service (430). The glasses message service (430) also acts as a controller that transmits messages between the wear sensor (436) and the lock screen service (418).

[0061] In some examples, the following remote procedure calls are supported by the glasses message service (430).

[0062] ● SetUserDeviceSecurityRequest: This RPC is sent by a messaging client (408) to set a password for non-security glasses (100) that do not have an existing password, or to disable security by removing the password and all security settings (including user settings) for glasses (100) that have an existing password, or to change settings such as delays before unlocking, events that trigger locking, etc.

[0063] ● GetUserDeviceSecurityRequest: This RPC returns all user settings for the glasses (100) as well as the lock state (432) status (e.g., device lock, timeout, etc.). This RPC is used by both the messaging client (408) and the lock screen controller (414) to query the current lock state (432) status during initiation or refresh operations.

[0064] ● SetUserDevicePasswordRequest: This RPC is used by the messaging client (408) to change the password for the glasses (100).

[0065] ● IsDeviceSecure: This RPC is used to determine whether a password for the glasses (100) is set during initial startup.

[0066] ● VerifyPasscodeRequest: This RPC is used to verify the password, and once the password is verified, then the specified request is authorized. The following requests are supported: a) User Unlock: If the password is verified and successful, the glasses (100) are unlocked; b) Proximity Unlock: If the password is verified and successful, the device is unlocked or remains unlocked while the client device (328) and the messaging client (408) are connected to the glasses (100) via Bluetooth or other short-range link; c) Verification: Used only to verify the password, which can be used by the messaging client (408) to verify the current PIN during pairing with the glasses (100) or when changing the password.

[0067] ● PairingRequest: This RPC is used by the low-power circuit (304) firmware to verify whether the glasses are in a secure state for pairing via Bluetooth. This is allowed only when the operating system (402) is fully started and the lock state (432) is false, i.e., when the glasses (100) are unlocked.

[0068] The low-power circuit (304) includes a Bluetooth controller (434) that handles Bluetooth communication with a messaging client (408).

[0069] FIGS. 5A and 5B illustrate a flowchart for initializing the lock screen upon startup of glasses (100) according to some examples. The flowchart illustrates information flows and calls between state changes associated with components.

[0070] The flowchart begins when the operating system (402) completes the lock state start process and reports that the lock state start process is complete. This is broadcast as a system status message. In response to the lock state start complete status message, the lock screen service (418) starts in operation 502. Then, the lock screen service (418) queries the operating system (402) (via the lock setting service (420)) regarding whether the glasses (100) are in a secure state. The glasses (100) are considered a secure device if a password has already been set. If the lock setting service (420) responds true, then the lock screen service (418) locks the glasses (100) by setting the lock state to true (via the lock setting service (420)). If the lock setting service (420) responds false, then the lock screen service (418) unlocks the glasses (100) by setting the lock state to false.

[0071] In response to the lock state startup complete status message, the camera service (428) starts in operation 504. The camera service (428) is a startup recognition process. Additionally, the glasses application (412) starts in operation 506. After that, the glasses application (412) initializes the lock screen controller (414) in operation 508.

[0072] Next, the glasses application (412) requests a lock status (432) from the lock screen service (418). The flowchart continues in FIG. 5b, and the operating system (402) responds as either true or false. If the operating system (402) reports to the glasses application (412) that the device is locked (true), then, in operation 510, the lock screen UI (416) is launched. Assuming a successful unlock in operation 512, the glasses application (412) starts the full user interface of the glasses (100) in operation 514. If the operating system (402) reports to the glasses application (412) that the device is unlocked (false), then, in operation 514, the full user interface of the glasses (100) is launched.

[0073] Next, in operation 516, the operating system (402) completes the startup process and broadcasts a startup complete system message. At this point, the glasses (100) are fully started and functioning.

[0074] FIGS. 6a and 6b illustrate flowcharts for user unlocking of glasses (100) according to some examples. The flowcharts illustrate information flows and calls between state changes associated with components.

[0075] The flowchart illustrated in FIG. 6a begins with receiving user input of a PIN by the lock screen UI (416) in operation 602. Then, the lock screen UI (416) transmits the PIN to the lock screen controller (414) along with a PIN verification request. The lock screen controller (414) then transmits the PIN to the lock screen service (418) along with a PIN verification request. The lock screen service (418) then transmits the PIN to the lock setting service (420) along with a PIN verification request. The lock setting service (420) then transmits the PIN to the lock screen controller (414) along with a PIN verification request.

[0076] If the PIN is correct, the security OS (404) responds to the lock setting service (420) with a success message, which is transmitted by the lock setting service (420) to the lock screen service (418). In response, the lock screen service (418) instructs the lock setting service (420) to set the lock state to false, which indicates that the glasses (100) are in an unlocked state. Then, the lock screen service (418) transmits the success message to the lock screen controller (414), and the lock screen controller (414) unlocks the lock screen UI (416) in operation 604 and starts the main user interface of the glasses.

[0077] The flowchart illustrated in FIG. 6b also begins with the reception of user input of a PIN by the lock screen UI (416) in operation 606 and the transmission of the PIN and PIN verification request to the security OS (404) along the line as in FIG. 6a. However, in this case, the PIN is incorrect, and the security OS (404) responds to the lock setting service (420) with a failure message. This message is transmitted down the line from the lock setting service (420) to the lock screen service (418), to the lock screen controller (414), and to the lock screen UI (416), where in operation 608, a PIN entry error animation is displayed to the user by the glasses (100).

[0078] FIGS. 7a and 7b illustrate a flowchart for setting a PIN on non-security glasses (100) according to some examples. The flowchart illustrates information flows and calls between state changes associated with components. In some examples, this sequence may occur when the user of the new glasses (100) receives them after pairing.

[0079] FIG. 7a illustrates a sequence in which no password is set. The flowchart begins with a messaging client (408) requesting a device security status from a lock screen service (418). This is performed via a glasses message service (430) on the client device (328), low-power circuit (304), and operating system (402). In this particular case, the lock screen service (418) responds to the messaging client (408) with a non-secure security status. Then, the messaging client (408) prompts the user to enter a new PIN entered in operation 702. Then, the messaging client (408) sends a PIN setting message to the lock screen service (418), which in turn passes it to the lock setting service (420), which in turn passes it to the security OS (404). When the PIN is initially set, the security OS (404) turns on file-based encryption in operation 704 to further secure the device.

[0080] Afterward, the security OS (404) responds to the lock setting service (420) with a success or failure message, which is transmitted along the line to the messaging client (408). In the case of a failure message, an appropriate failure dialog box or animation is presented on the client device (328) by the messaging client (408).

[0081] In the case of a success message, the PIN is stored in the keychain of the messaging client (408) in operation 708. Additionally, where applicable, for example, if this is the first setting of the PIN on the glasses (100), new settings are stored in the lock screen service (418) in operation 706. The settings may be, for example, the amount of time after the glasses (100) are removed or the glasses being locked when proximity between the glasses (100) and the client device (328) is lost. Additionally, the lock screen service (418) may set an alarm in the alarm manager service (422) in operation 710 if required or specified by the new settings.

[0082] FIG. 7a illustrates a sequence in which no password is set. The flowchart begins with a messaging client (408) requesting a device security status from a lock screen service (418). In this particular case, the lock screen service (418) responds to the messaging client (408) with a security status. Then, the messaging client (408) receives input of a new PIN from the user in operation 712 and retrieves the current PIN from the keychain in operation 714. Subsequently, the messaging client (408) sends a security status setting message containing both the new PIN and the current PIN to the lock screen service (418). Then, the lock screen service (418) sends a PIN verification message to a lock setting service (420) with the previous PIN, and the lock setting service (420) sends it to the security OS (404).

[0083] The security OS (404) checks the previous PIN against the PIN stored in the key master (406) and sends a success or failure message back to the lock setting service (420). The failure message will be sent back to the messaging client (408) along the line, and an appropriate failure dialog box or animation will be presented on the client device (328) by the messaging client (408).

[0084] In response to a request to verify the previous PIN, a success message from the security OS (404) to the lock setting service (420) is transmitted by the lock setting service (420) to the lock screen service (418). Then, the lock screen service (418) transmits a new PIN setting message to the lock setting service (420), which in turn transmits a new PIN setting message to the security OS (404). Then, the lock screen service (418) transmits a new PIN verification message to the lock setting service (420), which in turn transmits a new PIN verification message to the security OS (404). In response to verifying the new PIN, the security OS (404) transmits a success message to the lock setting service (420), which is transmitted back along the line to the messaging client (408). When the messaging client (408) receives the success message, it stores the new PIN in the keychain in operation 716.

[0085] FIG. 8 illustrates a flowchart for changing settings on the security glasses (100) according to some examples. The flowchart illustrates information flows and calls between state changes associated with the components.

[0086] The flowchart begins with a messaging client (408) requesting a device security status from a lock screen service (418). This is performed via a glasses message service (430) on the client device (328), low-power circuit (304), and operating system (402). In this particular case, the lock screen service (418) responds to the messaging client (408) with a security status. Then, the messaging client (408) prompts the user to enter new settings received by the messaging client (408) in operation 802. Then, the messaging client (408) retrieves the current PIN from the keychain in operation 804 and sends a security setting message to the lock screen service (418). Then, the lock screen service (418) sends a Verify PIN message to the lock setting service (420), and the lock setting service (420) forwards it to the security OS (404).

[0087] After that, the security OS (404) responds to the lock setting service (420) with a success or failure message, which is then forwarded back to the messaging client (408) along the line. In the case of a failure message, an appropriate failure dialog box or animation is presented on the client device (328) by the messaging client (408).

[0088] In the case of a success message, new settings are stored in the lock screen service (418) in operation 806. The new settings may relate, for example, to the amount of time after the glasses (100) are removed or to the glasses being locked when proximity between the glasses (100) and the client device (328) is lost. Additionally, the lock screen service (418) may set an alarm in the alarm manager service (422) in operation 810 if required or specified by the new settings. Finally, upon receipt of a success message, the messaging client (408) updates the UI with any settings defining changes to the UI in operation 808.

[0089] FIG. 9 illustrates a flowchart for clearing a PIN on security glasses (100) according to some examples. The flowcharts illustrate information flows and calls between state changes associated with components.

[0090] The flowchart begins with a messaging client (408) requesting a device security status from a lock screen service (418). In this particular case, the lock screen service (418) responds to the messaging client (408) with a security status. Then, the messaging client (408) retrieves the current PIN from the keychain in operation 902 and sends a security status setting message to the lock screen service (418) that includes both the PIN and the erase PIN command. Next, the lock screen service (418) sends a lock erase message to the lock setting service (420). The lock setting service (420) sends a password removal message to the security OS (404), which verifies the PIN, and if verified, erases the password removal message from the key master (406).

[0091] Next, the lock screen service (418) sends a PIN-less verification message to the lock setting service (420). In response to a success message from the lock setting service (420), the lock screen service (418) sends a lock state setting message to the lock setting service (420) to set the lock state to false. Next, the lock screen service (418) clears all settings in operation 906 and clears any alarms in operation 904.

[0092] Next, the lock screen service (418) delivers a success message to the messaging client (408), and the messaging client removes the PIN from the keychain in operation 908.

[0093] FIGS. 10a and FIGS. 10b illustrate a flowchart for pairing a client device (328) with glasses (100) according to some examples. The flowchart illustrates information flows and calls between state changes associated with components.

[0094] FIG. 10a illustrates a sequence for the case of security glasses (100). A messaging client (408) sends a pairing request to a low-power circuit (304) via a Bluetooth controller (434) and a glasses message service (430). The pairing request is passed to a lock screen service (418), and the lock screen service (418) inquires from a lock setting service (420) whether the glasses (100) are in a secure state. In this case, the lock setting service (420) responds with true, indicating that the glasses are in a secure state. Then, the lock screen service (418) sends a lock state acquisition message to the lock setting service (420) to determine whether the glasses are locked.

[0095] If the lock setting service (420) responds with true indicating that the device is locked, the lock screen service (418) responds to the low-power circuit (304) with a disallow message, and pairing does not occur. The low-power circuit (304) then provides a failure message to the messaging client (408).

[0096] If the lock setting service (420) responds falsely indicating that the device is not locked, the lock screen service (418) responds to the low-power circuit (304) with an allow message, and pairing occurs. The low-power circuit (304) then provides a success message to the messaging client (408).

[0097] If there is no response from the lock screen service (418) to the pairing request during the timeout period, then pairing is not allowed by the low-power circuit (304) in operation 1002.

[0098] FIG. 10b illustrates a sequence for the case of non-secure glasses (100). A messaging client (408) sends a pairing request to a low-power circuit (304) via a Bluetooth controller (434) and a glasses messaging service (430). The pairing request is passed to a lock screen service (418), and the lock screen service (418) inquires from a lock setting service (420) whether the glasses (100) are in a secure state. In this case, the lock setting service (420) responds with a false value indicating that the glasses are in a non-secure state. The lock screen service (418) responds to the low-power circuit (304) with an allow message, and pairing occurs. The low-power circuit (304) then provides a success message to the messaging client (408).

[0099] If there is no response from the lock screen service (418) to the pairing request during the timeout period, then pairing is not allowed by the low-power circuit (304) in operation 1004.

[0100] FIGS. 11a and FIGS. 11b illustrate flowcharts regarding authorizations associated with the security and operation status of glasses (100) according to some examples. The flowcharts illustrate information flows and calls between state changes associated with components.

[0101] In FIG. 11a, the messaging client (408) determines that the feature to be executed in operation 1102 is blocked by the lock screen, that is, the feature is not allowed to be executed when the glasses (100) are in a secure and locked state. Then, the messaging client (408) sends a device security acquisition message to the lock screen service (418). The lock screen service (418) responds with a message including the secure state and activation state of the glasses (100).

[0102] In operation 1104, if the lock status is true, an unlock dialog box is displayed on either or both of the client device (328) and the glasses (100). In operation 1104, if the lock status is false, the messaging client (408) proceeds normally to the feature in operation 1106.

[0103] In FIG. 11b, the messaging client (408) determines that the feature in operation 1108 cannot operate in direct activation. The messaging client (408) sends a device security acquisition message to the lock screen service (418). The lock screen service (418) responds with a message including the security status and activation status of the glasses (100).

[0104] In operation 1104, if the direct launch status is true and the feature cannot be operated in direct launch, then in operation 1110, an unlock dialog box is displayed on either or both of the client device (328) and the glasses (100). In operation 1104, if the direct launch status is false, the messaging client (408) proceeds normally to the feature in operation 1112.

[0105] FIG. 12 illustrates a flowchart for PIN entries on security glasses (100) according to some examples. The flowcharts illustrate information flows and calls between state changes associated with components.

[0106] The sequence begins at operation 1202, and an unlock dialog box is displayed on either or both of the client device (328) and the glasses (100). At operation 1204, user input to unlock the glasses (100) is received. The messaging client (408) retrieves a PIN from the keychain and sends a PIN verification message to the lock screen service (418). When the lock screen service (418) responds to the messaging client (408) with a success message, the messaging client (408) proceeds normally at operation 1206.

[0107] If the lock screen service (418) responds to the messaging client (408) with a failure message and the number of attempts exceeds a specified amount, in action 1208, relevant lockout information is displayed on either or both of the client device (328) and the glasses (100). If the lock screen service (418) responds to the messaging client (408) with a failure message and the number of attempts does not exceed a specified amount, in action 1210, a user prompt for the user to manually enter a PIN is displayed on the client device (328) and the glasses (100), or both.

[0108] FIG. 13 illustrates a lock screen UI (1300) used on glasses (100) in some examples for manual input of a PIN. The glasses (100) are awakened upon receiving a touch input on the touchpad (124) or pressing on one of the buttons (126). In response to this, and in some examples, when the glasses (100) are locked, the lock screen UI (1300) is presented to prompt for an entry of the PIN. As can be seen, the lock screen UI (1300) includes a keypad display (1302) and entry fields (1304).

[0109] Forward and reverse swipe inputs received on the touchpad (124) traverse the keypad display (1302), and when a tap input is received on the touchpad (124) as exemplified by the tap and swipe inputs (1306), a highlighted value at the center position can be selected, included in the entry fields (1304), and displayed. Upon receipt of the correct PIN, the display on the glasses (100) transitions to a user interface screen corresponding to the AR function of the glasses (100).

[0110] FIG. 14 is a flowchart (1400) illustrating operations performed by glasses (100) and a client device (328) according to some examples. For the purposes of explanation, the operations of the flowchart (1400) are described herein as occurring in series or linearly. However, multiple operations of the flowchart (1400) may occur in parallel. Additionally, the operations of the flowchart (1400) do not need to be performed in the order shown, and / or one or more blocks of the flowchart (1400) do not need to be performed and / or may be replaced with other operations.

[0111] The method starts at operation 1402, and at operation 1402, the glasses (100) that are in an off state or sleep state to conserve power receive an activation input. This may be, for example, an initial power-up or a button (126) press when in a sleep state.

[0112] In response to receiving an activation input, in operation 1404, the glasses (100) perform the direct boot routine discussed above. In direct boot mode, only specific direct boot-aware applications can be executed and are allowed to be executed. Direct boot mode is indicated, and the execution of direct boot-aware applications is triggered by the LOCKED_BOOT_COMPLETED system status and system status message. Optionally, in operation 1406, a boot UI or boot screen is displayed by the glasses (100).

[0113] In operation 1408, the glasses (100) receive a command to perform a function. This may be transmitted from a client device (328) or may be a function indicated by user input on the glasses (100), such as pressing a button to capture an image or video.

[0114] In operation 1410, the glasses (100) or the client device (328) determines whether the function requires the glasses (100) to be unlocked. If so, the method proceeds to operation 1412. Otherwise, the glasses (100) or the client device (328) determines in operation 1426 whether the function is direct-operation compatible. If not, the method proceeds to operation 1412. If the function is direct-operation compatible, the function is performed in operation 1428, and the method returns to operation 1408 and proceeds further from there.

[0115] If the function requires unlocking as determined in operation 1410 or operation 1426, or if direct-action compatibility is not available, the glasses display a lock screen UI to the user in operation 1412. In operation 1414, the glasses (100) receive and verify a PIN. This may be received from a keychain on the client device (328) (in response to the "unlock" user input), from manual user input of a PIN on the glasses (100) (100), or from manual input on the client device (328). If the PIN is not verified, unlocking fails in operation 1424, and the glasses (100) display an error message or animation. Then, the method returns to operation 1412 for manual PIN input. A number of failed attempts will be handled as discussed above and are not exemplified here for clarity.

[0116] When the PIN is verified in operation 1416, the glasses (100) then complete the entire activation procedure in operation 1418, and the function is performed in operation 1420. Subsequently, the operation of the glasses generally continues in operation 1422.

[0117] Examples of functions that are directly activated and compatible, whether in a locked or unlocked state, are management functions such as setting the volume of built-in or linked speakers (e.g., wireless headset) from a messaging client (408) or the brightness of the display on glasses. Examples of functions that are not directly activated and compatible, whether in a locked or unlocked state, are higher-level functions that do not involve the user's privacy or the device itself, such as capturing photos or videos, checking for firmware updates, or performing firmware updates. Examples of functions that are not directly activated and not compatible with a locked state (i.e., available only in the unlocked state) include selecting augmented reality effects to apply to a video feed from a camera ("opening the lens") and transmitting, posting, or uploading captured images or recorded videos.

[0118] Networked computing environment

[0119] FIG. 15 is a block diagram illustrating an exemplary messaging system (1500) for exchanging data (e.g., messages and associated content) over a network. The messaging system (1500) includes multiple instances of client devices (328), each of which hosts multiple applications including a messaging client (408) and other applications (1502). Each messaging client (408) is communicably coupled to other instances of the messaging client (408) (e.g., hosted on each other client device (328)), a messaging server system (1504), and third-party servers (1506) over a network (330) (e.g., the Internet). The messaging client (408) may also communicate with locally hosted applications (1502) using application program interfaces (APIs).

[0120] A messaging client (408) can communicate with other messaging clients (408) and a messaging server system (1504) and exchange data through a network (330). The data exchanged between messaging clients (408) and between a messaging client (408) and a messaging server system (1504) includes functions (e.g., commands for calling functions) as well as payload data (e.g., text, audio, video, or other multimedia data).

[0121] The messaging server system (1504) provides server-side functionality to a specific messaging client (408) via the network (330). Although the specific functions of the messaging system (1500) are described in this specification as being performed by the messaging client (408) or by the messaging server system (1504), the location of the specific functionality within the messaging client (408) or the messaging server system (1504) may be a design choice. For example, it may be technically desirable to initially place the specific technology and functionality within the messaging server system (1504), but later migrate this technology and functionality to the messaging client (408) where the client device (328) has sufficient processing capacity.

[0122] The messaging server system (1504) supports various services and operations provided to the messaging client (408). These operations include sending data to the messaging client (408), receiving data from it, and processing data generated by it. This data may include, for example, message content, client device information, geolocation information, media augmentations and overlays, message content persistence conditions, social network information, and live event information. Data exchanges within the messaging system (1500) are invoked and controlled through functions available via the user interface (UI) of the messaging client (408).

[0123] Now, referring specifically to the messaging server system (1504), an application program interface (API) server (1508) is coupled to the application servers (1512) and provides a program interface thereto. The application servers (1512) are communicably coupled to a database server (1514), which facilitates access to a database (1518) that stores data associated with messages processed by the application servers (1512). Similarly, a web server (1522) is coupled to the application servers (1512) and provides web-based interfaces to the application servers (1512). To this end, the web server (1522) processes incoming network requests via HTTP (Hypertext Transfer Protocol) and some other related protocols.

[0124] An API (Application Program Interface) server (1508) receives and transmits message data (e.g., commands and message payloads) between a client device (328) and application servers (1512). Specifically, the API (Application Program Interface) server (1508) provides a set of interfaces (e.g., routines and protocols) that can be called or queried by a messaging client (408) to call the functionality of the application servers (1512). The API (Application Program Interface) server (1508) exposes various functions supported by the application servers (1512), including account registration, login functionality, transmission of messages via application servers (1512) from a specific messaging client (408) to another messaging client (408), transmission of media files (e.g., images or videos) from a messaging client (408) to a messaging server (1510), and settings of a collection of media data (e.g., stories) for possible access by another messaging client (408), searching of a list of friends of a user of a client device (328), searching of such collections, searching of messages and content, adding and deleting entities (e.g., friends) to an entity graph (e.g., social graph), the location of friends within the social graph, and opening application events (e.g., related to the messaging client (408)).

[0125] Application servers (1512) host a number of server applications and subsystems, including, for example, a messaging server (1510), an image processing server (1516), and a social network server (1520). The messaging server (1510) implements a number of message processing techniques and functions related to the aggregation and other processing of content (e.g., text and multimedia content) contained in messages received from a number of instances of the messaging client (408). As described in more detail, text and media content from a number of sources may be aggregated into collections of content (e.g., called stories or galleries). These collections are then made available to the messaging client (408). Other processor and memory-intensive processing of data may also be performed on the server side by the messaging server (1510), taking into account the hardware requirements for such processing.

[0126] The application servers (1512) also include an image processing server (1516) dedicated to performing various image processing operations with respect to images or videos in the payload of a message typically transmitted from or received from the messaging server (1510).

[0127] The social network server (1520) supports various social networking functions and services and makes these functions and services available to the messaging server (1510). To this end, the social network server (1520) maintains and accesses an entity graph within the database (1518). Examples of functions and services supported by the social network server (1520) include identifying other users of the messaging system (1500) with whom a specific user has relationships or is "following," and also identifying other entities and interests of the specific user.

[0128] Returning to the messaging client (408), features and functions of an external resource (e.g., an application (1502) or an applet) are made available to the user through the interface of the messaging client (408). In this context, "external" refers to the fact that the application (1502) or applet is outside the messaging client (408). The external resource is often provided by a third party, but may also be provided by the creator or provider of the messaging client (408). The messaging client (408) receives a user selection of options for launching or accessing the features of this external resource. The external resource may be an application (1502) installed on the client device (328) (e.g., a "native app"), or a small version of the application (e.g., an "applet") hosted on the client device (328) or remotely from the client device (328) (e.g., on third-party servers (1506)). A small version of the application contains a subset of the features and functions of the application (e.g., a full-scale native version of the application) and is implemented using a markup language document. In some examples, a small version of the application (e.g., an “applet”) is a web-based markup language version of the application and is embedded in a messaging client (408). In addition to using markup language documents (e.g., .*ml files), the applet may incorporate a scripting language (e.g., .*js files or .json files) and a style sheet (e.g., .*ss files).

[0129] In response to receiving a user selection of an option to launch or access features of an external resource, the messaging client (408) determines whether the selected external resource is a web-based external resource or a locally installed application (1502). In some cases, applications (1502) installed locally on the client device (328) may be launched independently and separately from the messaging client (408), such as by selecting an icon corresponding to the application (1502) on the home screen of the client device (328). As used herein, an icon may include one or both of text and graphic elements. Small versions of these applications may be launched or accessed through the messaging client (408), and in some examples, no part of the small application may be accessed outside the messaging client (408), or only limited parts may be accessed. A small application can be launched by a messaging client (408) receiving, for example, a markup language document associated with the small application from a third-party server (1506) and processing such document.

[0130] In response to determining that the external resource is a locally installed application (1502), the messaging client (408) instructs the client device (328) to launch the external resource by executing locally stored code corresponding to the external resource. In response to determining that the external resource is a web-based resource, the messaging client (408) communicates with (e.g.) third-party servers (1506) to obtain a markup language document corresponding to the selected external resource. Then, the messaging client (408) processes the obtained markup language document to present the web-based external resource within the user interface of the messaging client (408).

[0131] A messaging client (408) may notify a user of a client device (328), or other users associated with such user (e.g., “friends”) of activity occurring in one or more external resources. For example, the messaging client (408) may provide notifications to participants of a conversation (e.g., a chat session) in the messaging client (408) regarding current or recent use of an external resource by one or more members of a group of users. One or more users may be invited to participate in an active external resource or to launch an external resource that is recently used (in a group of friends) but is currently inactive. The external resource may provide participants in the conversation, each using their respective messaging clients (408), with the ability to share items, states, locations, or positions in the external resource with one or more members of a group of users in the chat session. The shared items may be interactive chat cards that members of the chat can interact with, for example, to launch a corresponding external resource, observe specific information within the external resource, or take a member of the chat to a specific location or state within the external resource. Within a given external resource, response messages can be sent to users on a messaging client (408). The external resource may optionally include different media items in the responses based on the current context of the external resource.

[0132] FIG. 16 is a block diagram (1600) illustrating a software architecture (1604) that may be installed in any one or more of the devices described herein. The software architecture (1604) is supported by hardware such as a machine (1602) that includes processors (1620), memory (1626), and I / O components (1638). In this example, the software architecture (1604) may be conceptualized as a stack of layers, where each layer provides specific functionality. The software architecture (1604) includes layers such as an operating system (1612), libraries (1608), frameworks (1610), and applications (1606). Operationally, applications (1606) call API calls (1650) through the software stack and receive messages (1652) in response to the API calls (1650).

[0133] The operating system (1612) manages hardware resources and provides common services. The operating system (1612) includes, for example, a kernel (1614), services (1616), and drivers (1622). The kernel (1614) acts as an abstraction layer between the hardware and other software layers. For example, the kernel (1614) provides, among other functions, memory management, processor management (e.g., scheduling), component management, networking, and security settings. Services (1616) may provide other common services for other software layers. Drivers (1622) are responsible for controlling the underlying hardware or interfacing with it. For example, drivers (1622) may include display drivers, camera drivers, BLUETOOTH® or BLUETOOTH® Low Energy drivers, flash memory drivers, serial communication drivers (e.g., USB (Universal Serial Bus) drivers), WI-FI® drivers, audio drivers, power management drivers, etc.

[0134] Libraries (1608) provide low-level common infrastructure used by applications (1606). Libraries (1608) may include system libraries (1618) (e.g., C standard library) that provide functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. Additionally, the libraries (1608) may include API libraries (1624) such as media libraries (e.g., libraries that support the presentation and manipulation of various media formats such as MPEG4 (Moving Picture Experts Group-4), H.264 or AVC (Advanced Video Coding), MP3 (Moving Picture Experts Group Layer-3), AAC (Advanced Audio Coding), AMR (Adaptive Multi-Rate) audio codecs, JPEG or JPG (Joint Photographic Experts Group), or PNG (Portable Network Graphics), graphics libraries (e.g., OpenGL frameworks used to render graphic content in two dimensions (2D) and three dimensions (3D) on a display), database libraries (e.g., SQLite, which provides various relational database functions), web libraries (e.g., WebKit, which provides web browsing functions), etc. The libraries (1608) may also include a wide variety of other libraries (1628) to provide many different APIs to the applications (1606).

[0135] Frameworks (1610) provide high-level common infrastructure used by applications (1606). For example, frameworks (1610) provide various graphical user interface (GUI) functions, high-level resource management, and high-level location services. Frameworks (1610) may provide a wide spectrum of other APIs that can be used by applications (1606), some of which may be specific to a particular operating system or platform.

[0136] In the example, applications (1606) may include a wide range of other applications such as a home application (1636), a contact application (1630), a browser application (1632), a book reader application (1634), a location application (1642), a media application (1644), a messaging application (1646), a game application (1648), and third-party applications (1640). Applications (1606) are programs that execute functions defined in programs. Various programming languages ​​may be used to create one or more of applications (1606) structured in various ways, such as object-oriented programming languages ​​(e.g., Objective-C, Java, or C++) or procedural programming languages ​​(e.g., C or assembly language). In a specific example, third-party applications (1640) (e.g., applications developed by entities other than the vendor of a specific platform using an ANDROID™ or IOS™ software development kit (SDK)) may be mobile software running on a mobile operating system such as IOS™, ANDROID™, WINDOWS® Phone, or other mobile operating systems. In this example, the third-party applications (1640) may call API calls (1650) provided by the operating system (1612) to facilitate the functionality described herein.

[0137] FIG. 17 is a schematic representation of a machine (1700) on which instructions (1710) (e.g., software, program, application, applet, app, or other executable code) can be executed to cause the machine (1700) to perform any one or more of the methodologies discussed herein. For example, instructions (1710) can cause the machine (1700) to perform any one or more of the methods described herein. Instructions (1710) convert a general-purpose, non-programming machine (1700) into a specific machine (1700) programmed to perform the described and illustrated functions in the described manner. The machine (1700) may operate as a standalone device or may be coupled to other machines (e.g., networked). In a networked deployment, the machine (1700) may operate as a server machine or client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine (1700) may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a PDA, an entertainment media system, a cellular phone, a smartphone, a mobile device, a head-worn device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web appliance, a network router, a network switch, a network bridge, or any machine capable of executing commands (1710) that specify actions to be taken by the machine (1700) sequentially or otherwise. Additionally, although only a single machine (1700) is exemplified, the term “machine” should also be considered to include a collection of machines that execute commands (1710) individually or jointly to perform any one or more of the methodologies discussed herein.

[0138] The machine (1700) may include processors (1702), memory (1704), and I / O components (1706) that can be configured to communicate with each other via a bus (1744). In the example, the processors (1702) (e.g., a CPU (Central Processing Unit), a RISC (Reduced Instruction Set Computing) processor, a CISC (Complex Instruction Set Computing) processor, a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an ASIC, a RFIC (Radio-Frequency Integrated Circuit), other processors, or any suitable combination thereof) may include, for example, a processor (1708) and a processor (1712) that execute instructions (1710). The term "processor" is intended to include multi-core processors that may include two or more independent processors (sometimes referred to as "cores") capable of executing instructions simultaneously. FIG. 17 illustrates multiple processors (1702), but the machine (1700) may include a single processor having a single core, a single processor having multiple cores (e.g., a multi-core processor), multiple processors having a single core, multiple processors having multiple cores, or any combination thereof.

[0139] Memory (1704) includes main memory (1714), static memory (1716), and storage unit (1718), both of which are accessible to processors (1702) via a bus (1744). The main memory (1704), static memory (1716), and storage unit (1718) store instructions (1710) that implement any one or more of the methodologies or functions described herein. The instructions (1710) may also exist, wholly or partially, during their execution by the networked system (300), in the main memory (1714), in the static memory (1716), in the machine-readable medium (1720) within the storage unit (1718), in at least one of the processors (1702) (e.g., in the processor's cache memory), or any suitable combination thereof.

[0140] The I / O components (1706) may include a wide variety of components for receiving inputs, providing outputs, generating outputs, transmitting information, exchanging information, capturing measurements, etc. The specific I / O components (1706) included in a specific machine will depend on the type of machine. For example, portable machines such as mobile phones may include touch input devices or other such input mechanisms, whereas a headless server machine may not include such touch input devices. It will be understood that the I / O components (1706) may include many other components not shown in FIG. 17. In various examples, the I / O components (1706) may include output components (1728) and input components (1732). Output components (1728) may include visual components (e.g., displays such as a PDP (plasma display panel), an LED (light emitting diode) display, an LCD (liquid crystal display), a projector, or a CRT (cathode ray tube)), acoustic components (e.g., speakers), haptic components (e.g., vibration motors, resistive mechanisms), other signal generators, etc. Input components (1732) may include alphanumeric input components (e.g., a keyboard, a touch screen configured to receive alphanumeric input, a photo-optical keyboard, or other alphanumeric input components), point-based input components (e.g., a mouse, a touchpad, a trackball, a joystick, a motion sensor, or other pointing mechanism), tactile input components (e.g., a physical button, a touch screen providing the position and / or force of touches or touch gestures, or other tactile input components), audio input components (e.g., a microphone), etc.

[0141] In additional examples, I / O components (1706) may include biometric components (1734), motion components (1736), environment components (1738), or location components (1740), among a wide range of other components. For example, biometric components (1734) include components that detect expressions (e.g., hand expressions, facial expressions, voice expressions, body gestures, or eye tracking), measure biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brainwaves), and identify a person (e.g., voice identification, retinal identification, face identification, fingerprint identification, or brainwave-based identification). Motion components (1736) include acceleration sensor components (e.g., accelerometers), gravity sensor components, rotation sensor components (e.g., gyroscopes), etc. Environmental components (1738) include, for example, light sensor components (e.g., photometers), temperature sensor components (e.g., one or more thermometers for detecting ambient temperature), humidity sensor components, pressure sensor components (e.g., barometers), acoustic sensor components (e.g., one or more microphones for detecting background noise), proximity sensor components (e.g., infrared sensors for detecting nearby objects), gas sensors (e.g., gas detection sensors for detecting concentrations of hazardous gases for safety or measuring pollutants in the atmosphere), or other components capable of providing indications, measurements, or signals corresponding to the surrounding physical environment. Position components (1740) include position sensor components (e.g., GPS receiver components), altitude sensor components (e.g., altimeters or barometers for detecting atmospheric pressure—from which altitude can be derived), orientation sensor components (e.g., magnetometers), etc.

[0142] Communication can be implemented using a wide variety of technologies. The I / O components (1706) further include communication components (1742) operable to connect the networked system (300) to a network (1722) or devices (1724), respectively, through coupling (1730) and coupling (1726). For example, the communication components (1742) may include a network interface component or other suitable device for interfacing with the network (1722). In additional examples, the communication components (1742) may include wired communication components, wireless communication components, cellular communication components, NFC (Near Field Communication) components, Bluetooth ® Components (e.g., Bluetooth) ® Low Energy), Wi-Fi ® It may include other communication components that provide communication through components and other aspects. The devices (1724) may be any of other machines or a wide variety of peripheral devices (e.g., peripheral devices connected via USB).

[0143] Furthermore, the communication components (1742) may include components capable of detecting identifiers or operable to detect identifiers. For example, the communication components (1742) may include Radio Frequency Identification (RFID) tag reader components, NFC smart tag detection components, optical reader components (e.g., optical sensors for detecting one-dimensional barcodes such as Universal Product Code (UPC) barcodes, multi-dimensional barcodes such as Quick Response (QR) codes, Aztec codes, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCC RSS-2D barcodes, and other optical codes), or acoustic detection components (e.g., microphones for identifying tagged audio signals). Additionally, various information such as location via Internet Protocol (IP) geolocation, location via Wi-Fi® signal triangulation, and location via detecting NFC beacon signals capable of indicating a specific location may be derived through the communication components (1742).

[0144] Various memories (e.g., memory (1704), main memory (1714), static memory (1716), and / or memory of processors (1702)) and / or storage unit (1718) may store one or more sets of instructions and data structures (e.g., software) that implement any one or more of the methodologies or functions described herein. These instructions (e.g., instructions (1710)) cause various operations to implement the disclosed examples when executed by processors (1702).

[0145] Commands (1710) may be transmitted or received over a network (1722) using a transmission medium through a network interface device (e.g., a network interface component included in the communication components (1742)) and using any one of a number of well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, commands (1710) may be transmitted or received using a transmission medium through a connection (1726) (e.g., peer-to-peer connection) to the devices (1724).

[0146] "Carrier signal" refers to any intangible medium capable of storing, encoding, or carrying instructions for execution by a machine, and includes digital or analog communication signals or other intangible media to facilitate the communication of such instructions. Instructions may be transmitted or received over a network using a transmission medium through a network interface device.

[0147] "Client device" refers to any machine that interfaces with a communication network to obtain resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop, PDAs (portable digital assistants), smartphones, tablets, ultrabooks, netbooks, laptops, multi-processor systems, microprocessor-based or programmable consumer electronics, game consoles, set-top boxes, or any other communication device that a user can use to access the network.

[0148] "Communication network" refers to one or more parts of a network that may be an ad-hoc network, intranet, extranet, VPN (virtual private network), LAN (local area network), wireless LAN (WLAN), WAN (wide area network), wireless WAN (WWAN), MAN (metropolitan area network), the Internet, part of the Internet, part of the PSTN (Public Switched Telephone Network), POTS (plain old telephone service) network, cellular telephone network, wireless network, Wi-Fi® network, other types of networks, or a combination of two or more of these networks. For example, a network or part of a network may include a wireless or cellular network, and a combination may be a CDMA (Code Division Multiple Access) connection, a GSM (Global System for Mobile communications) connection, or other types of cellular or wireless combinations.In this example, the combination can implement any of the various types of data transmission technologies, such as 1xRTT (Single Carrier Radio Transmission Technology), EVDO (Evolution-Data Optimized) technology, GPRS (General Packet Radio Service) technology, EDGE (Enhanced Data rates for GSM Evolution) technology, 3GPP (third Generation Partnership Project) including 3G, 4G (fourth generation wireless) networks, UMTS (Universal Mobile Telecommunications System), HSPA (High Speed ​​Packet Access), WiMAX (Worldwide Interoperability for Microwave Access), LTE (Long Term Evolution) standards, other things defined by various standards-setting organizations, other long-range protocols, or other data transmission technologies.

[0149] "Component" refers to a device, physical entity, or logic having boundaries defined by other techniques that provide function or subroutine calls, branch points, APIs, or partitioning or modularization of specific processing or control functions. Components may be combined with other components through their interfaces to perform machine processes. A component may be a packaged functional hardware unit designed to be used with other components, and generally a part of a program that performs a specific function among related functions. Components may constitute either software components (e.g., code embodied on a machine-readable medium) or hardware components. "Hardware component" is a type of unit capable of performing specific operations and may be configured or arranged in a specific physical manner. In various examples, one or more computer systems (e.g., standalone computer systems, client computer systems, or server computer systems) or one or more hardware components of a computer system (e.g., a processor or a group of processors) may be configured by software (e.g., an application or a part of an application) as hardware components that operate to perform specific operations as described herein. Hardware components may also be implemented mechanically, electronically, or any suitable combination thereof. For example, a hardware component may include dedicated circuits or logic permanently configured to perform specific operations. A hardware component may be a special-purpose processor, such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). A hardware component may also include programmable logic or circuits that are temporarily configured by software to perform specific operations.For example, a hardware component may include software executed by a general-purpose processor or another programmable processor. Once configured by such software, the hardware components become specific machines (or specific components of a machine) uniquely tailored to perform the configured functions and are no longer general-purpose processors. It will be understood that the decision to implement a hardware component as a mechanically, permanently configured dedicated circuit, or as a temporarily configured circuit (e.g., configured by software) may be driven by cost and time considerations. Accordingly, the phrase "hardware component" (or "hardware implementation component") should be understood to encompass entities of type that are physically configured, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a specific manner or perform the specific operations described herein. When considering examples where hardware components are temporarily configured (e.g., programmed), it is not necessary for each hardware component to be configured or instantiated at any single time instance. For example, in the case of a general-purpose processor configured by software such that a hardware component becomes a special-purpose processor, the general-purpose processor may be configured as different special-purpose processors (e.g., including different hardware components) at different times. Accordingly, the software configures a specific processor or processors to configure a specific hardware component at one time instance and a different hardware component at a different time instance. Hardware components may provide information to other hardware components and receive information from them. Accordingly, the described hardware components may be considered to be communicably coupled.In cases where multiple hardware components exist simultaneously, communication may be achieved through the transmission of signals between or between two or more of the hardware components (e.g., via appropriate circuits and buses). In examples where multiple hardware components are configured or instantiated at different times, communication between these hardware components may be achieved, for example, through the storage and retrieval of information within memory structures accessible to the multiple hardware components. For example, one hardware component may perform an operation and store the output of that operation in a memory device coupled to it for communication. Additional hardware components may subsequently access the memory device to retrieve and process the stored output. Hardware components may also initiate communication with input or output devices and operate on resources (e.g., information collections). Various operations of the exemplary methods described herein may be performed, at least partially, by one or more processors configured temporarily or permanently (e.g., by software) to perform the relevant operations. Whether configured temporarily or permanently, these processors may comprise processor implementation components that operate to perform one or more operations or functions described herein. As used herein, "processor implementation component" refers to a hardware component implemented using one or more processors. Similarly, the methods described herein may be processor-implemented at least partially, and specific processors or processors are examples of hardware. For example, at least some of the operations of the method may be performed by one or more processors or processor implementation components.Furthermore, one or more processors may also operate to support the execution of related operations in a "cloud computing" environment or as "SaaS (software as a service)." For example, at least some of the operations may be performed by a group of computers (e.g., machines including processors), and these operations may be accessible via a network (e.g., the Internet) and through one or more appropriate interfaces (e.g., APIs). The execution of specific operations may be distributed among processors, not only within a single machine but also deployed across multiple machines. In some examples, processors or processor implementation components may be located in a single geographic location (e.g., a home environment, an office environment, or within a server farm). In some examples, processors or processor implementation components may be distributed across multiple geographic locations.

[0150] "Computer-readable medium" refers to both machine storage media and transmission media. Accordingly, this term includes both storage devices / medias and carriers / modulated data signals. The terms "machine-readable medium," "computer-readable medium," and "device-readable medium" mean the same thing and may be used interchangeably in this disclosure.

[0151] "Short-term messages" refer to messages accessible for a limited duration. Short-term messages can be text, images, videos, etc. The access time for short-term messages can be set by the message sender. Alternatively, the access time may be a default setting or a setting specified by the recipient. Regardless of the setting technique, messages are temporary.

[0152] "Machine storage media" refers to one or more storage devices and / or media (e.g., centralized or distributed databases, and / or associated caches and servers) that store executable instructions, routines, and / or data. Accordingly, the term should be interpreted as including, but not limited to, solid-state memories, including memory located inside or outside processors, and optical and magnetic media. Specific examples of machine storage media, computer-storage media, and / or device-storage media include, for example, non-volatile memory including semiconductor memory devices, e.g., EPROM (erasable programmable read-only memory), EEPROM (electrically erasable programmable read-only memory), FPGAs, and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms “machine storage medium,” “device-storage medium,” and “computer-storage medium” mean the same thing and may be used interchangeably in this disclosure. The terms “machine storage media,” “computer-storage media,” and “device-storage media” specifically exclude carrier waves, modulated data signals, and other such media, at least some of which are covered under the term “signal medium.”

[0153] "Processor" refers to any circuit or virtual circuit (a physical circuit emulated by logic executed on an actual processor) that manipulates data values ​​according to control signals (e.g., "commands," "op codes," "machine code," etc.) and generates corresponding output signals applied to operate the machine. A processor may be, for example, a CPU (Central Processing Unit), a RISC (Reduced Instruction Set Computing) processor, a CISC (Complex Instruction Set Computing) processor, a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an RFIC (Radio-Frequency Integrated Circuit), or any combination thereof. A processor may also be a multi-core processor having two or more independent processors (sometimes referred to as "cores") capable of executing instructions simultaneously.

[0154] "Signal medium" refers to any intangible medium capable of storing, encoding, or carrying instructions for execution by a machine, and includes digital or analog communication signals or other intangible media to facilitate the communication of software or data. The term "signal medium" should be considered to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal in which one or more of its characteristics are set or altered to encode information in the signal. The terms "transmitting medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

[0155] Changes and modifications may be made to the disclosed examples without departing from the scope of the present disclosure. These and other changes or modifications are intended to be included within the scope of the present disclosure as expressed in the following claims.

Claims

Claim 1 A method executed by one or more processors within a head-wearing device comprising one or more display devices, comprising: receiving an activation input; performing a partial startup of the head-wearing device; receiving a user input command to perform a function; determining whether the user input command is partial startup compatible; determining whether the user input command allows partial startup execution; executing the user input command based on whether the user input command allows partial startup execution and is partial startup compatible; and completing the startup of the head-wearing device even if the user input command is partial startup compatible based on whether the user input command does not allow partial startup execution. Claim 2 A method according to claim 1, further comprising: a step of determining whether user authentication is required for the execution of the user input command; and a step of executing the user input command based on the fact that the user input command is partially bootable compatible and does not require user authentication. Claim 3 A method according to claim 1, wherein the partial startup of the head-worn device includes starting a lock screen service and a camera service. Claim 4 A method according to claim 1, further comprising: a step of determining whether user authentication is required for the execution of the user input command; and a step of executing a user authentication procedure regardless of whether the user input command is partially bootable, based on whether the user input command requires user authentication. Claim 5 A method according to claim 4, further comprising: a step of verifying correct user authentication; a step of completing the startup of the head-worn device; and a step of executing the user input command. Claim 6 A method according to claim 5, wherein the step of verifying correct user authentication includes the step of receiving a password from a relevant mobile device. Claim 7 A method according to claim 6, further comprising the step of prompting user input of a password through the head-worn device based on the fact that correct user authorization has not been verified using the password received from the relevant mobile device. Claim 8 A method according to claim 6, further comprising the step of maintaining an authenticated state of the head-wearing device based on the proximity of the head-wearing device to the relevant mobile device. Claim 9 A head-wearing device comprising: one or more cameras; one or more display devices; one or more processors; and a memory storing user input commands, wherein the device is configured to perform a method according to any one of claims 1 to 8 when the user input commands are executed by the one or more processors. Claim 10 delete Claim 11 delete Claim 12 delete Claim 13 delete Claim 14 delete Claim 15 delete Claim 16 A non-transient computer-readable storage medium, wherein the computer-readable storage medium comprises user input commands, and when the user input commands are executed by a head-wearing device comprising one or more display devices and one or more cameras, the head-wearing device enables the head-wearing device to perform a method according to any one of claims 1 to 8. Claim 17 delete Claim 18 delete Claim 19 delete Claim 20 delete

Images