Head mounted device system, method, and non-transitory computer readable storage medium for receiving and processing navigation input

Abstract

A method for receiving and processing navigation inputs executed by one or more processors in a head-mounted device system is disclosed, the head-mounted device system including one or more display devices, one or more camera devices, and a touchpad arranged generally vertically. The method includes: displaying a first carousel of AR effect icons; receiving a first horizontal input on the touchpad; rotating the first carousel of AR effect icons in response to the first horizontal input; receiving a first touch input on the touchpad, the touch input being used to select a specific AR effect icon in a selected position in the first carousel; displaying a scene viewed through one or more camera devices, the scene being enhanced with an AR effect corresponding to a specific AR effect icon; receiving a content capture user input; and capturing a new content item corresponding to the scene in response to the content capture user input.

Description

[0001] Related application data

[0002] This application claims the benefit of U.S. Patent Application No. 17 / 448,169, filed on September 20, 2021, and U.S. Provisional Patent Application No. 63 / 190,686, filed on May 19, 2021, the contents of which are incorporated herein by reference as if expressly set forth. Technical Field

[0003] The present disclosure relates generally to head-mounted devices with displays for augmented reality or virtual reality, and more particularly to head-mounted devices including a touchpad for navigating content or user interfaces provided by the head-mounted device or an associated device. Background Art

[0004] A head mounted device may be implemented with a transparent or translucent display through which a user of the head mounted device can view the surrounding environment. Such a device enables the user to look through the transparent or translucent display to view the surrounding environment, and also to see objects (e.g., virtual objects such as 3D renderings, images, videos, text, etc.) generated for display that appear to be part of the surrounding environment and / or superimposed on the surrounding environment. This is generally referred to as "augmented reality."

[0005] The head mounted device may also completely obstruct the user's field of view and display a virtual environment through which the user can move or be moved. This is often referred to as "virtual reality". As used herein, unless the context indicates otherwise, the term "augmented reality" or "AR" refers to both augmented reality and virtual reality as conventionally understood.

[0006] A user of the head mounted device may access a messaging application or a social networking application to view content or share content with other users of the application. In some cases, real-time content or stored content may be viewed and enhanced or modified by the user. That is, images, videos, or other media for enhancement may be captured from a real-time camera or retrieved from a local or remote data storage device.

[0007] As referred to herein, the phrase "augmented reality experience" includes or refers to various image processing operations corresponding to image modification, filtering, media overlays, transformations, etc., as further described herein. In some examples, these image processing operations provide an interactive experience of a real-world environment, where objects, surfaces, backgrounds, lighting, etc. in the real world are enhanced by computer-generated perceptual information. In this context, an "augmented reality effect" includes a collection of data, parameters, and other assets required to apply a selected augmented reality experience to an image or video feed. In some examples, augmented reality effects are provided by Snap Inc. under the registered trademark LENSES.

[0008] In some examples, the augmented reality effect includes augmented reality (or "AR") content that is configured to modify or transform image data presented within the GUI of the head-mounted device in some way. For example, AR effect data can be used to perform complex additions or transformations to content images, such as adding rabbit ears to a person's head, adding a floating heart with a background color, changing the scale of a person's features, adding augmentations to landmarks in a scene viewed on the head-mounted device, or many other such transformations. This includes both real-time modifications that modify the image when it is captured using a camera associated with the head-mounted device, which is then displayed by the head-mounted device with the AR effect modification, and modifications to stored content (such as video clips in a gallery), which can be modified using the AR effect. Similarly, real-time video capture can be used with AR effects to show a user of the head-mounted device how a video image currently captured by the device's sensors will modify the captured data. Such data may be displayed merely on the screen without being stored in memory, content captured by the device sensor may be recorded and stored in memory with or without AR effect modification (or both), or content captured by the device sensor may be sent to a server or another device via network 102 with AR effect modification.

[0009] Thus, AR effects and associated systems and modules for modifying content using AR effects may involve: detection of objects (e.g., faces, hands, bodies, cats, dogs, surfaces, objects, etc.), tracking of such objects as they leave, enter, and move around the field of view in a video frame, and modification or transformation of such objects as they are tracked. In various examples, different methods for implementing such transformations may be used. For example, some examples may involve generating a 3D mesh model of one or more objects, and implementing the transformation using transformations and animated textures of the models within a video. In other examples, tracking of points on an object may be used to place an image or texture (which may be two-dimensional or three-dimensional) at the tracked location. In another example, a neural network analysis of a video frame may be used to place an image, model, or texture in content (e.g., an image or video frame). Thus, AR effect data may include both: images, models, and textures used to create transformations in content, and additional modeling and analysis information required to implement such transformations using object detection, tracking, and placement. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] To easily identify the discussion of any particular element or action, the highest digit(s) in a reference number refers to the figure number in which the element is first introduced.

[0011] Figure 1 is a perspective view of a head mounted device according to some examples.

[0012] Figure 2 Shows Figure 1 Additional views of a head mounted device according to some examples.

[0013] Figure 3 Some examples are shown including Figure 1 Block diagram of networking system 300 showing details of a head mounted device.

[0014] Figure 4 is a diagrammatic representation of a networked environment in which the present disclosure may be deployed, according to some examples.

[0015] Figure 5 is a perspective view of a head mounted device according to another example.

[0016] FIG. 6A to FIG. 6D shows the Figure 1 or Figure 5 A series of user interface screens displayed on a headset.

[0017] Fig. 7A and Figure 7B shows the Figure 1 or Figure 5 An additional series of user interface screens displayed on the headset.

[0018] FIG. 8A to FIG. 8C shows the Figure 1 or Figure 5 An additional series of user interface screens displayed on the headset.

[0019] 9A to 9F shows the Figure 1 or Figure 5 An additional series of user interface screens displayed on the headset.

[0020] Fig.10 It is shown that according to some examples, Figure 1 or Figure 5 A flowchart of the user interface implemented for a head-mounted device.

[0021] Fig.11 Based on some examples Figure 1 or Figure 5 A flowchart of operations performed by a head mounted device in response to receiving user input on a touch panel.

[0022] Fig.12 Based on some examples Figure 1 or Figure 5 A flowchart of operations performed by a head mounted device in response to receiving user input on a touch panel.

[0023] Fig.13 is a block diagram illustrating a software architecture in which the present disclosure may be implemented, according to some examples.

[0024] Fig.14 is a diagrammatic representation of a machine in the form of a computer system according to some examples within which a set of instructions may be executed, for causing the machine to perform any one or more of the methodologies discussed. DETAILED DESCRIPTION

[0025] Known head-mounted devices such as AR glasses include a transparent or translucent display, enabling a user to look through the transparent or translucent display to view the surrounding environment. Additional information or objects (e.g., virtual objects such as 3D renderings, images, videos, text, etc.) are shown on the display and appear as part of the surrounding environment and / or superimposed on the surrounding environment to provide an augmented reality experience for the user. The display may, for example, include a waveguide that receives a beam from a projector, but any suitable display for presenting augmented or virtual content to the wearer may be used.

[0026] Navigation of information or user interfaces of the head mounted device may in some cases be provided by voice commands or inputs to an associated device such as a smart phone. In the present case, a touch pad is provided on the head mounted device, which may be used to provide xy touch input and tap input to the head mounted device. Because the touch pad may have a limited size and the display may also have limited display capabilities, or because of size, power or other considerations (e.g., in order to provide a display through which the user can still view the real world), it may be necessary to simplify the interaction between the head mounted device and the user.

[0027] The user interface flows described use specific patterns based on the capabilities of the head mounted device hardware and software, which typically have both different input options and different output capabilities than a mobile phone and a touch screen of a mobile phone. For example, a display used in AR glasses may have a narrower field of view than a mobile phone screen, have a different aspect ratio, and may be able to present less detail. Additionally, the touch interface described below is not juxtaposed or parallel to the display. These technical challenges can be addressed by providing simplified user interface screens in which each scene displays fewer elements and has specific responses to received input, thereby improving the functionality of the head mounted device and associated applications.

[0028] In one example, the head mounted device described herein uses absolute, indirect positioning for user interaction. In this example, there is no free cursor, nor the ability to select any displayed element as on a mobile phone touch screen. The head mounted device indirectly converts the user's finger motion on the touch pad to display coordinates in the case of absolute motion, combined with a single fixed selector that responds to a specific set of touch pad gesture actions for item selection, and displayed navigation cues to allow selection of logical transitions between screens.

[0029] In one example, a method for receiving and processing navigation inputs performed by one or more processors in a head-mounted device system includes one or more display devices, one or more cameras, and a touchpad that is generally arranged vertically. The method includes: displaying a first carousel of AR effect icons; receiving a first horizontal input on the touchpad; rotating the first carousel of AR effect icons in response to the first horizontal input; and receiving a first touch input on the touchpad, the touch input being used to select a specific AR effect icon in a selected position in the first carousel. The method also includes: displaying a scene viewed through one or more cameras, the scene being enhanced with an AR effect corresponding to a specific AR effect icon; receiving content capture user input; and capturing a new content item corresponding to the scene in response to the content capture user input.

[0030] The method may further include: receiving a first vertical input on the touch panel while the first carousel of AR effect items is being displayed; and displaying a second carousel including a plurality of content items in response to the first vertical input.

[0031] In addition, the method may include: receiving a second horizontal input on the touch pad; rotating a second carousel of content items in response to the second horizontal input; receiving a second touch input on the touch pad, the touch input being used to select a specific content icon in a selection position in the second carousel; and displaying additional user interface options in response to receiving the second touch input.

[0032] The first vertical input may be a downward swipe, and the method further includes displaying a content icon and a downward pointer above the first carousel. While a second carousel including a plurality of content items is being displayed, a second vertical input on the touch pad may be received, and in response to the second vertical input, the first carousel of AR effect items may be displayed.

[0033] The method may further include: when the first carousel is being displayed, receiving a third vertical input on the touch panel, the third vertical input having a direction opposite to the first vertical input; and displaying system information in response to the third vertical input.

[0034] In another example, a head mounted device system includes: one or more camera devices; one or more display devices; a generally vertically arranged touch panel; one or more processors; and a memory storing instructions that, when executed by the one or more processors, configure the system to perform operations. A non-transitory computer readable storage medium is also provided, the computer readable storage medium including instructions that, when executed by the head mounted device system including the one or more display devices, the one or more camera devices, and the generally vertically arranged touch panel, cause the head mounted device system to perform operations.

[0035] The operations may include: displaying a first carousel of AR effect icons; receiving a first horizontal input on a touch pad; rotating the first carousel of AR effect icons in response to the first horizontal input; and receiving a first touch input on the touch pad for selecting a specific AR effect icon in a selected position in the first carousel. The operations also include: displaying a scene viewed through one or more cameras, the scene enhanced with an AR effect corresponding to a specific AR effect icon; receiving a content capture user input; and capturing a new content item corresponding to the scene in response to the content capture user input.

[0036] The operation may also include: receiving a first vertical input on the touch panel while a first carousel of AR effect items is being displayed; and displaying a second carousel including a plurality of content items in response to the first vertical input.

[0037] In addition, the operation may include: receiving a second horizontal input on the touch pad; rotating a second carousel of content items in response to the second horizontal input; receiving a second touch input on the touch pad, the touch input being used to select a specific content icon in a selection position in the second carousel; and displaying additional user interface options in response to receiving the second touch input.

[0038] The first vertical input may be a downward slide, and the operation further includes displaying a content icon and a downward pointer above the first carousel. While a second carousel including a plurality of content items is being displayed, a second vertical input on the touch pad may be received, and in response to the second vertical input, the first carousel of AR effect items may be displayed.

[0039] The operation may further include: when the first carousel is being displayed, receiving a third vertical input on the touch panel, the third vertical input being in a direction opposite to the first vertical input, and displaying system information in response to the third vertical input.

[0040] Other technical features may be readily apparent to those skilled in the art from the following drawings, descriptions and claims.

[0041] Figure 1 1 is a perspective view of a head mounted device (e.g., glasses 100) according to some examples. The glasses 100 may include a frame 102, which is made of any suitable material, such as plastic or metal, including any suitable shape memory alloy. In one or more examples, the frame 102 includes a first optical element holder or a left optical element holder 104 (e.g., a display or lens holder) and a second optical element holder or a right optical element holder 106 connected by a bridge 112. A first optical element or a left optical element 108 and a second optical element or a right optical element 110 may be disposed in the left optical element holder 104 and the right optical element holder 106, respectively. Each of the right optical element 110 and the left optical element 108 may be a lens, a display, a display assembly, or a combination of the foregoing. Any suitable display assembly may be disposed in the glasses 100.

[0042] The frame 102 additionally includes a left arm or temple piece 120 and a right arm or temple piece 122. In some examples, the entire frame 102 can be formed from a single piece of material to have a unitary or unitary construction.

[0043] The glasses 100 may include a computing device such as a computer 118, which may be of any suitable type so as to be carried by the frame 102, and in one or more examples, the computing device may be of a suitable size and shape to be at least partially disposed in one of the temple pieces 120 and temple pieces 122. The computer 118 may include one or more processors as well as memory, wireless communication circuitry, and a power source. As discussed below, the computer 118 includes a low-power circuitry, a high-speed circuitry, and a display processor. Various other examples may include these elements in different configurations or integrated together in different ways. Additional details of various aspects of the computer 118 may be implemented as shown in the data processor 302 discussed below.

[0044] The computer 118 additionally includes a battery 116 or other suitable portable power supply. In one example, the battery 116 is disposed in the left temple piece 120 and is electrically coupled to the computer 118 disposed in the right temple piece 122. The glasses 100 may include a connector or port (not shown) suitable for charging the battery 116, a wireless receiver, transmitter or transceiver (not shown), or a combination of such devices.

[0045] The glasses 100 include a camera 114. Although two cameras are depicted, other examples contemplate the use of a single or additional (i.e., more than two) cameras. In one or more examples, the glasses 100 include any number of input sensors or other input / output devices in addition to the camera 114. Such sensors or input / output devices may additionally include biometric sensors, position sensors, motion sensors, etc.

[0046] The glasses 100 may also include a touchpad 124 mounted to or integrated with one or both of the left temple piece 120 and the right temple piece 122. The touchpad 124 is generally arranged vertically, and the touchpad 124 is approximately parallel to the temple of the user in one example. As used herein, generally vertical alignment means that the touchpad is at least more vertical than horizontal, although preferably more vertical than the vertical. Additional user input can be provided by one or more buttons 126, which in the illustrated embodiment are disposed on the outer upper edges of the left optical element holder 104 and the right optical element holder 106. One or more touchpads 124 and buttons 126 provide the following means by which the glasses 100 can receive input from the user of the glasses 100.

[0047] Figure 2 The eyeglasses 100 are shown from the wearer's perspective. Figure 1Several elements shown in FIG. have been omitted. Figure 1 As described, Figure 2 The illustrated eyeglasses 100 include a left optical element 108 and a right optical element 110 secured within each of a left optical element holder 104 and a right optical element holder 106, respectively.

[0048] The glasses 100 include a forward optical assembly 202 including a right projector 204 and a right near-eye display 206 , and a forward optical assembly 210 including a left projector 212 and a left near-eye display 216 .

[0049] In one embodiment, the near eye display is a waveguide. The waveguide includes a reflective structure or a diffractive structure (e.g., a grating and / or an optical element such as a mirror, lens, or prism). Light 208 emitted by projector 204 encounters the diffractive structure of the waveguide of near eye display 206, which directs the light toward the right eye of the user to provide an image on or in the right optical element 110 that is superimposed on the view of the real world seen by the user. Similarly, light 214 emitted by projector 212 encounters the diffractive structure of the waveguide of near eye display 216, which directs the light toward the left eye of the user to provide an image on or in the left optical element 108 that is superimposed on the view of the real world seen by the user.

[0050] However, it should be understood that other display technologies or configurations can be provided that can display images to a user in a forward field of view. For example, instead of providing a projector 204 and a waveguide, an LCD, LED or other display panel or surface can be provided instead.

[0051] In use, the wearer of the glasses 100 will be presented with information, content, and various user interfaces on the near-eye display. As described in more detail below, in addition to the associated devices such as Figure 3 In addition to providing voice input or touch input on the client device 328 as shown, the user can then interact with the glasses 100 using the touchpad 124 and / or buttons 126.

[0052] Figure 3 is a block diagram illustrating a networked system 300 including details of glasses 100 according to some examples.

[0053] The networked system 300 includes the glasses 100, a client device 328, and a server system 332. The client device 328 may be a smartphone, a tablet computer, a tablet phone, a laptop, an access point, or any other such device capable of connecting to the glasses 100 using both a low power wireless connection 336 and a high speed wireless connection 334. The client device 328 is connected to the server system 332 via a network 330. The network 330 may include any combination of wired and wireless connections. The server system 332 may be one or more computing devices that are part of a service or network computing system. The client device 328 and any elements of the server system 332 and the network 330 may use Fig.13 and Fig.14 The details of the software architecture 1304 or machine 1400 described in are implemented.

[0054] The glasses 100 include a data processor 302, a display 310, one or more cameras 308, and additional input / output elements 316. The input / output elements 316 may include a microphone, an audio speaker, a biometric sensor, an additional sensor, or an additional display element integrated with the data processor 302. Fig.13 and Fig.14 Examples of input / output elements 316 are further discussed. For example, input / output elements 316 may include any I / O components 1406, including output components 1428, motion components 1436, etc. Figure 2 Examples of display 310 are discussed in . In the particular examples described herein, display 310 includes a display for each of the user's left eye and right eye.

[0055] Data processor 302 includes image processor 306 (eg, video processor), GPU and display driver 338, tracking module 340, interface 312, low power circuitry 304, and high speed circuitry 320. Components of data processor 302 are interconnected by bus 342.

[0056] The interface 312 refers to any source of user commands provided to the data processor 302. In one or more examples, the interface 312 is a physical button that, when pressed, sends a user input signal from the interface 312 to the low-power processor 314. The low-power processor 314 can process pressing such a button and then immediately releasing it as a request to capture a single image, and vice versa. The low-power processor 314 can process pressing such a button within a first time period as a request to capture video data when the button is pressed and stop video capture when the button is released, wherein the video captured when the button is pressed is stored as a single video file. Alternatively, pressing the button for a long period of time can capture a still image. In other examples, the interface 312 can be any mechanical switch or physical interface capable of accepting user input associated with a data request from the camera 308. In other examples, the interface 312 can have a software component, or can be associated with a command received wirelessly from another source, such as from the client device 328.

[0057] Image processor 306 includes circuitry for receiving signals from camera 308 and processing those signals from camera 308 into a format suitable for storage in memory 324 or for transmission to client device 328. In one or more examples, image processor 306 (e.g., a video processor) includes a microprocessor integrated circuit (IC) customized for processing sensor data from camera 308, and volatile memory used by the microprocessor in operation.

[0058] The low power circuit system 304 includes a low power processor 314 and a low power wireless circuit system 318. These elements of the low power circuit system 304 can be implemented as separate elements or can be implemented on a single IC as part of a single system on a chip. The low power processor 314 includes logic for managing other elements of the glasses 100. As described above, for example, the low power processor 314 can accept user input signals from the interface 312. The low power processor 314 can also be configured to receive input signals or command communications from a client device 328 via a low power wireless connection 336. The low power wireless circuit system 318 includes circuit elements for implementing a low power wireless communication system. Bluetooth TM Smart, also known as Bluetooth TM Low power consumption is a standard implementation of a low power wireless communication system that may be used to implement the low power wireless circuitry 318. In other examples, other low power communication systems may be used.

[0059] High-speed circuit system 320 includes a high-speed processor 322, a memory 324, and a high-speed wireless circuit system 326. High-speed processor 322 can be any processor capable of managing high-speed communications and operations of any general-purpose computing system required by data processor 302. High-speed processor 322 includes the processing resources required to manage high-speed data transmission over high-speed wireless connection 334 using high-speed wireless circuit system 326. In some examples, high-speed processor 322 executes an operating system such as the LINUX operating system or a program such as Fig.13 The high-speed processor 322, which executes the software architecture of the data processor 302, manages data transmission with the high-speed wireless circuit system 326, in addition to any other duties. In a specific example, the high-speed wireless circuit system 326 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, which is also referred to herein as Wi-Fi. In other examples, the high-speed wireless circuit system 326 can implement other high-speed communication standards.

[0060] The memory 324 includes any storage device capable of storing camera data generated by the camera 308 and the image processor 306. Although the memory 324 is shown as being integrated with the high-speed circuitry 320, in other examples, the memory 324 may be a separate, independent element of the data processor 302. In particular such examples, electrical wiring may provide a connection from the image processor 306 or the low-power processor 314 to the memory 324 through a chip including the high-speed processor 322. In other examples, the high-speed processor 322 may manage the addressing of the memory 324 so that the low-power processor 314 will direct the high-speed processor 322 whenever a read or write operation involving the memory 324 is needed.

[0061] The tracking module 340 estimates the pose of the glasses 100. For example, the tracking module 340 uses image data and corresponding inertial data from the camera 308 and the positioning component 1440 and GPS data to track the position and determine the pose of the glasses 100 relative to a reference frame (e.g., a real-world environment). The tracking module 340 continuously collects and uses updated sensor data describing the movement of the glasses 100 to determine an updated three-dimensional pose of the glasses 100, which indicates changes in relative position and orientation relative to physical objects in the real-world environment. The tracking module 340 allows the glasses 100 to visually place virtual objects relative to physical objects within the user's field of view via the display 310.

[0062] The GPU and display driver 338 can use the pose of the glasses 100 to generate frames of virtual content or other content to be presented on the display 310 when the glasses 100 are operating in a conventional augmented reality mode. In this mode, the GPU and display driver 338 generate updated frames of virtual content based on the updated three-dimensional pose of the glasses 100, which reflects changes in the user's position and orientation relative to physical objects in the user's real environment.

[0063] One or more functions or operations described herein may also be performed in an application resident on the glasses 100 or on the client device 328 or on a remote server. For example, one or more functions or operations described herein may be performed by one of the applications 1306, such as the messaging application 1346.

[0064] Networked computing environment

[0065] Figure 4 is a block diagram illustrating an example messaging system 400 for exchanging data (e.g., messages and associated content) over a network. The messaging system 400 includes multiple instances of a client device 328, each of the multiple instances hosting a number of applications including a messaging client 402 and other applications 404. Each messaging client 402 is communicatively coupled to other instances of the messaging client 402 (e.g., hosted on respective other client devices 328), a messaging server system 406, and a third-party server 408 via a network 330 (e.g., the Internet). The messaging client 402 may also communicate with the locally hosted application 404 using an application program interface (API).

[0066] The messaging clients 402 are able to communicate and exchange data with other messaging clients 402 and with a messaging server system 406 via the network 330. The data exchanged between the messaging clients 402 and between the messaging clients 402 and the messaging server system 406 includes functions (e.g., commands for activating functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0067] The messaging server system 406 provides server-side functionality to specific messaging clients 402 via the network 330. Although specific functionality of the messaging system 400 is described herein as being performed by the messaging client 402 or by the messaging server system 406, the location of specific functionality within the messaging client 402 or within the messaging server system 406 may be a design choice. For example, it may be technically preferable to initially deploy specific technologies and functionality within the messaging server system 406, but to later migrate the technologies and functionality to the messaging client 402 where the client device 328 has sufficient processing power.

[0068] The messaging server system 406 supports various services and operations provided to the messaging clients 402. Such operations include sending data to the messaging clients 402, receiving data from the messaging clients 402, and processing data generated by the messaging clients 402. As examples, the data may include message content, client device information, geolocation information, media enhancements and overlays, message content persistence conditions, social network information, and live event information. The data exchange within the messaging system 400 is activated and controlled by functions available via the user interface (UI) of the messaging client 402.

[0069] Turning now specifically to the messaging server system 406, an application program interface (API) server 410 is coupled to and provides a programming interface to an application server 414. The application server 414 is communicatively coupled to a database server 416, which facilitates access to a database 420 that stores data associated with messages processed by the application server 414. Similarly, a web server 424 is coupled to and provides a web-based interface to the application server 414. To this end, the web server 424 handles incoming network requests via the Hypertext Transfer Protocol (HTTP) and several other related protocols.

[0070] The application program interface (API) server 410 receives and sends message data (e.g., commands and message payloads) between the client device 328 and the application server 414. Specifically, the application program interface (API) server 410 provides a set of interfaces (e.g., routines and protocols) that can be called or queried by the messaging client 402 to activate the functions of the application server 414. The application program interface (API) server 410 exposes various functions supported by the application server 414, including: account registration; login functionality; sending messages from a particular messaging client 402 to another messaging client 402 via the application server 414, sending media files (e.g., images or videos) from the messaging client 402 to the messaging server 412, and possible access by another messaging client 402; setting up a collection of media data (e.g., a story); retrieving a friend list of a user of the client device 328; retrieving such a collection; retrieving messages and content; adding and removing entities (e.g., friends) to an entity graph (e.g., a social graph); locating friends in a social graph; and opening application events (e.g., related to the messaging client 402).

[0071] The application server 414 hosts several server applications and subsystems, including, for example, a messaging server 412, an image processing server 418, and a social network server 422. The messaging server 412 implements several message processing technologies and functions, particularly those related to the aggregation and other processing of content (e.g., text and multimedia content) included in messages received from multiple instances of the messaging client 402. As will be described in further detail, text and media content from multiple sources can be aggregated into collections of content (e.g., referred to as stories or galleries). These collections are then made available to the messaging client 402. In view of the hardware requirements for such processing, other processor- and memory-intensive processing of data can also be performed on the server side by the messaging server 412.

[0072] The application server 414 also includes an image processing server 418 that is dedicated to performing various image processing operations, typically with respect to images or videos within the payload of messages sent from or received at the messaging server 412.

[0073] The social network server 422 supports various social networking functions and services and makes these functions and services available to the messaging server 412. To this end, the social network server 422 maintains and accesses an entity graph within the database 420. Examples of functions and services supported by the social network server 422 include identifying other users in the messaging system 400 with whom a particular user has relationships or who the particular user is "following," and also includes identifying interests and other entities of a particular user.

[0074] Returning to the messaging client 402, the features and functions of the external resource (e.g., application 404 or applet) are available to the user via the interface of the messaging client 402. In this context, "external" refers to the fact that the application 404 or applet is outside the messaging client 402. External resources are usually provided by a third party, but can also be provided by the creator or provider of the messaging client 402. The messaging client 402 receives a user selection of an option for launching or accessing the features of such an external resource. The external resource can be an application 404 (e.g., a "local app") installed on the client device 328, or a small-scale version of the application (e.g., a "applet") hosted on the client device 328 or away from the client device 328 (e.g., on a third-party server 408). The small-scale version of the application includes a subset of the features and functions of the application (e.g., the full-scale, local version of the application) and is implemented using a markup language document. In one example, the small-scale version of the application (e.g., a "applet") is a web-based markup language version of the application and is embedded in the messaging client 402. In addition to using markup language documents (eg, .*ml files), applets may include scripting languages ​​(eg, .*js files or .json files) and style sheets (eg, .*ss files).

[0075] In response to receiving a user selection of an option to launch or access a feature of an external resource, the messaging client 402 determines whether the selected external resource is a web-based external resource or a locally installed application 404. In some cases, the application 404 installed locally on the client device 328 can be launched independently of the messaging client 402 and separately from the messaging client 402, for example, by selecting an icon corresponding to the application 404 on the home screen of the client device 328. As used herein, an icon may include one or both of a text element and a graphical element. A small-scale version of such an application can be launched or accessed via the messaging client 402, and in some examples, no part or a limited part of the small-scale application can be accessed outside the messaging client 402. The small-scale application can be launched by the messaging client 402, for example, receiving a markup language document associated with the small-scale application from a third-party server 408 and processing such a document.

[0076] In response to determining that the external resource is a locally installed application 404, the messaging client 402 instructs the client device 328 to launch the external resource by executing the locally stored code corresponding to the external resource. In response to determining that the external resource is a web-based resource, the messaging client 402 communicates with (for example) a third-party server 408 to obtain a markup language document corresponding to the selected external resource. The messaging client 402 then processes the obtained markup language document to present the web-based external resource within the user interface of the messaging client 402.

[0077] The messaging client 402 may notify the user of the client device 328 or other users (e.g., "friends") associated with such a user of activities occurring in one or more external resources. For example, the messaging client 402 may provide the following notification to participants in a conversation (e.g., a chat session) in the messaging client 402, which is related to the current or recent use of the external resource by one or more members in the user group. One or more users may be invited to join an active external resource or to start an external resource that is recently used but currently inactive (in the friend group). The external resource may provide the participants in the conversation, each participant using the corresponding messaging client 402, with the ability to share items, conditions, states, or locations in the external resource with one or more members in the group of users entering the chat session. The shared item may be an interactive chat card, and members of the chat may interact using the interactive chat card, such as to start the corresponding external resource, view specific information in the external resource, or bring members of the chat to a specific location or state in the external resource. In a given external resource, a response message may be sent to the user on the messaging client 402. The external resource may selectively include different media items in the response based on the current context of the external resource.

[0078] Figure 5 5 is a perspective view of a head mounted device (e.g., glasses 500) according to another example. It can be seen that in this example, the touch pad 502 is integrated into the frame 504, the front portion of which surrounds the user's eyes. The presence of the touch pad 502 in the glasses 500 is therefore less obtrusive and the overall appearance of the head mounted device is more aesthetically pleasing.

[0079] Navigating the user interface on glasses 500 using touchpad 502 is performed by presenting some user interface elements (e.g., icons or content galleries) using a carousel arrangement, which is arranged in Figure 5 506. As seen by the wearer of the glasses 500, user interface elements (e.g., text 508) are presented, for example, in a curved or quasi-curved arrangement on the surface of the cylinder 506. Forward and backward swipe inputs 510 received by the glasses 500 from the user along the touchpad 502 are converted into rotations 512 of the user interface elements around the cylinder 506. More specifically, a forward swipe gesture on the right-hand touchpad 502 results in a forward movement of the right-hand side of the cylinder 506, causing the user interface element to perceptibly rotate around the surface of the cylinder 506 in a counterclockwise direction (as viewed from above), and vice versa.

[0080] An upward or downward swipe gesture on touch pad 502 typically translates into selection of, or transition to, a different user interface screen, although the action may also translate into scrolling up or down, respectively, in the current user interface screen, such as in zooming in or out of a zoomed image presented in near-eye display 216 of glasses 100.

[0081] When the forward or backward sliding gesture continues to be received by the glasses 500, the user interface element or content will usually disappear from the right or left side, respectively, while the additional user interface element or content will appear from the left or right side, respectively. When the user interface element or content moves toward the edge of the display, a visual cue can be provided to illustrate the cylinder metaphor. For example, the movement of the user interface element away from the center toward the edge can cause the size of a particular element to decrease, and vice versa. In addition, the content or user interface elements located closer to the center can gradually become overlapped with the content or user interface elements adjacent to the outside, where they become more stacked as these elements approach the edge of the cylinder. Depending on the implementation, as the content or user interface elements move toward the edge, the content or user interface elements themselves can rotate away around their central axis, or the content or user interface elements themselves may not rotate away around their central axis. That is, in some cases, when the element is reduced in size or becomes stacked, the element still faces the user.

[0082] FIG. 6 shows a diagram according to some examples. Figure 1 or Figure 5 More specifically, Fig. 6A In some examples Figure 1 or Figure 5 6 shows an example of a user interface screen 602 shown on one or both of the displays 310 / 216 of the glasses shown in FIG. As can be seen from the figure, the user interface screen 602 includes a carousel 610 of icons, which in the illustrated embodiment includes an AR effect icon 612, an AR effect icon 614, an AR effect icon 616, an AR effect icon 618, and an AR effect icon 620 occupying positions from left to right on the user interface screen 602. An empty icon 622 occupying a center position 624 in the carousel 610 is also disclosed. The circle showing the center position 624 is larger to indicate that the icon in this position is available for selection in this example, but it can also indicate that the icon in this position has been selected or is active. When in the center position 624 in the carousel 610, the empty icon 622, which is a solid color such as white or black in the example, indicates that no icon or action is available for selection or has been selected.

[0083] In addition, it can be seen that the icons in the carousel 610 overlap in a gradually larger amount along the direction from the center position 624 to the edge of the user interface screen 602 to provide the above reference Figure 5 For example, AR effect icon 612 overlaps slightly with AR effect icon 614, which overlaps more with AR effect icon 616, which in turn overlaps even more with AR effect icon 618. In this example of a cylinder metaphor, the icons are shown facing up across carousel 610 and are the same size (except for the icon in center position 624). In other implementations, the icons may also or alternatively be reduced in size or rotated away, as discussed above.

[0084] A forward or backward sliding input on the touch pad 124 / 502 will cause the AR effect icons in the carousel 610 to move left or right, as shown below. Fig. 8A and Figure 8B The specific direction in which the carousel moves will depend on whether the touchpad on the left or right side of the glasses 100 / 500 is used as discussed above. When the AR effect icon occupies the center position, the AR effect icon can be selected by tapping on the touchpad 124 / 502.

[0085] The user interface screen 602 also includes navigation prompts to provide the user with instructions on the type and direction of input required to transition between different screens or modes. The navigation prompts provide information including both orientation prompts and contextual prompts for system navigation. The navigation prompts show both the input action to be taken and the destination or result of the input action.

[0086] For example, navigation prompt 626 includes a content gallery icon 630 and a pointer 632. Content gallery icon 630 provides informational indication that a content gallery can be accessed, and downward pointer 632 provides informational indication that swiping downward on touch pad 124 / 502 will initiate a transition to the gallery screen as indicated by content gallery icon 630. Navigation prompt 626 is located in a position in user interface screen 602 at the starting point of the direction of the requested user input. That is, navigation prompt 626 is located at the top of user interface screen 602, while the corresponding input action on the touch pad is downward toward the bottom of the touch pad.

[0087] Similarly, navigation prompt 628 includes pointer 634 and settings icon 636. Settings icon 636 provides informational indication of settings screens that can be accessed, and upward pointer 634 provides informational indication that an upward swipe on touchpad 124 / 502 will initiate a transition to the settings screen as indicated by settings icon 636. Navigation prompt 628 is located in a position in user interface screen 602 at the starting point of the direction of the requested user input. That is, navigation prompt 628 is located at the bottom of user interface screen 602, while the corresponding input action on the touchpad is upward toward the top of touchpad 124 / 502.

[0088] Figure 6B The user interface screen 602 and Figure 6C 6. As indicated by the content gallery icon 630 and pointer 632 discussed above, while the user interface screen 602 is displayed, the user interface screen 604 begins to be shown in response to receiving a downward swipe on the touch pad 124 / 502.

[0089] It can be seen that in the user interface screen 604, the carousel 610, the navigation prompt 626, and the navigation prompt 628 all move downward, which is consistent with the downward swipe received on the touchpad 124 / 502. Additionally, the carousel 610 bends downward at the center position 624 as an additional visual confirmation of the downward swipe input. When the swipe input is received, the content gallery icon 630 will also transition to a square. Since the user input received is opposite to the swipe direction corresponding to the navigation prompt 628, the navigation prompt 628 can also be reduced in size or grayed out to de-emphasize the option.

[0090] When a continuous downward swipe is received on the touch pad 124 / 502 beyond a certain predetermined point, the user interface screen 604 will transition to the user interface screen 606 discussed below. If the downward swipe is terminated before reaching the predetermined point, the user interface screen 604 will quickly return to the user interface screen 602.

[0091] Figure 6C 6 is an example of a content gallery user interface screen 606 as prompted by the content gallery icon 630 and the pointer 632 and as discussed above in response to receiving a downward swipe on the touch pad 124 / 502 while the user interface screen 602 is displayed.

[0092] The user interface screen 606 includes a carousel 646 of content items, which may be, for example, images or videos saved to a content gallery associated with the user and stored on the client device 328, glasses 100 / 500, or stored remotely. In one example, the content items include images or videos that have been captured by the user using the client device 328 or glasses 100 / 500 while having the selected AR effect applied.

[0093] In the example shown, it can be seen that carousel 646 includes content item 650 in the center position in user interface screen 606, as well as partially obscured content item 648 and content item 652. As before, a forward or backward swipe input on touchpad 124 / 502 will cause the content items in carousel 646 to move left or right, wherein as content item 650 moves left or right, the content items will sequentially replace the current content item in the center position. The specific direction in which content item 650 moves will depend on whether the left or right touchpad of glasses 100 / 500 is used as discussed above. Content item 650 occupying the center position can be selected by tapping on touchpad 124 / 502. Reference will be made to Fig.9A Discuss the consequences of such choices.

[0094] The user interface screen 606 also includes a navigation prompt 638, which includes an AR carousel icon 640 and a pointer 642. As before, the navigation prompt 638 provides information that the AR effect carousel 610 shown in the user interface screen 602 can be accessed / returned, and the downward pointer 642 provides information that swiping downward on the touch pad 124 / 502 will initiate a transition to the user interface screen 602 indicated by the AR carousel icon 640. In addition, as before, the navigation prompt 638 is located in the user interface screen 606 at a location at the starting point of the direction of the required user input. That is, the navigation prompt 638 is located at the top of the user interface screen 606, while the corresponding input action on the touch pad is downward toward the bottom of the touch pad.

[0095] Also included is a menu indicator or overflow indicator icon 644 indicating that additional user interface options are available in the user interface screen 606. The lack of a pointer indicates that these options are accessed not by a vertical swipe on the touchpad 124 / touchpad 502 but by a long press on the touchpad 124 / 502. In one example, such a long press will open a carousel of UI icons that can be navigated using scroll input, selected using touch input, and cancelled using a downward swipe. In one example, the carousel of selectable UI icons includes the following reference Fig. 9B A zoom icon 912, a send icon 914, and a delete icon 916 are discussed.

[0096] Fig.6D is in the user interface screen 606 and Fig. 6A 602. As indicated by the AR carousel icon 640 and pointer 642 discussed above, while user interface screen 606 is displayed, user interface screen 608 begins to be displayed in response to receiving a downward swipe on touchpad 124 / 502.

[0097] It can be seen that in the user interface screen 608, the carousel 646, the navigation prompt 638, and the icon 644 all move downward, which is consistent with the downward swipe received on the touch pad 124 / 502. Additionally, the content item 650 in the center position 624 moves further downward and decreases in size as an additional visual confirmation of the downward swipe input. When the swipe input is received, the size of the content gallery icon 630 also increases.

[0098] When a continuous downward swipe is received on the touch pad 124 / 502 beyond a certain predetermined point, the user interface screen 608 will transition to the user interface screen 602 discussed above. If the downward swipe is terminated before reaching the predetermined point, the user interface screen 608 will quickly return to the user interface screen 606.

[0099] Fig. 7A and Figure 7B shows the Figure 1 or Figure 5 More specifically, Fig. 7A and Figure 7B A series of user interface screens are shown that are shown when an upward swipe is received on the touch pad while displaying the user interface screen 602. Upon receiving such an upward swipe prompted by the navigation prompt 628 including the settings icon 636 and the pointer 634, the settings user interface screen 702 is displayed. It can be seen that the user interface screen 702 includes various information about the glasses 100 / 500 or the client device 328, such as battery power, speaker volume, Wi-Fi network identification and signal strength, display brightness, user name and avatar, time, date, etc.

[0100] As indicated by pointer 704 , upon receiving a downward swipe on touchpad 124 / 502 while user interface screen 702 is displayed, the screen will return to user interface screen 602 .

[0101] FIG. 8A to FIG. 8C According to some examples, Figure 1 or Figure 5 More specifically, FIG. 8A to FIG. 8C Shows the use of Fig. 6A Carousel 610 in FIG. 6 depicts an example user interface for selection and application of AR effects.

[0102] Fig. 8A The user interface screen 602 is shown with the empty icon 622 occupying the center position 624 in the carousel 610. Upon receiving an appropriate forward or backward sliding action on the touchpad 124 / 502, the icons in the carousel 610 rotate or scroll to the position shown in the user interface screen 802, where the AR effect icon (e.g., the AR effect icon 612) occupies the center position 624. It can be seen that the AR effect icon 612 is more prominent not only by displaying it in the center position, but also by increasing its size compared to its previous position shown in the user interface screen 602. When the AR effect icon 612 occupies the center position 624, related information (bibliographic information) 806, such as the name of the AR effect corresponding to the AR effect icon 612 and its author, can also be displayed in the user interface screen 802.

[0103] The AR effect icon 612 shown in the center position 624 in the user interface screen 802 is now selectable or activated. Figure 8C In one example, user interface screen 804 is shown, receiving a tap on touchpad 124 / 502 by a user to cause application of an AR effect corresponding to AR effect icon 612. In this example, a real-time scene captured by one or more of cameras 114 includes a building 808 that has been enhanced by several individual AR effects, such as 3D text AR effect 810 and floating sphere AR effect 812. As will be appreciated, the AR effect corresponding to any AR effect icon may include several individual AR effects.

[0104] User interface screen 804 also includes a navigation prompt 814, which includes "Exit" text 816 and a downward pointer 818. Upon receiving a downward swipe on touch pad 124 / 502 as indicated by navigation prompt 814, user interface screen 804 will be dismissed and will return to user interface screen 802.

[0105] When the glasses are 100 / 500 Figure 8CWhen in the state shown in and described above, one of the buttons 126 on the glasses can be used to start capturing the AR effects of the scene and its various applications. In one example, receiving a short button press on the left button 126 starts video capture of the scene, and then receiving a long button press on the left button terminates the ongoing video capture. If there is no video capture in progress, receiving a long button press on the left button starts still image capture. Of course, various combinations of button presses and button durations can be used to start and terminate the capture of content items. When the content capture initiated by receiving user input on button 126 is completed, the user interface screen 902 described below is displayed.

[0106] 9A to 9F According to some examples, Figure 1 or Figure 5 More specifically, 9A to 9F An example user interface depicting forwarding a content item is shown. Fig. 9B The content preview user interface screen 902 is shown in FIG. Fig.9A The illustrated display user interface screen 606 is reached by receiving a tap on the touch pad 124 / 502 (to select the content item 918 in the center position in the carousel 646) or as described above with reference to Figure 8C Arrived after the described content capture is terminated. For purposes of clarity, the remaining user interface screens of FIG. 9 will be described with reference to content item 918 in content gallery user interface screen 606, but it should be understood that these examples are equally applicable to content that has just been captured.

[0107] Fig. 9B 902 includes a preview of a content item 918, a zoom icon 912, a send icon 914, and a delete icon 916. In the example shown, the send icon 914 is highlighted as an initial default value for input to be received in the user interface screen 902. In this state, receiving a tap on the touchpad 124 / touchpad 502 will cause a transition to the recipient selection user interface screen 904. However, upon a related forward or backward swipe input on the touchpad 124 / 502, the zoom icon 912 or the delete icon 916 may be highlighted (and therefore selectable).

[0108] When zoom icon 912 is selected, receiving a tap on touchpad 124 / 502 will transition to a content viewer user interface (not shown) where content item 918 will be played back (if the content item is a video or gif or has a dynamic AR effect), or displayed in a larger format if the content item 918 is a still image. As before, receiving a downward swipe in the content viewer user interface will return to user interface screen 902.

[0109] With delete icon 916 selected, receiving a tap on touchpad 124 / 502 will discard content item 918 (possibly after requesting confirmation) and return to the previous user interface, which may be user interface screen 606 or user interface screen 804 .

[0110] When the send icon 914 is selected, receiving a tap on the touch pad 124 / 502 will transition to Fig. 9C 904, in which a carousel 920 of potential recipients, such as recipient 922 and recipient 924, is shown. In this context, a recipient may be a single recipient, a group of recipients (e.g., "Brunch Club" recipient 924, "Best Friends"), or a non-human recipient (e.g., "My Story" recipient 922). For example, selection of a non-human recipient may cause a content item to be added to a social media or messaging application feed, or a map indicating the location of the scene and identifying user information, provided that appropriate permissions have been obtained to allow this.

[0111] As before, a forward or backward swipe input on the touchpad 124 / 502 will cause the direction of the recipients in the carousel 920 to move left or right, with the recipients sequentially replacing the recipients in the center position as the carousel 920 scrolls left or right. The specific direction that the carousel 920 moves in response to a forward or backward swipe will depend on whether the left or right touchpad of the glasses 100 / 500 is engaged as described above with reference to FIG. Figure 1 Use as discussed.

[0112] The recipient occupying the center position of the carousel 920 may be selected by tapping on the touch pad 124 / 502. The center-occupied recipient may be highlighted, such as by being enclosed in a frame 928 or by using another highlighting technique such as increasing the size or by adjusting the color relative to adjacent recipients.

[0113] While in user interface screen 904, receiving a tap input on touchpad 124 / 502 will select (or deselect if already selected) a Fig.9D924 in the center position. As shown, the selected recipient is indicated using a check mark 930. Additionally, once the recipient has been selected, a navigation prompt 932 including a send icon 934 and a pointer 936 is displayed. The navigation prompt 932 is located in the user interface screen 908 at a location at the starting point of the direction of the required user input. That is, the navigation prompt 932 is located at the bottom of the user interface screen 602, while the corresponding input action on the touch pad is upward toward the top of the touch pad 124 / 502.

[0114] like Fig.9E As shown, after one recipient has been selected, additional recipients may be selected by continued scrolling of carousel 920 in response to forward or backward touch input on touch pad 124 / 502 to bring another recipient 922 to center position 624, the selection of which is also shown using check mark 938. After one or more recipients have been selected as shown in user interface screen 906 or user interface screen 908, receiving an upward swipe on touch pad 124 / 502 as prompted by navigation prompt 932 will cause a transition to user interface screen 910.

[0115] like Fig.9F As shown, user interface screen 910 includes a send icon 934 and a cloud or list of selected recipients 940. Also included is a text prompt 942 indicating "Tap to Send". Receiving a tap input on touchpad 124 / 500 will send content item 918 to the selected recipient. Glasses 100 / 500 will then return to displaying the content item 918 based on how user interface screen 902 was reached. Figure 8C User interface screen 804 or Fig.9A and Figure 6C In other examples, the glasses may return to another user interface screen, such as user interface screen 602.

[0116] As shown by the downward pointer 926, when a downward swipe is received on the touch pad 124 / 502, FIG. 9B to FIG. 9F In one example, a downward swipe received while displaying user interface screen 910 returns to user interface screen 908, while a downward swipe received while displaying user interface screen 904, user interface screen 906, or user interface screen 908 returns to user interface screen 902, while a downward swipe received while displaying user interface screen 902 returns to user interface screen 606.

[0117] Fig.10A user interface flow chart 1000 is shown that can be implemented by the glasses 100 / 500 according to some examples. The flow chart 1000 begins with the glasses 100 / 500 in a sleep state. The glasses 100 / 500 wake up upon receiving a touch input 1004 on the touch pad 124 / 502 or a press on one of the buttons 126. In response, with the glasses 100 / 500 locked, a user interface screen 1002 is presented prompting for a PIN code. As can be seen, the user interface screen 1002 includes a keyboard display 1006 and an input field 1008.

[0118] As shown by the tap and swipe input 1010, the keyboard display 1006 is traversed by forward and backward swipe inputs received on the touchpad 124 / 502, and upon receiving a tap input on the touchpad 124 / 502, the highlighted number is selected for inclusion and displayed in the input field 1008. Upon receiving the correct PIN code, the glasses 100 / 500 will transition to the user interface screen 602 showing the AR effects carousel 610 described above.

[0119] Receiving downward swipe input 1012 transitions from user interface screen 602 to user interface screen 606 corresponding to the content gallery, and receiving downward swipe input 1012 transitions from content gallery user interface screen 606 back to user interface screen 602 .

[0120] In the user interface screen 606, the slide input 1014 can be used to Fig.9A As shown by input 1016, within user interface screen 606, receiving a tap input transitions to user interface screen 902 showing a preview of a content item as discussed in more detail above, and within user interface screen 902, receiving a downward swipe returns to user interface screen 606. Within user interface screen 902, swipe input 1018 can be used to select zoom icon 912, send icon 914, or delete icon 916 as discussed above with reference to FIG. 9.

[0121] As shown by input 1020 , receiving an upward swipe input transitions from user interface screen 602 to settings or system information user interface screen 702 , while receiving a downward swipe input transitions from user interface screen 702 back to user interface screen 602 .

[0122] Similarly, as shown by input 1022, from user interface screen 602 in which the icon is in center position 624, receiving a tap input causes user interface screen 804 to be displayed in which the AR effect is applied, and receiving a downward swipe input transitions from user interface screen 804 back to user interface screen 602. Fig.10 Associated additional functions or screens.

[0123] Fig.11 1100 is an illustration of operations performed by the glasses 100 / 500 in response to receiving user input on the touch pad according to some examples. For illustrative purposes, the operations of flowchart 1100 are described herein as occurring serially or linearly. However, multiple operations of flowchart 1100 may occur in parallel. In addition, the operations of flowchart 1100 need not be performed in the order shown and / or one or more blocks of flowchart 1100 need not be performed and / or may be replaced by other operations.

[0124] Fig.11 The operations shown will generally be performed on the related hardware and data processor 302 in or associated with the glasses 100 / 500. For the purpose of clarity, the flowchart 1100 is discussed herein with reference to such an example. Various implementations are of course possible, where some of the operations described occur in an application such as a messaging application 1346 in a client device 328, on a server system 332, or where an application on a client device 328 calls another application or SDK to obtain the desired functionality. In one example, the operations are performed in conjunction between the messaging application 1346 running on the client device 328 and the related hardware and data processor 302 in or associated with the glasses 100.

[0125] The method starts at operation 1102, where the glasses 100 / 500 display a user interface screen 602 showing an AR effect carousel 610, wherein above the AR effect carousel 610 is a navigation prompt 626, and below the AR effect carousel 610 is a navigation prompt 628. As mentioned, the navigation prompt includes an icon and a pointer, the icon indicating the result or destination of the sliding input on the touchpad, and the pointer indicating the direction of the sliding input, wherein the navigation prompt is located in the user interface screen 602 at a position at the starting point of the direction of the required user input.

[0126] In operation 1104, upon receiving a vertical (i.e., upward or downward) touch input on the touchpad 124 / 502, in operation 1106, the glasses 100 / 500 display a corresponding user interface as indicated by the navigation prompt 626 or the navigation prompt 628, for example, the user interface screen 606 or the user interface screen 702 as described above with reference to Figures 6 and 7, respectively. At operation 1108, upon receiving a downward touch input in the user interface screen, in operation 1110, the glasses return to displaying the AR effect carousel 610 in operation 1102.

[0127] While displaying the user interface screen at operation 1106, receipt of a horizontal (e.g., forward or backward) touch input at operation 1120 causes scrolling of the displayed items at operation 1122 (if appropriate, e.g., for user interface screen 606 rather than for user interface screen 702), followed by a return to display of the relevant user interface screen at operation 1106.

[0128] During the display of the user interface screen at operation 1106, receiving a tap input at operation 1112 causes display of a corresponding user interface screen at operation 1114 (if appropriate, e.g., from user interface screen 606 instead of from user interface screen 702), wherein the item selected by scrolling as described in operations 1120 to 1122 is presented on the user interface screen. For example, content item 918 as presented in user interface screen 606. Then, additional touch input may be received at operation 1116, and corresponding actions taken at operation 1118. With reference herein to FIGS. 9, Fig.10 and Fig.13 Examples of such inputs and corresponding actions are discussed.

[0129] During display of the AR effect user interface screen 602 by the glasses 100 / 500 at operation 1102, receiving a horizontal (e.g., forward or backward) touch input at operation 1126 causes the AR effect icon to be scrolled at operation 1128, as discussed above with reference to FIG. 6. With the AR effect icon in the center position, upon receiving a tap input at operation 1130, the user interface screen 804 is displayed, in which the corresponding AR effect is applied to the scene viewed by the camera 114 at operation 1132. Then, additional touch inputs received at operation 1134 will cause corresponding actions to be taken at operation 1136. With reference to FIGS. 8, 9, Fig.10 and Fig.13 Examples of such inputs and corresponding actions are discussed.

[0130] After completing the corresponding actions in operation 1118 or 1136, glasses 100 / 300 typically return to user interface screen 602 or user interface screen 606 to perform additional operations as described herein.

[0131] Fig.121 is a flowchart 1200 illustrating content item selection and sending operations performed by the glasses 100 / 500 in response to receiving user input on a touch pad according to some examples. For illustrative purposes, the operations of the flowchart 1200 are described herein as occurring serially or linearly. However, multiple operations of the flowchart 1200 may occur in parallel. In addition, the operations of the flowchart 1200 need not be performed in the order shown and / or one or more blocks of the flowchart 1200 need not be performed and / or may be replaced by other operations.

[0132] Fig.12 The operations shown will generally be performed on the related hardware and data processor 302 in or associated with the glasses 100 / 500. For the purpose of clarity, the flowchart 1200 is discussed herein with reference to such an example. Various implementations are of course possible, where some of the operations described occur in an application such as a messaging application 1346 in a client device 328, on a server system 332, or where an application on a client device 328 calls another application or SDK to obtain the desired functionality. In one example, the operations are performed in conjunction between the messaging application 1346 running on the client device 328 and the related hardware and data processor 302 in or associated with the glasses 100.

[0133] The method begins at operation 1202, where the glasses 100 / 500 display a content preview user interface screen 902. As discussed above, this screen can be reached by scrolling and tapping to select a content item in the user interface screen 606 or due to a content capture user input received in the user interface screen 804. The user interface screen 902 includes three options, which can be selected in operation 1204 in response to scrolling in response to horizontal touch input (forward or backward) and touch input received on the touch pad 124 / 502, as discussed above with reference to Fig. 9B discussed.

[0134] When delete icon 916 is selected, upon receiving the touch input, the content item shown in user interface screen 902 is deleted in operation 1206, and glasses 100 / 500 return to the previous user interface screen displayed before user interface screen 902 in operation 1208. In one example, this would be user interface screen 606 or user interface screen 804.

[0135] When the zoom icon 912 is selected, in operation 1204, upon receiving a tap on the touchpad 124 / 502, a content viewer user interface is displayed by the glasses 100 / 500 in operation 1210, in which the content item shown in the user interface screen 902 is played back (in the case where the content item is a video or gif or has a dynamic AR effect), or displayed in a larger format if the content item is a still image. In operation 1212, receiving a downward swipe on the touchpad 124 / 502 will cause a return in operation 1214 to display the content preview user interface screen 902 at operation 1202.

[0136] When the send icon 914 is selected, upon receiving a tap on the touch pad 124 / 502 in operation 1204 , the address book or recipient selection user interface screen 904 is displayed in operation 1216 , showing a carousel 920 of potential recipients.

[0137] As before, in operation 1218, a forward or backward sliding input on touch pad 124 / 502 will cause the recipient direction in recipient selection carousel 920 to move left or right in operation 1220, where the recipients will sequentially replace the recipient in the center position as the carousel scrolls left or right.

[0138] In operation 1224, the recipient occupying the center position in the carousel 920 may be selected in response to the tap input on the touch pad 124 / 502 in operation 1222, or deselected if already selected. Fig.9D and Fig.9E As discussed, the recipient that has been selected is indicated with a check mark, and once the recipient is selected, a "Send" navigation prompt is displayed.

[0139] If a touch input corresponding to a send instruction is not received at operation 1226, the method returns to operation 1216 to allow selection of additional recipients in operations 1218 through 1224. As shown in user interface screen 906 or user interface screen 908, after one or more recipients have been selected, receipt of a send input at operation 1126, such as an upward swipe on touch pad 124 / 502 as prompted by navigation prompt 932, will cause a transition to a confirmation user input screen, such as user interface screen 910, at operation 1228. Receiving a confirmation user input at operation 1230, such as a tap input in response to a send icon 934, causes glasses 100 / 500 or an associated client device 328 to send the content item to the selected recipients in operation 1232.

[0140] In operation 1232, after completing the transmission of the content item to the selected recipient, in operation 1234, the glasses 100 / 300 will typically return to displaying the content preview user interface screen 902, depending on how the content preview user interface screen 902 was reached. Figure 8C User interface screen 804 or Fig.9A / Figure 6C 606. In other examples, the glasses can return to another user interface screen, such as user interface screen 602, at which point additional operations can be performed as described herein.

[0141] Fig.13 1300 is a block diagram illustrating a software architecture 1304 that may be installed on any one or more of the devices described herein. The software architecture 1304 is supported by hardware, such as a machine 1302, including a processor 1320, a memory 1326, and an I / O component 1338. In this example, the software architecture 1304 may be conceptualized as a stack of layers in which each layer provides specific functionality. The software architecture 1304 includes layers such as an operating system 1312, a library 1308, a framework 1310, and an application 1306. In operation, the application 1306 invokes an API call 1350 through the software stack and receives a message 1352 in response to the API call 1350.

[0142] The operating system 1312 manages hardware resources and provides public services. The operating system 1312 includes, for example, a kernel 1314, services 1316, and drivers 1322. The kernel 1314 serves as an abstraction layer between the hardware and other software layers. For example, the kernel 1314 provides functions such as memory management, processor management (e.g., scheduling), component management, networking, and security settings. Services 1316 can provide other public services to other software layers. Drivers 1322 are responsible for controlling or interfacing with the underlying hardware. For example, drivers 1322 may include display drivers, camera drivers, or Low-power drivers, Flash drivers, Serial communications drivers (e.g., Universal Serial Bus (USB) drivers), Drivers, audio drivers, power management drivers, etc.

[0143] The libraries 1308 provide a low-level common infrastructure used by the applications 1306. The libraries 1308 may include system libraries 1318 (e.g., C standard libraries) that provide functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. In addition, the libraries 1308 may include API libraries 1324, such as media libraries (e.g., libraries for supporting presentation and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), graphics libraries (e.g., OpenGL framework for presentation in two dimensions (2D) and three dimensions (3D) in graphics content on a display), database libraries (e.g., SQLite providing various relational database functions), web libraries (e.g., WebKit providing web browsing functions), etc. The library 1308 may also include various other libraries 1328 to provide many other APIs to the application 1306 .

[0144] The framework 1310 provides a high-level common infrastructure used by the applications 1306. For example, the framework 1310 provides various graphical user interface (GUI) functions, advanced resource management, and advanced location services. The framework 1310 can provide a wide range of other APIs that can be used by the applications 1306, some of which can be specific to a particular operating system or platform.

[0145] In an example, applications 1306 may include a home application 1336, a contacts application 1330, a browser application 1332, a book reader application 1334, a location application 1342, a media application 1344, a messaging application 1346, a game application 1348, and a variety of other applications such as third-party applications 1340. Applications 1306 are programs that execute functions defined in the program. Various programming languages ​​may be used to create one or more of the applications 1306 constructed in various ways, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C language or assembly language). In a specific example, third-party applications 1340 (e.g., those created by entities other than the vendor of a particular platform using ANDROID TM or IOS TM Software Development Kit (SDK) can be used to develop applications on platforms such as IOS TM ANDROID TM , Mobile software running on the mobile operating system of the phone or another mobile operating system. In this example, third-party applications 1340 can call API calls 1350 provided by the operating system 1312 to facilitate the functions described herein.

[0146] Fig.14 1400 in which instructions 1410 (e.g., software, programs, applications, applet, app, or other executable code) may be executed to cause the machine 1400 to perform any one or more of the methods discussed herein. For example, the instructions 1410 may cause the machine 1400 to perform any one or more of the methods described herein. The instructions 1410 transform a general-purpose, unprogrammed machine 1400 into a specific machine 1400 that is programmed to perform the functions described and illustrated in the manner described. The machine 1400 may operate as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine 1400 may operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 1400 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a PDA, an entertainment media system, a cellular phone, a smart phone, a mobile device, a head-mounted device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web device, a network router, a network switch, a network bridge, or any machine capable of executing instructions 1410 that specify actions to be taken by the machine 1400, either sequentially or otherwise. In addition, while only a single machine 1400 is shown, the term "machine" shall also be taken to include a collection of machines that individually or jointly execute instructions 1410 to perform any one or more of the methodologies discussed herein.

[0147] The machine 1400 may include a processor 1402, a memory 1404, and an I / O component 1406, which may be configured to communicate with each other via a bus 1444. In an example, the processor 1402 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), other processors, or any suitable combination thereof) may include, for example, a processor 1408 that executes instructions 1410 and a processor 1412. The term "processor" is intended to include a multi-core processor, which may include two or more independent processors (sometimes referred to as "cores") that may execute instructions simultaneously. Although Fig.14 Multiple processors 1402 are shown, but the machine 1400 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0148] The memory 1404 includes a main memory 1414, a static memory 1416, and a storage unit 1418, which can be accessed by the processor 1402 via a bus 1444. The main memory 1404, the static memory 1416, and the storage unit 1418 store instructions 1410 that implement any one or more of the methods or functions described herein. During execution of the instructions 1410 by the networked system 300, the instructions 1410 may also reside, in whole or in part, within the main memory 1414, within the static memory 1416, within the machine-readable medium 1420, within the storage unit 1418, within at least one of the processors 1402 (e.g., within a cache memory of the processor), or within any suitable combination thereof.

[0149] I / O components 1406 may include various components for receiving input, providing output, generating output, sending information, exchanging information, capturing measurements, etc. The specific I / O components 1406 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine may not include such a touch input device. It is to be understood that I / O components 1406 may include Fig.141406. In various examples, the I / O components 1406 may include output components 1428 and input components 1432. The output components 1428 may include visual components (e.g., displays such as plasma display panels (PDPs), light emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), acoustic components (e.g., speakers), tactile components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. The input components 1432 may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touch pads, trackballs, joysticks, motion sensors, or other pointing instruments), tactile input components (e.g., physical buttons, touch screens that provide location and / or force of touch or touch gestures, or other tactile input components), audio input components (e.g., microphones), etc.

[0150] In another example, the I / O component 1406 may include: a biometric component 1434, a motion component 1436, an environmental component 1438, or a positioning component 1440, as well as various other components. For example, the biometric component 1434 includes components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 1436 includes an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. The environment component 1438 includes, for example, an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor that detects concentrations of hazardous gases for safety or measures pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. The positioning component 1440 includes a position sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure that can obtain altitude), an orientation sensor component (e.g., a magnetometer), etc.

[0151] A variety of technologies may be used to achieve communication. I / O components 1406 also include communication components 1442 that are operable to couple networked system 300 to network 1422 or device 1424 via coupling 1430 and coupling 1426, respectively. For example, communication components 1442 may include a network interface component or another suitable device that interfaces with network 1422. In other examples, communication components 1442 may include wired communication components, wireless communication components, cellular communication components, near field communication (NFC) components, Parts (e.g. Low power consumption), Device 1424 may be another machine or any of a variety of peripheral devices (eg, a peripheral device coupled via USB).

[0152] In addition, the communication component 1442 can detect an identifier or include a component operable to detect an identifier. For example, the communication component 1442 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting the following: a one-dimensional barcode, such as a universal product code (UPC) barcode; a multi-dimensional barcode, such as a Quick Response (QR) code, Aztec code, Data Matrix, Dataglyph, MaxiCode, PDF417, Hypercode, UCC RSS-2D barcode, and other optical codes), or an acoustic detection component (e.g., a microphone for identifying an audio signal of a tag). In addition, various information can be obtained via the communication component 1442, such as a location via Internet Protocol (IP) geolocation, a location via Internet Protocol (IP), ... The location of signal triangulation, the location of an NFC beacon signal via detection that can indicate a specific location, etc.

[0153] Various memories (e.g., memory 1404, main memory 1414, static memory 1416, and / or memory of processor 1402) and / or storage unit 1418 may store one or more sets of instructions and data structures (e.g., software) implemented or used by any one or more of the methods or functions described herein. These instructions (e.g., instructions 1410) when executed by processor 1402 cause various operations to implement the disclosed examples.

[0154] Instructions 1410 may be sent or received via a network interface device (e.g., a network interface component included in communications component 1442) using a transmission medium and using any of a number of well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)) over network 1422. Similarly, instructions 1410 may be sent or received to device 1424 via coupling 1426 (e.g., a peer-to-peer coupling) using a transmission medium.

[0155] "Carrier signal" refers to any intangible medium that can store, encode or carry instructions for execution by a machine and includes digital or analog communications signals or other intangible media to facilitate communication of such instructions. Instructions may be sent or received over a network using a transmission medium via a network interface device.

[0156] "Client Device" refers to any machine that interfaces with a communications network to obtain resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a portable digital assistant (PDA), a smart phone, a tablet computer, an ultrabook, a netbook, a laptop computer, a multiprocessor system, a microprocessor-based or programmable consumer electronics product, a game console, a set-top box, or any other communications device that a user may use to access a network.

[0157] "Communications network" means one or more parts of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, A network, another type of network, or a combination of two or more such networks. For example, the network or a portion of the network may include a wireless network or a cellular network, and the coupling may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other type of cellular or wireless coupling. In this example, the coupling may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rates for GSM evolution (EDGE) technology, the third generation partnership project (3GPP) including 3G, fourth generation wireless (4G) network, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standards, other data transmission technologies defined by various standard setting organizations, other long distance protocols, or other data transmission technologies.

[0158] "Component" refers to a device, physical entity or logic with boundaries defined by function or subroutine calls, branch points, APIs or other technologies provided for partitioning or modularizing specific processing or control functions. Components can be combined with other components via their interfaces to perform machine processing. Components can be encapsulated functional hardware units designed for use with other components, and are usually part of a program that performs a specific function of related functions. Components can constitute software components (e.g., codes implemented on machine-readable media) or hardware components. "Hardware components" are tangible units that can perform certain operations and can be configured or arranged in a certain physical manner. In various examples, one or more computer systems (e.g., independent computer systems, client computer systems or server computer systems) or one or more hardware components (e.g., processors or processor groups) of a computer system can be configured by software (e.g., applications or application parts) to operate to perform certain operations as described herein. Hardware components can also be implemented mechanically, electronically or in any suitable combination thereof. For example, a hardware component may include a dedicated circuit system or logic that is permanently configured to perform certain operations. The hardware component may be a dedicated processor, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The hardware component may also include a programmable logic or circuit system that is temporarily configured to perform certain operations by software. For example, the hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or a specific component of a machine) that is uniquely customized to perform the configured function, and is no longer a general-purpose processor. It will be understood that the decision to mechanically implement the hardware component in a dedicated and permanently configured circuit system or in a temporarily configured (e.g., configured by software) circuit system can be driven by cost and time considerations. Accordingly, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity, that is, an entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate in some way or perform certain operations described herein. Considering an example where a hardware component is temporarily configured (e.g., programmed), each of the hardware components does not need to be configured or instantiated at any time. For example, where a hardware component includes a general purpose processor that is configured by software to be a special purpose processor, the general purpose processor may be configured as different special purpose processors (e.g., including different hardware components) at different times. Software configures one or more specific processors accordingly, such as to constitute a specific hardware component at one time and to constitute different hardware components at different times. Hardware components may provide information to other hardware components and receive information from other hardware components. Thus, the described hardware components may be considered to be communicatively coupled.In the case of multiple hardware components being present at the same time, communication can be achieved by signal transmission (e.g., by appropriate circuits and buses) between two or more hardware components or among two or more hardware components. In the example where multiple hardware components are configured or instantiated at different times, communication between such hardware components can be achieved, for example, by storing information in a memory structure accessible to multiple hardware components and retrieving information in the memory structure. For example, a hardware component can perform an operation, and the output of the operation is stored in a memory device coupled to it in communication. Then, other hardware components can access the memory device at a subsequent time to retrieve the stored output and process it. The hardware component can also initiate communication with an input device or an output device, and can operate on resources (e.g., the collection of information). The various operations of the example methods described herein can be performed at least in part by a temporary configuration (e.g., by software) or a permanent configuration to perform one or more processors of the related operations. Whether it is a temporary configuration or a permanent configuration, such a processor can constitute a processor-implemented component that operates to perform one or more operations or functions described herein. As used herein, "processor-implemented components" refer to hardware components implemented using one or more processors. Similarly, the method described herein can be implemented at least in part by a processor, wherein one or more specific processors are examples of hardware. For example, at least some of the operation of the method can be performed by one or more processors or the parts implemented by the processor. In addition, one or more processors can also operate to support the execution of the related operations in the "cloud computing" environment or operate as "software as a service" (SaaS). For example, at least some of the operation can be performed by a group of computers (as an example of a machine including a processor), wherein these operations can be accessed via a network (for example, the Internet) and via one or more appropriate interfaces (for example, API). The execution of some operations in the operation can be distributed between processors, not only resides in a single machine, but also deployed across multiple machines. In some examples, a processor or a part implemented by a processor can be located in a single geographical location (for example, in a home environment, an office environment or a server group). In other examples, a processor or a part implemented by a processor can be distributed across multiple geographical locations.

[0159] "Computer-readable media" refers to both machine storage media and transmission media. Therefore, these terms include both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium", "computer-readable medium" and "device-readable medium" mean the same thing and can be used interchangeably in this disclosure.

[0160] "Ephemeral messages" are messages that can be accessed for a limited duration of time. Ephemeral messages can be text, images, videos, etc. The access time of an ephemeral message can be set by the sender of the message. Alternatively, the access time can be a default setting or a setting specified by the recipient. Regardless of the setting technique, the message is ephemeral.

[0161] "Machine storage medium" refers to a single or multiple storage devices and / or media (e.g., centralized or distributed databases, and / or associated caches and servers) that store executable instructions, routines, and / or data. Thus, the term should be considered to include, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and / or device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGA, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage medium", "device storage medium", "computer storage medium" mean the same thing and may be used interchangeably in this disclosure. The terms "machine storage medium", "computer storage medium", and "device storage medium" explicitly exclude carrier waves, modulated data signals, and other such media, at least some of which are covered by the term "signal media".

[0162] A "processor" refers to any circuit or virtual circuit (a physical circuit simulated by logic executed on an actual processor) that manipulates data values ​​according to control signals (e.g., "commands," "opcodes," "machine codes," etc.) and produces corresponding output signals that are applied to operate a machine. For example, a processor may be a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), or any combination thereof. A processor may also be a multi-core processor having two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously.

[0163] "Signal medium" refers to any intangible medium that is capable of storing, encoding or carrying instructions executed by a machine, and "signal medium" includes digital or analog communication signals or other intangible media to facilitate the communication of software or data. The term "signal medium" should be deemed to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

[0164] Changes and modifications may be made to the disclosed examples without departing from the scope of the present disclosure. These and other changes or modifications are intended to be included within the scope of the present disclosure as expressed in the appended claims.

Claims

1. A method for receiving and processing navigation input executed by one or more processors in a head mounted device system, the head mounted device system comprising one or more display devices, one or more camera devices, and a vertically arranged touch panel, the method comprising: Display the first carousel of AR effect icons; receiving a first horizontal input on the touch pad; In response to the first horizontal input, rotating the first carousel of AR effect icons through a selection position; receiving a first touch input on the touch pad, the touch input being used to select a specific AR effect icon in the selection position in the first carousel; displaying, on the one or more display devices, an AR effect overlaid on the real-world scene, the AR effect corresponding to the specific AR effect icon; Receive content to capture user input; as well as In response to the content capture user input, a new video or image content item of the real world scene viewed through the one or more cameras and overlaid by the AR effect is captured.

2. The method according to claim 1, further comprising: When the first carousel of AR effect icons is being displayed, receiving a first vertical input on the touch pad; First, in response to receiving the first vertical input, bending the first carousel of AR effect icons in a direction of the first vertical input; and In response to the first vertical input exceeding a predetermined point, an additional user input interface is displayed.

3. The method according to claim 2, wherein: The additional user input interface is a second carousel including a plurality of content items, and the method further comprises: receiving a second horizontal input on the touch pad; causing the second carousel of content items to rotate in response to the second horizontal input; receiving a second touch input on the touch pad, the touch input being used to select a specific content item in a selection position in the second carousel; and In response to receiving the second touch input, additional user interface options are displayed.

4. The method according to claim 1, further comprising: receiving a vertical input on the touch panel instead of receiving the content capture user input; as well as In response to receiving the vertical input on the touch panel, display of the AR effect applied to the scene is canceled.

5. The method according to claim 3, further comprising: receiving a second vertical input on the touch pad while the second carousel including a plurality of content items is being displayed; as well as In response to the second vertical input, the first carousel of AR effect icons is displayed.

6. The method according to claim 5, wherein: The first vertical input is sliding downward, and the method further includes: A carousel icon and a downward pointer are displayed above the second carousel.

7. The method according to claim 2, further comprising: When the first carousel is being displayed, receiving a third vertical input on the touch panel, wherein the direction of the third vertical input is opposite to that of the first vertical input; as well as In response to the third vertical input, system information is displayed.

8. A head mounted device system, comprising: one or more camera devices; one or more display devices; A vertically arranged touchpad; one or more processors; as well as a memory storing instructions that, when executed by the one or more processors, configure the system to perform operations comprising: Display the first carousel of AR effect icons; receiving a first horizontal input on the touch pad; In response to a first horizontal input, rotating the first carousel of AR effect icons through a selection position; receiving a first touch input on the touch pad, the touch input being used to select a specific AR effect icon in the selection position in the first carousel; displaying, on the one or more display devices, an AR effect overlaid on the real-world scene, the AR effect corresponding to the specific AR effect icon; Receive content to capture user input; as well as In response to the content capture user input, a new video or image content item of the real world scene viewed through the one or more cameras and overlaid by the AR effect is captured.

9. The head mounted device system according to claim 8, wherein: The operations also include: When the first carousel of AR effect icons is being displayed, receiving a first vertical input on the touch pad; First, in response to receiving the first vertical input, bending the first carousel of AR effect icons in a direction of the first vertical input; and In response to the first vertical input exceeding a predetermined point, an additional user input interface is displayed.

10. The head mounted device system of claim 9, wherein: The additional user input interface is a second carousel including a plurality of content items, and the operations further include: receiving a second horizontal input on the touch pad; causing the second carousel of content items to rotate in response to the second horizontal input; receiving a second touch input on the touch pad, the touch input being used to select a specific content item in a selection position in the second carousel; and In response to receiving the second touch input, additional user interface options are displayed.

11. The head mounted device system according to claim 9, wherein: The operations also include: receiving a vertical input on the touch panel instead of receiving the content capture user input; and In response to receiving the vertical input on the touch panel, display of the AR effect applied to the scene is canceled.

12. The head mounted device system of claim 10, wherein: The operations also include: receiving a second vertical input on the touch pad while the second carousel including a plurality of content items is being displayed; and In response to the second vertical input, the first carousel of AR effect icons is displayed.

13. The head mounted device system of claim 12, wherein: The first vertical input is sliding downward, and the operation further includes: Content icons and a downward pointer are displayed above the first carousel.

14. The head mounted device system of claim 12, wherein: The operations also include: When the first carousel is being displayed, receiving a third vertical input on the touch panel, wherein a direction of the third vertical input is opposite to that of the first vertical input; and In response to the third vertical input, system information is displayed.

15. A non-transitory computer-readable storage medium comprising instructions that, when executed by a head mounted device system comprising one or more display devices, one or more camera devices, and a vertically arranged touch panel, cause the head mounted device system to perform operations comprising: Display the first carousel of AR effect icons; receiving a first horizontal input on the touch pad; In response to a first horizontal input, rotating the first carousel of AR effect icons through a selection position; receiving a first touch input on the touch pad, the touch input being used to select a specific AR effect icon in the selection position in the first carousel; displaying, on the one or more display devices, an AR effect overlaid on the real-world scene, the AR effect corresponding to the specific AR effect icon; Receive content to capture user input; as well as In response to the content capture user input, a new video or image content item of the real world scene viewed through the one or more cameras and overlaid by the AR effect is captured.

16. The computer-readable storage medium of claim 15, wherein: The operations also include: When the first carousel of AR effect icons is being displayed, receiving a first vertical input on the touch pad; First, in response to receiving the first vertical input, bending the first carousel of AR effect icons in a direction of the first vertical input; and In response to the first vertical input exceeding a predetermined point, an additional user input interface is displayed.

17. The computer-readable storage medium of claim 16, wherein: The additional user input interface is a second carousel including a plurality of content items, and the operations further include: receiving a second horizontal input on the touch pad; causing the second carousel of content items to rotate in response to the second horizontal input; receiving a second touch input on the touch pad, the touch input being used to select a specific content item in a selection position in the second carousel; and In response to receiving the second touch input, additional user interface options are displayed.

18. The computer-readable storage medium of claim 15, wherein: The operations also include: receiving a vertical input on the touch panel instead of receiving the content capture user input; and In response to receiving the vertical input on the touch panel, display of the AR effect applied to the scene is canceled.

19. The computer-readable storage medium of claim 17, wherein: The operations further include: receiving a second vertical input on the touch pad while the second carousel including a plurality of content items is being displayed; and In response to the second vertical input, the first carousel of AR effect icons is displayed.

20. The computer-readable storage medium of claim 16, wherein: The operations also include: When the first carousel is being displayed, receiving a third vertical input on the touch panel, wherein a direction of the third vertical input is opposite to that of the first vertical input; and In response to the third vertical input, system information is displayed.

Images