User control of avatar movement using augmented reality devices

Abstract

Systems and methods for performing operations on an augmented reality (AR) device are provided. The system accesses movement data including inertial measurement data and camera data through the AR device. The system determines three-dimensional (3D) movement of the AR device based on the movement data. The system presents, through the AR device, the AR object on a real-world environment being viewed using the AR device. The system modifies the AR object by the AR device in response to determining the 3D movement of the AR device.

Description

[0001] Priority Claim

[0002] This application claims the benefit of U.S. Patent Application Serial No. 17 / 983,040, filed on November 8, 2022, which is hereby incorporated by reference in its entirety. Background Art

[0003] When a user participates in some activities, some electronically enabled eyewear devices (such as so-called smart glasses) enable the user to interact with virtual content (e.g., augmented reality (AR) objects). The user wears the eyewear device and can view the real-world environment through the eyewear device while interacting with the virtual content displayed by the eyewear device. Brief Description of the Drawings

[0004] Each of the drawings in the figures only shows examples of the present disclosure and should not be regarded as limiting the scope of the present disclosure.

[0005] Figure 1 is a graphical representation of a networked environment in which the present disclosure may be deployed according to some examples.

[0006] Figure 2 is a graphical representation of a messaging system having both client-side functionality and server-side functionality according to some examples.

[0007] Figure 3 is a graphical representation of a data structure maintained in a database according to some examples.

[0008] Figure 4 is a graphical representation of a message according to some examples.

[0009] Figure 5 is a perspective view of an eyewear device according to some examples.

[0010] Figure 6 is a flowchart showing an example operation of an avatar control system according to some examples.

[0011] Figures 7 to 9 is an illustrative screen and module of an avatar control system according to some examples.

[0012] Figure 10 is a graphical representation of a machine in the form of a computer system within which a set of instructions may be executed to cause the machine to perform any one or more of the methods discussed herein.

[0013] Figure 11 is a block diagram showing a software architecture within which examples may be implemented. Detailed Description

[0014] The following description discusses illustrative examples of the present disclosure. In the following description, for purposes of illustration, numerous specific details are set forth to provide an understanding of the various examples of the disclosed subject matter. However, it will be apparent to those skilled in the art that examples of the disclosed subject matter may be practiced without these specific details. Generally, well-known instruction instances, protocols, structures, and techniques need not be shown in detail.

[0015] Typical smart glasses platforms enable users to interact with various types of virtual content. Such platforms are configured to display virtual content in the lenses of the smart glasses over the real-world environment seen through the lenses of the smart glasses. To interact with the virtual content, smart glasses typically include embedded sensors. The smart glasses can detect touch or swipe inputs based on information detected by the embedded sensors and then can update the display of the virtual content. Interacting with the embedded sensors to perform various modifications to the virtual content is not very intuitive and has a very steep learning curve. As a result, users cannot accurately perform various desired interactions with the virtual content, which detracts from the overall user experience. Additionally, due to the steep learning curve, users typically have to repeat certain actions multiple times until they learn how to use the sensors, which wastes the resources of the smart glasses.

[0016] Some smart glasses platforms use hand or gesture recognition to enable users to interact with virtual content. Specifically, the smart glasses can detect gestures in images captured by the smart glasses and can perform corresponding modifications to the virtual content. Learning how to perform such gestures correctly also involves a steep learning curve and can also be non-intuitive. Additionally, performing image processing to detect gestures involves multiple machine learning models, which consumes a significant amount of the smart glasses' hardware resources, which can be wasteful and drain the smart glasses' battery. This can also lead to the smart glasses not being used and detract from the overall enjoyment of using the smart glasses.

[0017] The disclosed examples improve the efficiency of using an electronic device by providing such an AR device that enables a user to interact with virtual content or AR objects displayed by the AR device based on the user's physical movement within a real-world environment. Specifically, the disclosed techniques access movement data including inertial measurement data and camera device data through the AR device. The disclosed techniques determine a three-dimensional (3D) movement of the AR device based on the movement data. The disclosed techniques present an AR object on the real-world environment being viewed using the AR device through the AR device. The disclosed techniques modify the AR object through the AR device in response to determining the 3D movement of the AR device. For example, a user wearing the AR device can jump, bend, run, or turn, and such movements are detected by the AR device. Based on detecting such movements, the AR device can animate an avatar or other virtual object to mimic the movements, such as by animating the avatar to jump, run, bend, and / or turn about its own axis.

[0018] In this way, the disclosed examples increase the efficiency of the electronic device by reducing the amount of input and information pages required to complete a task and by reducing the running of complex image processing algorithms on the AR device. The disclosed examples also increase the efficiency, attractiveness, and usability of an electronic AR device (such as a glasses device). Although the disclosed examples are provided in the context of an electronic glasses device, similar examples can be applied to any other type of AR wearable device, such as an AR hat, an AR watch, an AR belt, an AR ring, an AR bracelet, AR earrings, and / or an AR headset.

[0019] Networked computing environment

[0020] Figure 1 is a block diagram illustrating an example messaging system 100 for exchanging data (e.g., messages and associated content) over a network. The messaging system 100 includes multiple instances of client devices 102, each of the multiple instances hosting several applications including a messaging client 104 and other external applications 109 (e.g., third-party applications). Each messaging client 104 is communicatively coupled via a network 112 (e.g., the Internet) to other instances of the messaging client 104 (e.g., hosted on corresponding other client devices 102), a messaging server system 108, and an external app server 110. The messaging client 104 can also communicate with locally hosted third-party applications 109 using an application programming interface (API).

[0021] In some examples, messaging system 100 includes glasses device 119, which hosts avatar control system 107 among other applications. Although only one instance of glasses device 119 is shown, any number of glasses devices 119 may be included in messaging system 100.

[0022] Glasses device 119 may represent any type of AR device worn by a user, such as AR glasses, an AR hat, an AR watch, an AR belt, an AR ring, an AR bracelet, AR earrings, and / or AR headphones. Glasses device 119 is communicatively coupled to client device 102 via network 112, which may include a direct connection via a dedicated short-range communication path, such as Bluetooth TM or WiFi.

[0023] Avatar control system 107 enables a user to interact with virtual content or AR objects displayed by glasses device 119 based on movement data or 3D movement information determined by glasses device 119. Glasses device 119 may access or collect movement data, such as inertial measurement unit data and image data obtained from multiple cameras of glasses device 119. Glasses device 119 may determine the 3D movement of glasses device 119 based on the movement data. In some cases, glasses device 119 may apply one or more machine learning models to the 3D movement to estimate the corresponding movement to be applied to an AR object or avatar. Then, glasses device 119 may perform corresponding modifications to one or more displayed AR objects, such as animating an avatar to mimic the movement performed by the user and determined by glasses device 119.

[0024] In some examples, avatar control system 107 accesses movement data including inertial measurement data and camera device data via an AR device (e.g., glasses device 119). Avatar control system 107 determines the 3D movement of the AR device based on the movement data and presents an AR object via the AR device on a real-world environment being viewed using the AR device. Avatar control system 107 modifies the AR object via the AR device in response to determining the 3D movement of the AR device.

[0025] In some examples, the movement data includes gyroscope data. In some examples, the AR device includes an AR glasses device. Avatar control system 107 determines the orientation, rotation, position, and velocity of the AR device in 3D space as the 3D movement of the AR device. In some examples, avatar control system 107 updates one or more body parts of the avatar based on the 3D movement of the AR device.

[0026] In some examples, the avatar control system 107 synchronizes inertial measurement data with image data that includes camera device data obtained from multiple camera devices. The avatar control system 107 corrects the position drift of the inertial measurement data in response to synchronizing the inertial measurement data with the image data. In some examples, the avatar control system 107 identifies an anchor object in the image data and periodically determines the movement of the position of the anchor object in the image data. The avatar control system 107 synchronizes the movement detected by the inertial measurement data with the movement of the position of the anchor object in the image data.

[0027] In some examples, the avatar control system 107 determines that the 3D movement of the AR device corresponds to movement in a specific direction. The avatar control system 107 moves the AR object in the specific direction in response to determining that the 3D movement of the AR device corresponds to movement in a specific direction. In some examples, the specific direction includes at least one of moving forward, backward, left, or right.

[0028] In some examples, the avatar control system 107 determines that the 3D movement of the AR device corresponds to movement at a specific speed. In such a case, the avatar control system 107 moves the AR object at the specific speed in the direction in response to determining that the 3D movement of the AR device corresponds to movement at a specific speed. In some examples, the avatar control system 107 determines that the movement at the specific speed exceeds a threshold speed. In response to determining that the movement at the specific speed exceeds the threshold speed, the avatar control system 107 changes the movement mode of the AR object from a first movement mode to a second movement mode. In some examples, the first movement mode includes walking, and the second movement mode includes running, sprinting, or flying.

[0029] In some examples, the avatar control system 107 determines that the 3D movement of the AR device corresponds to movement at a specific vertical height. The avatar control system 107 moves the AR object at the specific vertical height in response to determining that the 3D movement of the AR device corresponds to movement at a specific vertical height. In some examples, the avatar control system 107 determines that the movement at the specific vertical height exceeds a threshold height and a threshold rate (the specific vertical height is reached at the threshold rate). In response to determining that the movement at the specific vertical height exceeds the threshold height and the threshold rate (the specific vertical height is reached at the threshold rate), the avatar control system 107 selects a movement mode for the AR object. In some cases, the movement mode includes jumping or bending down.

[0030] In some examples, the avatar control system 107 determines that the movement at the specific vertical height exceeds the threshold height and does not exceed the threshold rate. In response to determining that the movement at the specific vertical height exceeds the threshold height and does not exceed the threshold rate, the avatar control system 107 prevents the selected movement mode from being applied to the AR object.

[0031] In some examples, the avatar control system 107 determines that the 3D movement of the AR device corresponds to a 360-degree turn along a specified curve. The avatar control system 107 rotates the AR object along an axis in response to determining that the 3D movement of the AR device corresponds to a 360-degree turn along a specified curve.

[0032] In some examples, the AR device is configured to apply one or more machine learning models to at least one of the 3D movement or movement data to estimate a modification to be applied to the AR object. In some examples, the AR object includes an avatar representing a user. In such a case, the avatar control system 107 determines an exercise goal associated with a fitness activity and animates the avatar based on the determined 3D movement of the AR device to represent the achievement of the exercise goal.

[0033] The messaging client 104 can communicate with and exchange data with other messaging clients 104, the glasses device 119, and the messaging server system 108 via the network 112. The data exchanged between the messaging clients 104 and between the messaging client 104 and the messaging server system 108 includes functionality (e.g., commands for activating functionality) and payload data (e.g., text, audio, video, or other multimedia data).

[0034] The messaging server system 108 provides server-side functionality to a specific messaging client 104 via the network 112. While certain functions of the messaging system 100 are described herein as being performed by the messaging client 104 or by the messaging server system 108, whether certain functions are located within the messaging client 104 or within the messaging server system 108 can be a design choice. For example, it may be technically preferable to initially deploy certain technologies and functionality within the messaging server system 108, but later migrate the technology and functionality to the messaging client 104 when the client device 102 has sufficient processing power.

[0035] The messaging server system 108 supports various services and operations provided to the messaging client 104. Such operations include sending data to the messaging client 104, receiving data from the messaging client 104, and processing data generated by the messaging client 104. As an example, the data can include message content, client device information, geographical location information, media enhancements and overlays, message content persistence conditions, social network information, and live event information. The data exchange within the messaging system 100 is initiated and controlled through functionality available via the user interface (UI) of the messaging client 104.

[0036] Now turning specifically to the messaging server system 108, an application programming interface (API) server 116 is coupled to the application server 114 and provides a programming interface to the application server 114. The application server 114 is communicatively coupled to a database server 120, which facilitates access to a database 126 that stores data associated with messages processed by the application server 114. Similarly, a web server 128 is coupled to the application server 114 and provides a web-based interface to the application server 114. To this end, the web server 128 processes incoming network requests via the Hypertext Transfer Protocol (HTTP) and several other relevant protocols.

[0037] The API server 116 receives and transmits message data (e.g., commands and message payloads) between the client device 102 and the application server 114. Specifically, the API server 116 provides a set of interfaces (e.g., routines and protocols) that can be invoked or queried by the messaging client 104 to facilitate activation of the functions of the application server 114. The API server 116 exposes various functions supported by the application server 114, including: account registration; login functionality; sending a message from a particular messaging client 104 to another messaging client 104 via the application server 114; sending a media file (e.g., an image or video) from the messaging client 104 to the messaging server 118 for possible access by another messaging client 104; setting up a collection of media data (e.g., a story); retrieving a list of friends of the user of the client device 102; retrieving such collections; retrieving messages and content; adding and deleting entities (e.g., friends) in an entity graph (e.g., a social graph); locating friends within the social graph; and opening application events (e.g., related to the messaging client 104).

[0038] The application server 114 hosts multiple server applications and subsystems, including, for example, a messaging server 118, an image processing server 122, and a social network server 124. The messaging server 118 implements multiple message processing techniques and functions, particularly those related to the aggregation and other processing of content (e.g., text and multimedia content) included in messages received from multiple instances of the messaging client 104. As will be described in more detail, text and media content from multiple sources can be aggregated into collections of content (e.g., referred to as stories or galleries). These collections are then made available to the messaging client 104. Given the hardware requirements for other processor-intensive and memory-intensive data processing, such processing can also be performed on the server side by the messaging server 118.

[0039] The application server 114 also includes an image processing server 122, which is dedicated to performing various image processing operations, typically on images or videos within the payload of messages sent from or received at the messaging server 118.

[0040] The image processing server 122 is used to implement the scanning function of the enhancement system 208. The scanning function includes: when an image is captured by the client device 102, activating and providing one or more AR experiences on the client device 102. Specifically, the messaging client 104 on the client device 102 can be used to activate the imaging device. The imaging device displays one or more real-time images or videos to the user and one or more icons or identifiers of one or more AR experiences. The user can select a given identifier among the identifiers to initiate the corresponding augmented reality experience. Initiating an AR experience includes: obtaining one or more augmented reality items associated with the AR experience and superimposing the augmented reality items on the image or video being presented.

[0041] The social network server 124 supports various social networking functions and services and makes these functions and services available to the messaging server 118. To this end, the social network server 124 maintains and accesses the entity graph 308 in the database 126 (as Figure 3 shown). Examples of the functions and services supported by the social network server 124 include: identifying other users in the messaging system 100 who have a relationship with a specific user or are "following" the specific user, and also identifying the interests of the specific user and other entities.

[0042] Return to the messaging client 104, and the features and functions of an external resource (e.g., a third-party application 109 or a mini-program) are made available to the user via the interface of the messaging client 104. The messaging client 104 receives a user selection of an option to initiate or access the features of an external resource (e.g., a third-party resource) (e.g., an external app 109). The external resource can be a third-party application (external app 109) (e.g., a "native app") installed on the client device 102 or a scaled-down version of a third-party application (e.g., a "mini-program") hosted on the client device 102 or remote from the client device 102 (e.g., on a third-party server 110). The scaled-down version of the third-party application includes a subset of the features and functions of the third-party application (e.g., the full-scale, native version of a third-party stand-alone application) and is implemented using a markup language document. In one example, the scaled-down version of the third-party application (e.g., a "mini-program") is a web-based markup language version of the third-party application and is embedded in the messaging client 104. In addition to using a markup language document (e.g., a.*ml file), the mini-program can also incorporate a scripting language (e.g., a.*js file or a.json file) and a style sheet (e.g., a.*ss file).

[0043] In response to receiving a user selection of an option to initiate or access the features of an external resource (external app 109), the messaging client 104 determines whether the selected external resource is a web-based external resource or a locally installed external application. In some cases, the external application 109 can be initiated independently of and separately from the messaging client 104, for example, by selecting an icon corresponding to the externally installed external application 109 on the home screen of the client device 102. The scaled-down version of such an external application can be initiated or accessed via the messaging client 104, and in some examples, no part or a limited part of the scaled-down external application can be accessed outside the messaging client 104. The scaled-down external application can be initiated by receiving, from an external app server 110 via the messaging client 104, a markup language document associated with the scaled-down external application and processing such a document.

[0044] In response to determining that the external resource is a locally installed external application 109, the messaging client 104 instructs the client device 102 to launch the external application 109 by executing locally stored code corresponding to the external application 109. In response to determining that the external resource is a web-based resource, the messaging client 104 communicates with the external app server 110 to obtain a markup language document corresponding to the selected resource. The messaging client 104 then processes the obtained markup language document to render the web-based external resource within the user interface of the messaging client 104.

[0045] The messaging client 104 may notify a user of the client device 102 or other users associated with such a user (e.g., "friends") of activities occurring in one or more external resources. For example, the messaging client 104 may provide notifications to participants in a conversation (e.g., a chat session) in the messaging client 104 regarding current or recent use of an external resource by one or more members of a user group. One or more users may be invited to join an active external resource or to launch an external resource that was recently used but is currently inactive (within a group of friends). The external resource may provide the ability for participants in the conversation (each using a respective messaging client 104) to share items, conditions, statuses, or locations within the external resource with one or more members of the user group entering the chat session. The shared item may be an interactive chat card that chat members can use to interact, such as to launch the corresponding external resource, view specific information within the external resource, or take the chat members to a specific location or status within the external resource. Response messages may be sent to users on the messaging client 104 within a given external resource. The external resource may selectively include different media items in the response based on the current context of the external resource.

[0046] The messaging client 104 may present a list of available external resources (e.g., third-party or external applications 109 or applets) to a user to launch or access a given external resource. The list may be presented in a contextually relevant menu. For example, icons representing different external applications within the external application 109 (or applet) may vary based on how the user launches the menu (e.g., from a conversation interface or from a non-conversation interface).

[0047] System Architecture

[0048] Figure 2FIG. is a block diagram showing further details of a messaging system 100 according to some examples. Specifically, the messaging system 100 is shown to include a messaging client 104 and an application server 114. The messaging system 100 includes a number of subsystems that are supported on the client side by the messaging client 104 and on the server side by the application server 114. These subsystems include, for example, an ephemeral timer system 202, a collection management system 204, an enhancement system 208, a map system 210, a game system 212, and an external resources system 220.

[0049] The ephemeral timer system 202 is responsible for implementing temporary or time-limited access to content by the messaging client 104 and the messaging server 118. The ephemeral timer system 202 includes a number of timers that selectively enable access (e.g., for presentation and display) to messages and associated content via the messaging client 104 based on the duration and display parameters associated with a message or a collection of messages (e.g., a story). Additional details regarding the operation of the ephemeral timer system 202 are provided below.

[0050] The collection management system 204 is responsible for managing groups and collections of media (e.g., collections of text, image, video, and audio data). Collections of content (e.g., messages including images, video, text, and audio) can be organized into an "event library" or an "event story". Such collections can be made available for a specified period of time, e.g., for the duration of an event related to the content. For example, content related to a concert can be made available as a "story" for the duration of that concert. The collection management system 204 can also be responsible for publishing an icon providing a notification of the existence of a particular collection to the user interface of the messaging client 104.

[0051] In addition, the collection management system 204 includes a curation interface 206 that enables a collection manager to manage and curate a particular collection of content. For example, the curation interface 206 enables an event organizer to curate a collection of content related to a particular event (e.g., to remove inappropriate content or redundant messages). In addition, the collection management system 204 employs machine vision (or image recognition technology) and content rules to automatically curate content collections. In some examples, compensation can be paid to a user for including user-generated content in a collection. In such cases, the collection management system 204 operates to automatically pay such users for the use of their content.

[0052] The enhancement system 208 provides various functions that enable a user to enhance (e.g., annotate or otherwise modify or edit) media content associated with a message. For example, the enhancement system 208 provides functions related to generating and publishing media overlays for messages processed by the messaging system 100. The enhancement system 208 operably supplies a media overlay or enhancement (e.g., an image filter) to the messaging client 104 based on the geolocation of the client device 102. In another example, the enhancement system 208 operably supplies a media overlay to the messaging client 104 based on other information such as the social network information of a user of the client device 102. The media overlay may include audio and visual content as well as visual effects. Examples of audio and visual content include pictures, text, logos, animations, and sound effects. Examples of visual effects include color overlays. The audio and visual content or visual effects may be applied to a media content item (e.g., a photo) at the client device 102. For example, the media overlay may include text, graphic elements, or images that may be superimposed over a photo taken by the client device 102. In another example, the media overlay includes a location identifier overlay (e.g., Venice Beach), a live event name, or a merchant name overlay (e.g., Beach Café). In another example, the enhancement system 208 uses the geolocation of the client device 102 to identify a media overlay that includes the name of a merchant at the geolocation of the client device 102. The media overlay may include other markers associated with the merchant. The media overlay may be stored in the database 126 and accessed via the database server 120.

[0053] In some examples, the enhancement system 208 provides a user-based publishing platform that enables a user to select a geolocation on a map and upload content associated with the selected geolocation. The user may also specify the circumstances under which a particular media overlay should be provided to other users. The enhancement system 208 generates a media overlay that includes the uploaded content and associates the uploaded content with the selected geolocation.

[0054] In other examples, the augmentation system 208 provides a merchant-based publishing platform that enables a merchant to select a specific media overlay associated with a geolocation via a bidding process. For example, the augmentation system 208 associates the media overlay of the highest-bidding merchant with the corresponding geolocation for a predefined amount of time. The augmentation system 208 communicates with the image processing server 122 to obtain an augmented reality experience and presents an identifier of such an experience in one or more user interfaces (e.g., as an icon on a live image or video, or as a thumbnail or icon in an interface dedicated to the presentation of the augmented reality experience). Once an augmented reality experience is selected, one or more images, videos, or augmented reality graphic elements are retrieved and presented as an overlay on top of the image or video captured by the client device 102. In some cases, the camera device is switched to a front-facing view (e.g., the front camera device of the client device 102 is activated in response to the activation of a specific augmented reality experience), and the image from the front camera device of the client device 102 rather than the rear camera device of the client device 102 starts to be displayed on the client device 102. One or more images, videos, or augmented reality graphic elements are retrieved and presented as an overlay on top of the image captured and displayed by the front camera device of the client device 102.

[0055] The map system 210 provides various geolocation functions and supports the presentation of map-based media content and messages by the messaging client 104. For example, the map system 210 enables the display on a map of user icons or avatars (e.g., stored in the profile data 316) to indicate the current or past locations of a user's "friends" and the media content (e.g., a collection of messages including photos and videos) generated by such friends in the context of the map. For example, on the map interface of the messaging client 104, a message posted by a user to the messaging system 100 from a specific geolocation can be displayed to the "friends" of the specific user in the context of that specific geolocation on the map. A user can also share his or her location and status information with other users of the messaging system 100 (e.g., using an appropriate status avatar) via the messaging client 104, where such location and status information is similarly displayed to the selected users in the context of the map interface of the messaging client 104.

[0056] The game system 212 provides various game functions in the context of the messaging client 104. The messaging client 104 provides a game interface that provides a list of available games (e.g., web-based games or web-based applications) that can be launched by a user in the context of the messaging client 104 and played with other users of the messaging system 100. The messaging system 100 also enables a particular user to invite such other users to participate in playing a particular game by sending an invitation from the messaging client 104 to the other users. The messaging client 104 also supports both voice messaging and text messaging (e.g., chat) in the context of playing a game, provides a leaderboard for the game, and also supports providing in-game rewards (e.g., game currency and items).

[0057] The external resource system 220 provides the messaging client 104 with an interface to communicate with an external app server 110 to launch or access external resources. Each external resource (app) server 110 hosts a small-scale version of an application (e.g., a game, utility, payment, or ride-sharing application outside of the messaging client 104) based on a markup language (e.g., HTML5) or an external application. The messaging client 104 can launch a web-based resource (e.g., an application) by accessing an HTML5 file from an external resource (app) server 110 associated with the web-based resource. In some examples, a software development kit (SDK) provided by the messaging server 118 is used to program the applications hosted by the external resource server 110 in JavaScript. The SDK includes application programming interfaces (APIs) whose functions can be invoked or activated by web-based applications. In some examples, the messaging server 118 includes a JavaScript library that provides access to certain user data of the messaging client 104 to a given third-party resource. HTML5 is used as an example technology for programming games, but applications and resources programmed based on other technologies can be used.

[0058] To integrate the functions of the SDK into a web-based resource, the SDK is downloaded by the external resource (app) server 110 from the messaging server 118 or otherwise received by the external resource (app) server 110. Once downloaded or received, the SDK is included as part of the application code of the web-based external resource. Then, the code of the web-based resource can invoke or activate certain functions of the SDK to integrate features of the messaging client 104 into the web-based resource.

[0059] The SDK stored on the messaging server 118 effectively provides a bridge between external resources (e.g., third-party or external applications 109 or applets) and the messaging client 104. This provides users with a seamless experience of communicating with other users on the messaging client 104 while still retaining the look and feel of the messaging client 104. To bridge the communication between the external resources and the messaging client 104, in some examples, the SDK facilitates the communication between the external resource server 110 and the messaging client 104. In some examples, the WebView JavaScript Bridge running on the client device 102 establishes two one-way communication channels between the external resources and the messaging client 104. Messages are sent asynchronously between the external resources and the messaging client 104 via these communication channels. Each SDK function call is sent as a message and a callback. Each SDK function is implemented by constructing a unique callback identifier and sending a message with that callback identifier.

[0060] By using the SDK, not all information from the messaging client 104 is shared with the external resource server 110. The SDK restricts which information is shared based on the needs of the external resources. In some examples, each external resource server 110 provides an HTML5 file corresponding to the web-based external resource to the messaging server 118. The messaging server 118 can add a visual representation (e.g., a block diagram or other graphic) of the web-based external resource in the messaging client 104. Once the user selects the visual representation through the GUI of the messaging client 104 or instructs the messaging client 104 to access the features of the web-based external resource, the messaging client 104 obtains the HTML5 file and instantiates the resources required to access the features of the web-based external resource.

[0061] The messaging client 104 presents a graphical user interface of an external resource (e.g., a login page or a splash screen). During, before, or after presenting the login page or the splash screen, the messaging client 104 determines whether the launched external resource has been previously authorized to access the user data of the messaging client 104. In response to determining that the launched external resource has been previously authorized to access the user data of the messaging client 104, the messaging client 104 presents another graphical user interface of the external resource that includes the functions and features of the external resource. In response to determining that the launched external resource has not been previously authorized to access the user data of the messaging client 104, after a threshold period of time (e.g., 3 seconds) of displaying the login page or the splash screen of the external resource, the messaging client 104 slides up a menu for authorizing the external resource to access the user data (e.g., animating the menu to emerge from the bottom of the screen to the middle or other part of the screen). The menu identifies the type of user data for which the external resource will be authorized to use. In response to receiving a user selection of an accept option, the messaging client 104 adds the external resource to the list of authorized external resources and enables the external resource to access the user data from the messaging client 104. In some examples, the messaging client 104 authorizes the external resource to access the user data according to the OAuth 2 framework.

[0062] The messaging client 104 controls the type of user data shared with an external resource based on the type of the authorized external resource. For example, an external resource that includes a full-scale external application (e.g., a third-party or external application 109) is provided access to a first type of user data (e.g., only two-dimensional avatars of users with or without different avatar characteristics). As another example, an external resource that includes a scaled-down version of an external application (e.g., a web-based version of a third-party application) is provided access to a second type of user data (e.g., payment information, two-dimensional avatars of users, three-dimensional avatars of users, and avatars with various avatar characteristics). Avatar characteristics include different ways of customizing the appearance and feel of an avatar, such as different poses, facial features, clothing, etc.

[0063] Data architecture

[0064] Figure 3 is a schematic diagram showing a data structure 300 that can be stored in the database 126 of the messaging server system 108 according to some examples. Although the contents of the database 126 are shown as including several tables, it should be understood that the data can be stored in other types of data structures (e.g., as an object-oriented database).

[0065] The database 126 includes message data stored in the message table 302. For any particular message, the message data includes at least message sender data, message recipient (or receiver) data, and a payload. Details are provided below with reference to Figure 4 describe additional details regarding information that may be included in a message and included in the message data stored in the message table 302.

[0066] The entity table 306 stores entity data and is (e.g., referentially) linked to the entity graph 308 and profile data 316. Entities for which records are maintained within the entity table 306 can include individuals, corporate entities, organizations, objects, locations, events, etc. Any entity for which the messaging server system 108 stores data about it can be an identified entity, regardless of the entity type. Each entity is provided with a unique identifier as well as an entity type identifier (not shown).

[0067] The entity graph 308 stores information about the relationships and associations between entities. Such relationships may be, for example, merely social, professional (e.g., working within a common company or organization), interest-based, or activity-based.

[0068] The profile data 316 stores various types of profile data about a particular entity. Based on privacy settings specified by the particular entity, the profile data 316 can be selectively used and presented to other users of the messaging system 100. In the case where the entity is a person, the profile data 316 includes, for example, a username, a phone number, an address, settings (e.g., notification and privacy settings), and a user-selected avatar representation (or a set of such avatar representations). Then, a particular user can selectively include one or more of these avatar representations in the content of messages transmitted via the messaging system 100 and on a map interface displayed by the messaging client 104 to other users. The set of avatar representations can include "status avatars" that present graphical representations of statuses or activities that the user can choose to transmit at a particular time.

[0069] In the case where the entity is a group, in addition to the group name, members, and various settings of the related group (e.g., notifications), the profile data 316 of the group can similarly include one or more avatar representations associated with the group.

[0070] The database 126 also stores enhancement data, such as overlays or filters, in the enhancement table 310. The enhancement data is associated with videos (whose data is stored in the video table 304) and images (whose data is stored in the image table 312) and is applied to the videos and images.

[0071] In one example, the filter is an overlay that is displayed as an overlay on an image or video during presentation to the receiving user. The filter can be of various types, including a filter selected by the user from a set of filters presented to the sending user by the messaging client 104 when the sending user is composing a message. Other types of filters include geolocation filters (also known as geo-filters), which can be presented to the sending user based on geolocation. For example, specific to a nearby or special location geolocation filters can be presented within the user interface by the messaging client 104 based on geolocation information determined by the global positioning system (GPS) unit of the client device 102.

[0072] Another type of filter is a data filter, which can be selectively presented to the sending user by the messaging client 104 based on other input or information collected by the client device 102 during the message creation process. Examples of data filters include the current temperature at a specific location, the current speed at which the sending user is traveling, the battery life of the client device 102, or the current time.

[0073] Other enhanced data that can be stored within the image table 312 includes augmented reality content items (e.g., corresponding to an applied augmented reality experience). Augmented reality content items or augmented reality items can be real-time special effects and sounds that can be added to an image or video.

[0074] As described above, enhanced data includes augmented reality content items, overlays, image transformations, AR images, and similar items that refer to modifications that can be applied to image data (e.g., video or images). This includes real-time modifications that modify an image as it is captured using a device sensor (e.g., one or more camera devices) of client device 102 and then displayed on the screen of client device 102 in the modified case. This also includes modifications to stored content (e.g., video clips in a library that can be modified). For example, in client device 102 that has access to multiple augmented reality content items, a user can use a single video clip with multiple augmented reality content items to see how different augmented reality content items would modify the stored clip. For example, by selecting different augmented reality content items for the same content, multiple augmented reality content items applying different pseudo-random movement models can be applied to that same content. Similarly, real-time video capture can be used with the shown modifications to show how the video image currently captured by the sensor of client device 102 would modify the captured data. Such data can simply be displayed on the screen without being stored in memory, or the content captured by the device sensor can be recorded and stored in memory with or without modifications (or both). In some systems, a preview feature can show how different augmented reality content items would look within different windows on the display simultaneously. For example, this can enable viewing multiple windows with different pseudo-random animations on the display at the same time.

[0075] Accordingly, systems that use data of augmented reality content items and various other such transformation systems that use such data to modify content can involve: detection of objects (e.g., faces, hands, bodies, cats, dogs, surfaces, objects, etc.); tracking of such objects as they leave, enter, and move around within the field of view of a video frame; and modification or transformation of such objects as they are being tracked. In various examples, different methods can be used to implement such transformations. Some examples can involve: generating a three-dimensional mesh model of one or more objects and using transformations and animated textures of the model within the video to implement the transformation. In other examples, tracking of points on an object can be used to place an image or texture (which can be two-dimensional or three-dimensional) at the tracked locations. In further examples, neural network analysis of video frames can be used to place an image, model, or texture within content (e.g., an image or frame of a video). Accordingly, augmented reality content items refer both to the images, models, and textures used to create transformations within content and to the additional modeling and analysis information needed to implement such transformations using object detection, tracking, and placement.

[0076] Real-time video processing can be performed using any kind of video data (e.g., video stream, video file, etc.) stored in the memory of any kind of computerized system. For example, a user can load a video file and save it in the memory of a device, or can use the sensors of the device to generate a video stream. Additionally, computer animation models can be used to process any object, such as a human face and various parts of the human body, an animal, or a non-biological object (e.g., a chair, a car, or other object).

[0077] In some examples, when a specific modification is selected along with the content to be transformed, the element to be transformed is identified by a computing device, and then, if the element to be transformed exists in a frame of the video, the element to be transformed is detected and tracked. The elements of the object are modified according to the modification request, thereby transforming the frame of the video stream. For different kinds of transformations, the transformation of the frame of the video stream can be performed by different methods. For example, for a frame transformation that mainly refers to changing the form of the elements of an object, the feature points of each element of the object are calculated (e.g., using an active shape model (ASM) or other known methods). Then, a grid based on the feature points is generated for each of at least one element of the object. This grid is used for the subsequent stage of tracking the elements of the object in the video stream. During the tracking process, the mentioned grid of each element is aligned with the position of each element. Then, additional points are generated on the grid. A first set of first points is generated for each element based on the modification request, and a set of second points is generated for each element based on this set of first points and the modification request. Then, the frame of the video stream can be transformed by modifying the elements of the object based on this set of first points, this set of second points, and the grid. In such a method, the background of the modified object can also be changed or distorted by tracking and modifying the background.

[0078] In some examples, a transformation that uses the elements of an object to change some regions of the object can be performed by calculating the feature points of each element of the object and generating a grid based on the calculated feature points. Points are generated on the grid, and then various regions based on these points are generated. Then, the elements of the object are tracked by aligning the regions of each element with the position of each of at least one element, and the attributes of the regions can be modified based on the modification request, thereby transforming the frame of the video stream. Depending on the specific modification request, the attributes of the mentioned regions can be transformed in different ways. Such modifications can involve: changing the color of the region; removing at least some partial regions from the frame of the video stream; including one or more new objects in the region based on the modification request; and modifying or distorting the region or the elements of the object. In various examples, any combination of such modifications or other similar modifications can be used. For some models to be animated, some feature points can be selected as control points for determining the entire state space of the options for model animation.

[0079] In some examples of computer animation models that use face detection to transform image data, a specific face detection algorithm (e.g., Viola-Jones) is used to detect faces in an image. Then, an Active Shape Model (ASM) algorithm is applied to the face region of the image to detect facial feature reference points.

[0080] Other methods and algorithms suitable for face detection can be used. For example, in some examples, landmarks are used to locate features, where a landmark represents a distinguishable point that exists in most images under consideration. For example, for face landmarks, the location of the left eye pupil can be used. If the initial landmark is not identifiable (e.g., if a person has an eye patch), secondary landmarks can be used. Such a landmark identification process can be used for any such object. In some examples, a set of landmarks forms a shape. The shape can be represented as a vector using the coordinates of the points in the shape. A similarity transformation (which allows translation, scaling, and rotation) is used to align one shape with another, which minimizes the average Euclidean distance between the shape points. The average shape is the average of the aligned training shapes.

[0081] In some examples, a search for landmarks begins from an average shape aligned with the position and size of the face determined by a full-face detector. Then, such a search repeats the steps of: adjusting the positioning of the shape points through template matching of the image texture around each point, thus suggesting a tentative shape; and then conforming the tentative shape to a global shape model until convergence occurs. In some systems, individual template matches are unreliable, and the shape model pools the results of weak template matches to form a stronger overall classifier. The entire search is repeated at each level of an image pyramid from a coarse resolution to a fine resolution.

[0082] The transformation system can capture an image or video stream on a client device (e.g., client device 102) and perform complex image manipulations locally on the client device 102 while maintaining an appropriate user experience, computation time, and power consumption. Complex image manipulations can include: size and shape changes, mood transitions (e.g., changing a face from a frown to a smile), state transitions (e.g., aging a subject, reducing apparent age, changing gender), style transitions, application of graphical elements, and any other suitable image or video manipulations implemented by a convolutional neural network that has been configured to execute efficiently on the client device 102.

[0083] In some examples, a computer animation model for transforming image data can be used by a system in which a user can capture an image or video stream (e.g., a selfie) using a client device 102 having a neural network operating as part of a messaging client 104 operating on the client device 102. A transformation system operating within the messaging client 104 determines the presence of a face within the image or video stream and provides a modification icon associated with the computer animation model to transform the image data, or the computer animation model can be presented in association with the interfaces described herein. The modification icon includes changes that can be the basis for modifying the user's face within the image or video stream as part of a modification operation. Once the modification icon is selected, the transformation system initiates a process of transforming the user's image to reflect the selected modification icon (e.g., generating a smiling face on the user). Once an image or video stream is captured and a specified modification is selected, the modified image or video stream can be presented in a graphical user interface displayed on the client device 102. The transformation system can implement a complex convolutional neural network on a portion of the image or video stream to generate and apply the selected modification. That is, the user can capture an image or video stream, and once the modification icon is selected, the result of the modification can be presented to the user in real time or near real time. Additionally, when a video stream is being captured and the selected modification icon remains toggled, the modification can be persistent. A machine-learned neural network can be used to implement such modifications.

[0084] A graphical user interface presenting the modifications performed by the transformation system can supply additional interaction options to the user. Such options can be based on the interface used to initiate content capture and selection for a particular computer animation model (e.g., initiated from a content creator user interface). In various examples, the modification can be persistent after an initial selection of the modification icon. The user can toggle the modification on or off by tapping or otherwise selecting the face modified by the transformation system and store it for later viewing or browsing to other areas of the imaging application. In the case where multiple faces are modified by the transformation system, the user can globally toggle the modification on or off by tapping or selecting a single face modified and displayed within the graphical user interface. In some examples, a single face within a group of multiple faces can be modified individually, or such modifications can be toggled individually by tapping or selecting a single face or a series of single faces displayed within the graphical user interface.

[0085] The story table 314 stores data regarding a collection of messages, as well as associated image, video, or audio data, which are compiled into a collection (e.g., a story or a library). The creation of a particular collection can be initiated by a particular user (e.g., each user whose record is maintained in the entity table 306). A user can create a "personal story" in the form of a collection of content that has been created and sent / broadcast by that user. To this end, the user interface of the messaging client 104 can include user-selectable icons to enable the sending user to add specific content to his or her personal story.

[0086] Collections can also constitute "live stories" as collections of content from multiple users, which are created manually, automatically, or using a combination of manual and automatic techniques. For example, a "live story" can constitute a curated stream of user-submitted content from various locations and events. Options can be presented, for example, via the user interface of the messaging client 104 to users of its client devices whose location services are enabled and who are at a public location event at a particular time, to contribute content to a particular live story. A live story can be identified to a user by the messaging client 104 based on his or her location. The end result is a "live story" told from a community perspective.

[0087] Another type of content collection is called a "location story", which enables users whose client devices 102 are located within a particular geolocation (e.g., on a college or university campus) to contribute to a particular collection. In some examples, contributing to a location story may require a second degree of authentication to verify that the end user belongs to a particular organization or other entity (e.g., is a student on a university campus).

[0088] As mentioned above, the video table 304 stores video data, which, in one example, is associated with messages whose records are maintained in the message table 302. Similarly, the image table 312 stores image data associated with messages whose message data is stored in the entity table 306. The entity table 306 can associate various enhancements from the enhancement table 310 with the various images and videos stored in the image table 312 and the video table 304.

[0089] Data Communication Architecture

[0090] Figure 4FIG. is a schematic diagram showing the structure of a message 400 according to some examples, which is generated by a messaging client 104 for transmission to another messaging client 104 or a messaging server 118. The content of a particular message 400 is used to populate a message table 302 stored in a database 126, which can be accessed by the messaging server 118. Similarly, the content of the message 400 is stored in memory as "in-transit" or "in-flight" data of the client device 102 or the application server 114. The message 400 is shown to include the following example components:

[0091] · Message identifier 402: A unique identifier that identifies the message 400;

[0092] · Message text payload 404: Text to be generated by a user via a user interface of the client device 102 and included in the message 400;

[0093] · Message image payload 406: Image data captured by a camera device component of the client device 102 or retrieved from a memory component of the client device 102 and included in the message 400. The image data for a sent or received message 400 can be stored in an image table 312;

[0094] · Message video payload 408: Video data captured by a camera device component or retrieved from a memory component of the client device 102 and included in the message 400. The video data for a sent or received message 400 can be stored in a video table 304;

[0095] · Message audio payload 410: Audio data captured by a microphone or retrieved from a memory component of the client device 102 and included in the message 400;

[0096] · Message enhancement data 412: Enhancement data (e.g., filters, stickers, or other annotations or enhancements) representing enhancements to be applied to the message image payload 406, the message video payload 408, or the message audio payload 410 of the message 400. The enhancement data for a sent or received message 400 can be stored in an enhancement table 310;

[0097] · Message duration parameter 414: A parameter value indicating, in seconds, the amount of time that the content of the message (e.g., the message image payload 406, the message video payload 408, the message audio payload 410) is to be presented to or accessible by a user via the messaging client 104;

[0098] · Message Geolocation Parameter 416: Geolocation data (e.g., latitude coordinates and longitude coordinates) associated with the content payload of a message. Multiple Message Geolocation Parameter 416 values can be included in the payload, with each such parameter value associated with a content item included in the content (e.g., a specific image in the Message Image Payload 406 or a specific video in the Message Video Payload 408);

[0099] · Message Story Identifier 418: An identifier value that identifies one or more content collections (e.g., a "story" identified in the Story Table 314), where a specific content item in the Message Image Payload 406 of Message 400 is associated with one or more content collections. For example, the identifier value can be used to associate each of multiple images within the Message Image Payload 406 with multiple content collections;

[0100] · Message Tag 420: Each Message 400 can be tagged with multiple tags, where each such tag indicates a theme of the content included in the message payload. For example, in the case where a specific image included in the Message Image Payload 406 depicts an animal (e.g., a lion), a tag value can be included within the Message Tag 420 indicating the relevant animal. The tag values can be generated manually based on user input or can be generated automatically using, for example, image recognition;

[0101] · Message Sender Identifier 422: An identifier (e.g., a messaging system identifier, an email address, or a device identifier) that indicates the user of the client device 102 on which Message 400 was generated and from which Message 400 was sent; and

[0102] · Message Recipient Identifier 424: An identifier (e.g., a messaging system identifier, an email address, or a device identifier) that indicates the user of the client device 102 to which Message 400 is addressed.

[0103] The content (e.g., values) of the various components of Message 400 can be pointers to the locations in a table where content data values are stored. For example, the image value in the Message Image Payload 406 can be a pointer to a location within the Image Table 312 (or an address of a location within the Image Table 312). Similarly, the value within the Message Video Payload 408 can point to data stored within the Video Table 304, the value stored within the Message Enhancement Data 412 can point to data stored within the Enhancement Table 310, the value stored within the Message Story Identifier 418 can point to data stored within the Story Table 314, and the values stored within the Message Sender Identifier 422 and the Message Recipient Identifier 424 can point to user records stored within the Entity Table 306.

[0104] Glasses device

[0105] Figure 5 FIG. 1 shows a front perspective view of a glasses device 119 in the form of a pair of smart glasses according to some examples, the pair of smart glasses including an avatar control system 107. The glasses device 119 includes a body 503 that includes a front piece or frame 506 and a pair of temples 509 coupled to the frame 506 for supporting the frame 506 in a proper position on a user's face when the glasses device 119 is worn. The frame 506 may be made of any suitable material, such as plastic or metal, including a memory alloy in any suitable shape. The frame 506 may include a touch input interface configured to receive touch inputs from a user (e.g., a single finger touch, a two-finger touch, or a combination thereof, and dragging a finger along the frame 506, such as a side piece 521).

[0106] The glasses device 119 includes a pair of optical elements in the form of a pair of lenses 512, the pair of optical elements being held by corresponding optical element holders in the form of a pair of lens frames 515 that form part of the frame 506. The lens frames 515 are connected by a bridge piece 518. In other examples, one or both of the optical elements may be a display, a display assembly, or a combination of a lens and a display.

[0107] The frame 506 includes a pair of end pieces 521 that define side ends of the frame 506. In this example, various electronic components are housed in one or both of the end pieces 521. The temples 509 are coupled to the respective end pieces 521. In this example, the temples 509 are coupled to the frame 506 by respective hinge members so as to be pivotally movable between a wearable mode and a folded mode, in which the temples 509 pivot towards the frame 506 to lie substantially flat against the frame 506. In other examples, the temples 509 may be coupled to the frame 506 by any suitable means, or may be rigidly or fixedly attached to the frame 506 so as to be integral therewith.

[0108] Each of the temples 509 includes a front portion coupled to the frame 506 and any suitable rear portion for coupling to a user's ear, such as Figure 5 the curve shown in the example of FIG. 2. In some examples, the frame 506 is formed from a single piece of material so as to have an integral or monolithic construction. In some examples, the entire body 503 (including both the frame 506 and the temples 509) may be of an integral or monolithic construction.

[0109] The eyewear device 119 has on-board electronic components, including a computing device such as a computer 524 or a low-power processor, which can be of any suitable type in different examples for being carried by the body 503. In some examples, the computer 524 is at least partially housed in one or both of the temple arms 509. In the present example, various components of the computer 524 are housed in the side end piece 521 of the frame 506. The computer 524 includes one or more processors and a memory (e.g., a volatile storage device such as a random access memory or a register), a storage device (e.g., a non-volatile storage device), wireless communication circuitry (e.g., a BLE communication device and / or a WiFi direct device), and a power source. The computer 524 includes low-power circuitry, high-speed circuitry, and in some examples, the computer 524 includes a display processor. Various examples may include these elements configured differently or integrated together in different ways.

[0110] The computer 524 also includes a battery 527 or other suitable portable power source. In one example, the battery 527 is disposed in one of the temple arms 509. In Figure 5 the eyewear device 119 shown, the battery 527 is shown as being disposed in one of the end pieces 521 and is electrically coupled to the remainder of the computer 524 housed in the corresponding end piece 521.

[0111] The eyewear device 119 has a camera device function, which in this example includes a plurality of camera devices 530 mounted in one of the end pieces 521 and facing forward so as to be more or less aligned with the viewing direction of the wearer of the eyewear device 119. The plurality of camera devices 530 are configured to capture digital images (also referred to herein as digital photos or pictures) as well as digital video content. The operation of the plurality of camera devices 530 is controlled by a camera device controller provided by the computer 524, and the image data representing the images or videos captured by the plurality of camera devices 530 is temporarily stored on the memory that forms part of the computer 524.

[0112] The on-board computer 524 and the lenses 512 are configured together to provide an avatar control system 107 that presents the avatar of the user participating in the AR session, either alone or with other users, and animates the avatar based on movement information associated with the user determined by the eyewear device 119. Specifically, the lenses 512 can together display virtual content (such as AR objects) including the avatar of the user participating in the AR session as well as one or more real-world objects of the real-world environment. This makes it appear to the user that the virtual content is integrated in the real-world environment that the user views through the lenses 512.

[0113] In some examples, virtual content is received from client device 102. In some examples, virtual content is received directly from application server 114. The on-board computer 524 receives input from the user that drags or moves the avatar to a specific display location. The input can indicate whether the display location is anchored to a specific real-world object. In such a case, when the lenses 512 are moved to view different parts of the real-world environment, the avatar remains fixed at the display location of the specific real-world object, and if the lenses 512 are rotated or moved to a distance far enough from the avatar's display location, the avatar can be removed from the field of view. In some examples, the display location is not anchored, in which case when the lenses 512 are moved to view different parts of the real-world environment, the avatar display location is also updated to remain within the field of view. This enables the user to move around their surroundings and continuously see the avatar of the user with whom they are having a voice-based conversation.

[0114] The glasses device 119 includes one or more inertial measurement units (IMUs), such as accelerometers and / or gyroscopes, as well as a touch interface. Based on the input received by the glasses device 119 from the IMUs and / or the touch interface, the glasses device 119 can control the user's interaction with the virtual content. The IMUs can be used to determine the movement, rotation, speed, direction, and orientation of the wearer's head to generate 3D movement information for the wearer's avatar or some other virtual object included and displayed by the glasses device 119.

[0115] The 3D movement information can represent head movement, facial feature movement, limbs, joints, and / or other body movements. The movement information can be used by the glasses device 119 to control one or more avatars presented in the lenses of the glasses device 119. For example, if the movement information indicates movement in a specific direction and / or at a specific speed exceeding a threshold, then such movement information can then be used to animate the displayed avatar as running or to animate the displayed avatar as running rather than walking in the same specific direction. As another example, if the movement information indicates a vertical acceleration exceeding a threshold acceleration or speed and a vertical displacement or height exceeding a threshold amount, then the movement information can then be used to animate the displayed avatar as jumping or bending over or to animate the displayed avatar as jumping or bending over. As another example, if the movement information indicates a 360-degree turn along a specified curve, then the movement information can then be used to animate the displayed avatar as rotating about an axis or to animate the displayed avatar as rotating about an axis to, for example, simulate a somersault.

[0116] The eyewear device 119 also includes one or more communication devices, such as a Bluetooth Low Energy (BLE) communication interface. Such a BLE communication interface enables the eyewear device 119 to communicate wirelessly with the client device 102. Instead of or in addition to the BLE communication interface, other forms of wireless communication, such as a WiFi direct interface, may be employed. The BLE communication interface implements a standard number of BLE communication protocols.

[0117] A first communication protocol in the communication protocols implemented by the BLE interface of the eyewear device 119 enables an unencrypted link to be established between the eyewear device 119 and the client device 102. In this first protocol, the link layer communication (physical interface or medium) between the eyewear device 119 and the client device 102 includes unencrypted data. In this first protocol, the application layer (the communication layer that operates on the physically exchanged data) encrypts and decrypts the data physically exchanged in unencrypted form by the link layer via the BLE communication interface. In this way, the data exchanged through the physical layer can be freely read by eavesdropping devices, but without performing a decryption operation in the application layer, the eavesdropping devices will not be able to decipher the exchanged data.

[0118] A second communication protocol in the communication protocols implemented by the BLE interface of the eyewear device 119 enables an encrypted link to be established between the eyewear device 119 and the client device 102. In this second protocol, the link layer communication (physical interface) between the eyewear device 119 and the client device 102 receives data from the application layer and adds a first type of encryption to the data before exchanging the data through the physical medium. In this second protocol, the application layer (the communication layer that operates on the physically exchanged data) may or may not use a second type of encryption to encrypt and decrypt the data, and the link layer via the BLE communication interface physically exchanges the data in encrypted form using the first type of encryption. That is, the data can first be encrypted by the application layer and then further encrypted by the physical layer before being exchanged through the physical medium. After being exchanged through the physical medium, the data is then decrypted by the physical layer and then decrypted again by the application layer (e.g., using a different type of encryption). In this way, the data exchanged through the physical layer cannot be read by eavesdropping devices because the data is encrypted in the physical medium.

[0119] In some examples, the client device 102 communicates with the eyewear device 119 using the first protocol and / or the second protocol to exchange images or videos or virtual content between the messaging client 104 and the eyewear device 119.

[0120] Avatar control system

[0121] Figure 6FIG. 600 is a flowchart showing an example operation of an avatar control system 107 during execution of process 600 according to some examples. Process 600 may be embodied as computer-readable instructions for execution by one or more processors such that the operations of process 600 may be performed in whole or in part by functional components of the avatar control system 107; thus, process 600 is described below by way of example with reference to the avatar control system 107. However, in other examples, at least some of the operations of process 600 may be deployed on a variety of other hardware configurations. Accordingly, process 600 is not intended to be limited to the avatar control system 107 and may be implemented in whole or in part by any other component. Some or all of the operations of process 600 may be performed in parallel, out of order, or entirely omitted.

[0122] At operation 601, the avatar control system 107 accesses movement data including inertial measurement data and camera device data via an AR device, as discussed above and below.

[0123] At operation 602, the avatar control system 107 determines three-dimensional (3D) movement of the AR device based on the movement data, as discussed above and below.

[0124] At operation 603, the avatar control system 107 presents an AR object on a real-world environment being viewed using the AR device, as discussed above and below.

[0125] At operation 604, the avatar control system 107 modifies the AR object via the AR device in response to determining the 3D movement of the AR device, as discussed above and below.

[0126] Figures 7 to 9 FIG. 601 is an illustrative view and components of an avatar control system 107 according to some examples. In some examples, the avatar control system 107 captures multiple types of data from sensors of the glasses device 119. For example, the avatar control system 107 may collect multiple images from multiple camera devices of the glasses device 119. The avatar control system 107 may use one or more trained machine learning models to segment one or more real-world objects in the images. The avatar control system 107 may then select an anchor object to remove jitter from other non-visual sensors and correct for position drift of these sensors.

[0127] Specifically, as shown in a set of components 700, the avatar control system 107 can accurately measure the 3D movement of the glasses device 119 based on a combination of measurements collected from various types of sensors. For example, the avatar control system 107 can include a camera device module 710 and an IMU data module 740. The camera device module 710 can collect image data 712 depicting the real-world environment 714. The avatar control system 107 can process the image data 712 to identify one or more real-world objects. The avatar control system 107 can select a given object among the real-world objects to be used as an anchor object 720. In some cases, the avatar control system 107 selects a given object among the real-world objects based on the size and / or movement of each real-world object. That is, the avatar control system 107 can select the largest real-world object and / or the real-world object with the least amount of movement across a set of video frames as the anchor object 720.

[0128] After selecting the anchor object 720, the avatar control system 107 periodically or continuously measures the displacement 730 of the anchor object 720. The avatar control system 107 converts the displacement 730 into, for example, a 3D displacement in a 3D coordinate system. The avatar control system 107 provides the displacement 730 to the IMU data module 740. The avatar control system 107 collects or obtains 3D displacement information from the IMU data module 740. The avatar control system 107 can synchronize the IMU data module 740 with the camera device module 710 based on a comparison of the 3D displacement information obtained from the IMU data module 740 with the 3D displacement determined according to the displacement 730. In response to determining that the 3D displacement information obtained from the IMU data module 740 does not match the 3D displacement determined based on the image data 712 according to the displacement 730 within a specified threshold, the avatar control system 107 updates the 3D displacement calculation generated by the IMU data module 740 to match the 3D displacement calculation determined by the image data 712. In this way, the avatar control system 107 can accurately determine the 3D movement of the glasses device 119 and can correct the position drift of the inertial measurement data obtained by the IMU data module 740.

[0129] The glasses device 119 can generate 3D movement information based on the movement data collected by the camera device module 710 and the IMU data module 740. The glasses device 119 can convert the 3D movement information into corresponding modifications or movements of the AR objects displayed by the glasses device 119. In some cases, the glasses device 119 can apply one or more machine learning models to the 3D movement information for interpreting and / or selecting specific types of modifications and / or movements to be applied to the AR objects.

[0130] A machine learning model can be trained using labeled or unlabeled training data that includes ground truth information. For example, one or more types of 3D movements can be included in the training data along with ground truth modifications made to the AR object. The machine learning model can be applied to a subset of the training data (such as one or more types of 3D movements in the training data), and can generate an estimate or prediction of the modifications to be applied to the AR object. The estimate or prediction can be compared with the corresponding ground truth information to calculate a deviation. The deviation can then be used to update one or more parameters of the machine learning model. After updating the one or more parameters, the machine learning model is applied to another subset of the training data, and these operations are repeated until a stopping criterion is reached. This enables the glasses device 119 to be trained to accurately distinguish such 3D movements when the 3D movement data corresponding to a bending movement and the 3D movement data corresponding to a sitting movement may be very similar.

[0131] Machine learning is a field of study that gives computers the ability to learn without being explicitly programmed. Machine learning explores the research and construction of algorithms (also referred to herein as tools) that can learn from existing data and make predictions for new data. Such machine learning tools operate by constructing a model from example training data in order to make data-driven predictions or decisions that are represented as outputs or evaluations. Although examples are given for several machine learning tools, the principles presented herein can be applied to other machine learning tools.

[0132] In some examples, different machine learning tools can be used. For example, logistic regression (LR), naive Bayes, random forest (RF), neural network (NN), matrix factorization, and support vector machine (SVM) tools can be used to classify or score task releases.

[0133] Two common types of problems in machine learning are classification problems and regression problems. Classification problems (also known as categorization problems) aim to classify items into one of several categorical values (e.g., is the object an apple or an orange?). Regression algorithms aim to quantify some items (e.g., by providing a value as a real number).

[0134] Machine learning algorithms use features to analyze data to generate evaluations. Each of the features is an individually measurable attribute of the observed phenomenon. The concept of a feature is related to the concept of an explanatory variable used in statistical techniques such as linear regression. Selecting features that provide useful information, are discriminative, and are independent is important for the effective operation of an MLP in pattern recognition, classification, and regression. Features can have different types, such as numerical features, strings, and graphics.

[0135] In one example, features can have different types and can include, by way of example only, one or more of content, concepts, attributes, historical data, and / or user data.

[0136] Machine learning algorithms use training data to find correlations between features that affect an outcome or an evaluation. In some examples, the training data includes labeled data, which is known data for one or more features of an identity and one or more outcomes, such as detecting communication patterns, detecting the meaning of a message, generating a summary of a message, detecting action items in a message, detecting urgency in a message, detecting the relationship between a user and a sender, calculating a scoring attribute, calculating a message score, detecting modifications to be applied to an avatar based on 3D movement of a device, etc.

[0137] Using the training data and the features of the identity, a machine learning tool is trained in a machine learning program training. The machine learning tool estimates the values of the features when the features are related to the training data. The result of the training is a trained machine learning program.

[0138] When using the trained machine learning program to perform an evaluation, new data is provided as input to the trained machine learning program, and the trained machine learning program generates an evaluation as output.

[0139] A machine learning program supports two types of phases, namely a training phase and a prediction phase. In the training phase, supervised learning, unsupervised learning, or reinforcement learning can be used. For example, the machine learning program (1) receives features (e.g., as structured or labeled data in supervised learning) and / or (2) identifies features in the training data (e.g., unstructured or unlabeled data for unsupervised learning). In the prediction phase, the machine learning program uses the features to analyze query data to generate a result or a prediction, as an example of an evaluation.

[0140] In the training phase, feature engineering is used to identify features and can include identifying useful, discriminative, and independent features for the effective operation of a machine learning program in pattern recognition, classification, and regression. In some examples, the training data includes labeled data, which is known data for pre-identified features and one or more outcomes. Each of the features can be a variable or an attribute, e.g., a separately measurable attribute of a process, an item, a system, or a phenomenon represented by a data set (e.g., training data).

[0141] In the training phase, the machine learning program uses the training data to find correlations between features that affect a prediction or an evaluation.

[0142] Train a machine learning program during a training phase in machine learning program training using training data and identified features. The machine learning program estimates the values of the features when the features are relevant to the training data. The result of the training is a trained machine learning program (e.g., a trained or learned model).

[0143] Additionally, the training phase can involve machine learning, in which the training data is structured (e.g., labeled during a preprocessing operation), and the trained machine learning program implements a relatively simple neural network capable of performing operations such as classification and clustering. In other examples, the training phase can involve deep learning, in which the training data is unstructured, and the trained machine learning program implements a deep neural network capable of performing both feature extraction and classification / clustering operations.

[0144] The neural network generated during the training phase and implemented within the trained machine learning program can include a hierarchical (e.g., layered) organization of neurons. For example, the neurons (or nodes) can be hierarchically arranged into several layers, including an input layer, an output layer, and multiple hidden layers. Each layer in the neural network can have one or more neurons, and each of these neurons operably computes a small function (e.g., an activation function). For example, if the activation function generates a result exceeding a specific threshold, the output can be transmitted from that neuron (e.g., the sending neuron) to a connected neuron in a successive layer (e.g., the receiving neuron). The connections between neurons also have associated weights that define the influence of the input from the sending neuron to the receiving neuron.

[0145] In some examples, the neural network can also be one of several different types of neural networks including, by way of example only: a single-layer feedforward network, an artificial neural network (ANN), a recurrent neural network (RNN), a symmetrically connected neural network and an unsupervised pre-training network, a convolutional neural network (CNN), a generative adversarial network (GAN), and / or a recurrent neural network (RNN).

[0146] During a prediction phase, use the trained machine learning program to perform an evaluation. Query data is provided as input to the trained machine learning program, and the trained machine learning program generates an evaluation as output in response to the receipt of the query data.

[0147] Based on the 3D movement information determined by the glasses device 119, the glasses device 119 can update the user's avatar or AR object. For example, the glasses device 119 can animate such an avatar to be similar to the user's posture (the positioning of limbs and joints) determined based on the movement information collected by the glasses device 119. In such a case, if the user walks in a specific direction, the glasses device 119 can animate the avatar to depict the avatar as walking in the same direction. For example, if the user walks forward, the avatar can be depicted as walking forward. Similarly, if the user walks backward, the avatar can be depicted as walking backward. Similarly, if the user walks left / right, the avatar can be depicted as walking left / right.

[0148] In some cases, the glasses device 119 can interpret the 3D movement as exceeding a specific speed. For example, the glasses device 119 can determine that the glasses device 119 has been displaced in a specific direction at a specific speed. The glasses device 119 can compare the speed or rate with a threshold speed. In response to determining that the speed or rate at which the glasses device 119 is being displayed exceeds the threshold or is beyond the threshold, the glasses device 119 can animate the avatar from having a first type of movement (e.g., walking) to having a second type of movement (e.g., running or sprinting or flying). Subsequently, the glasses device 119 can compare the speed or rate with the threshold speed, and in response to determining that the speed or rate at which the glasses device 119 is being displayed now does not exceed the threshold or is not beyond the threshold, the glasses device 119 can animate the avatar from having a second type of movement (e.g., running) back to having a first type of movement (e.g., walking).

[0149] In some cases, the glasses device 119 can modify the mouse or pointer displayed as an AR object based on the 3D movement. For example, if the user is walking in a specific direction, the mouse or pointer can move in that direction to point to a real-world element or another AR object.

[0150] In some examples, the glasses device 119 can detect that the 3D movement corresponds to a vertical displacement. The glasses device 119 can determine the rate at which the vertical displacement exceeds or surpasses a threshold rate. In response, the glasses device 119 interprets the 3D movement as a jumping action and modifies the AR object to animate or perform a jumping action. For example, the glasses device 119 can determine that the vertical displacement has increased by an amount that exceeds a jump threshold amount. In such a case, the glasses device 119 interprets the 3D movement as a jumping action. As another example, the glasses device 119 can determine that the vertical displacement has decreased by an amount that exceeds a bending threshold amount. In such a case, the glasses device 119 interprets the 3D movement as a bending action. If the glasses device 119 detects a vertical displacement but determines that the vertical displacement occurs at a rate that does not exceed or surpass the threshold rate (e.g., the user is climbing a flight of stairs and not jumping), then the glasses device 119 maintains the AR object in its current form without modifying the way the avatar moves (preventing the avatar from being depicted as jumping in place).

[0151] For example, as Figure 8 shown in the user interface 800 of, the avatar 820 is presented in the real-world environment 810 that the user is viewing using the glasses device 119. The real-world environment 810 can include one or more real-world objects 812. In response to capturing and calculating the 3D movement of the glasses device 119, the glasses device 119 updates the entire body of the avatar 820 presented as an overlay on the real-world environment 810.

[0152] Specifically, the avatar 820 can be presented together with other virtual objects that are part of the AR experience being animated. The avatar 820 can be a runner running along a path (virtual road). Virtual objects (e.g., virtual obstacles) can be continuously animated as they approach the avatar 820. The user wearing or using the glasses device 119 can move their body (left, right, up, down, bend, jump, etc.) to try to avoid or capture or collect the virtual objects. For example, the user wearing or using the glasses device 119 can tilt to the left by a certain angle or walk to the left at a certain speed or rate. In response, the glasses device 119 can capture the 3D movement data corresponding to the movement and update the orientation and / or position of the avatar 820 to mimic the same movement. The glasses device 119 can keep a score based on the number of virtual objects that the avatar 820 has successfully avoided and / or collected. The score can be displayed on the lenses of the glasses device 119 through a visual representation.

[0153] In some examples, the glasses device 119 can be used to run or execute fitness activity applications, such as weightlifting, running, climbing stairs, yoga, skipping rope, etc. In such a case, the glasses device 119 can present Figure 9The user interface 900 shown in [description], the user interface includes a fitness avatar 930 on the real-world environment 910. When the user is wearing the glasses device 119 and performing a fitness activity (e.g., skipping rope), the glasses device 119 can present an indication of how close or far the user is from achieving the fitness goal. The glasses device 119 can use multiple on-board sensors of the glasses device 119 to collect or determine 3D movement information, and in response to determining the 3D movement information, the glasses device 119 updates the entire body of the avatar 930 presented as being superimposed on the real-world environment 910. The glasses device 119 can directly receive input from the user on the glasses device 119 (or indirectly receive input from the client device 102) to move or position the fitness avatar 930 at a specified location or position in 3D space.

[0154] The glasses device 119 moves the avatar 930 in a manner similar to the user's interpreted 3D movement based on the 3D movement information determined by the glasses device 119. For example, if the 3D movement information is interpreted as indicating that the user's limb 940 has moved upward by a certain amount and the user has jumped upward by a certain amount, the glasses device 119 updates the limb 932 of the avatar 930 to move upward by the same amount and updates the legs and orientation of the avatar 930 to perform a jumping motion, as well as updates the virtual rope grasped by the avatar 930. In some cases, the glasses device 119 can access data of a fitness application to identify one or more goals of the fitness activity. For example, the goal can include performing one or more repetitions, iterations, or repetitions of a certain fitness activity. In such a case, the data can specify the following corresponding to the completion of a single repetition, iteration, or repetition of the fitness activity: a specified limb 932 of the avatar 930 moves in a specific direction and jumps a certain number of times until a specific level or threshold is reached. The glasses device 119 can track the number of jumping motions of the avatar 930 performing skipping rope by updating a counter whenever the limb 932 and / or the avatar 930 is animated to perform a skipping rope activity in response to interaction data received from the real-world movement device 920. The counter can be presented on the screen or lens of the glasses device 119 so that the user can track their progress in the fitness activity.

[0155] In some cases, the fitness activity goal can correspond to maintaining a pose for a specific period or a threshold period. In such cases, the glasses device 119 can track the number of times limb 932 is lifted to a specific level or threshold and held at that position for a threshold period by updating a counter whenever limb 932 is animated to reach a specific level or threshold in response to interaction data received from the real-world mobile device 920 and held for a specific threshold period. For example, the glasses device 119 can determine when the avatar 930 is performing a predetermined pose (by matching the limbs of the avatar 930 to a predetermined configuration of the limbs), which matches the pose represented by the interaction data generated by and received from the real-world mobile device 920. In response to determining that the avatar 930 is performing a pose that matches the predetermined pose, the glasses device 119 can start a counter or a timer. Once the counter or the timer reaches a certain threshold indicating that the user has maintained the pose for a threshold period, the glasses device 119 updates an iteration counter or a repeat counter to indicate that one complete iteration of the fitness activity has been performed or completed. The iteration counter can be presented on the screen or the lens of the glasses device 119 so that the user can track their progress in the fitness activity.

[0156] The glasses device 119 can determine, for example, that when the user is performing a skipping-rope fitness activity, the movement information corresponds to the jumping action. The glasses device 119 can count the number of times the jumping action occurs within a given time interval and can update the color of the avatar 930 to indicate how close or far the user is from achieving the desired goal (specifying the number of jumps within a given interval).

[0157] Machine architecture

[0158] Figure 10is an illustrative representation of a machine 1000 within which instructions 1008 (e.g., software, program, application, applet, app, or other executable code) can be executed to cause the machine 1000 to perform any one or more of the methods discussed herein. For example, the instructions 1008 can cause the machine 1000 to perform any one or more of the methods described herein. The instructions 1008 transform the general, non-programmed machine 1000 into a particular machine 1000 programmed to perform the described and illustrated functions in the described manner. The machine 1000 can operate as a stand-alone device or can be coupled (e.g., networked) to other machines. In a networked deployment, the machine 1000 can operate in a server-client network environment as a server machine or a client machine, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 1000 can include, but is not limited to: a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smartphone, a mobile device, a wearable device (e.g., a smartwatch), a smart home device (e.g., a smart appliance), other smart devices, a web device, a network router, a network switch, a network bridge, or any machine capable of sequentially or otherwise executing the instructions 1008 specifying actions to be taken by the machine 1000. Further, while only a single machine 1000 is shown, the term "machine" shall also be taken to include a collection of machines that individually or jointly execute the instructions 1008 to perform any one or more of the methods discussed herein. For example, the machine 1000 can include a client device 102 or any one of a number of server devices forming part of a messaging server system 108. In some examples, the machine 1000 can also include both a client system and a server system, where certain operations of a particular method or algorithm are executed on the server side and where certain operations of a particular method or algorithm are executed on the client side.

[0159] Machine 1000 may include a processor 1002, a memory 1004, and an input / output (I / O) component 1038 that may be configured to communicate with each other via a bus 1040. In an example, the processor 1002 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor 1006 and a processor 1010 that execute instructions 1008. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") that may execute instructions simultaneously. Although Figure 10 multiple processors 1002 are shown, machine 1000 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0160] Memory 1004 includes a main memory 1012, a static memory 1014, and a storage unit 1016, all of which are accessible by processor 1002 via bus 1040. Main memory 1004, static memory 1014, and storage unit 1016 store instructions 1008 that implement any one or more of the methods or functions described herein. The instructions 1008 may also reside, in whole or in part, within the main memory 1012, within the static memory 1014, within the machine-readable medium 1018 within the storage unit 1016, within at least one of the processors 1002 (e.g., within the cache memory of the processor), or within any suitable combination thereof, during execution by machine 1000.

[0161] I / O component 1038 may include various components for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurement results, etc. The specific I / O components 1038 included in a particular machine will depend on the type of the machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine may not include such a touch input device. It should be understood that I / O component 1038 may include Figure 10Many other components not shown. In various examples, the I / O component 1038 can include a user output component 1024 and a user input component 1026. The user output component 1024 can include visual components (e.g., displays such as plasma display panels (PDPs), light emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), acoustic components (e.g., speakers), haptic components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. The user input component 1026 can include alphanumeric input components (e.g., keyboards, touchscreens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touchpads, trackballs, joysticks, motion sensors, or other pointing instruments), haptic input components (e.g., physical buttons, touchscreens that provide the location and force of a touch or touch gesture, or other haptic input components), audio input components (e.g., microphones), etc.

[0162] In another example, the I / O component 1038 can include: a biometric component 1028, a motion component 1030, an environmental component 1032, or a location component 1034, as well as a variety of other components. For example, the biometric component 1028 includes components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biometric signals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice recognition, retina recognition, facial recognition, fingerprint recognition, or electroencephalogram-based recognition), etc. The motion component 1030 includes acceleration sensor components (e.g., accelerometers), gravity sensor components, rotational sensor components (e.g., gyroscopes).

[0163] The environmental component 1032 includes, for example: one or more camera devices (with still image / photo and video capabilities), lighting sensor components (e.g., photometers), temperature sensor components (e.g., one or more thermometers for detecting ambient temperature), humidity sensor components, pressure sensor components (e.g., barometers), acoustic sensor components (e.g., one or more microphones for detecting background noise), proximity sensor components (e.g., infrared sensors for detecting nearby objects), gas sensors (e.g., gas detection sensors for detecting the concentration of hazardous gases for safety or measuring pollutants in the atmosphere), or other components that can provide an indication, measurement, or signal corresponding to the surrounding physical environment.

[0164] Regarding the imaging device, the client device 102 may have an imaging device system that includes, for example, a front imaging device on the front surface of the client device 102 and a rear imaging device on the rear surface of the client device 102. The front imaging device may be used, for example, to capture still images and videos of the user of the client device 102 (e.g., "selfies"), and then the still images and videos may be enhanced with the above-described enhancement data (e.g., filters). For example, the rear imaging device may be used to capture still images and videos in a more traditional imaging device mode, and these images may be similarly enhanced with the enhancement data. In addition to the front imaging device and the rear imaging device, the client device 102 may also include a 360° imaging device for capturing 360° photos and videos.

[0165] Furthermore, the imaging device system of the client device 102 may include a dual rear imaging device (e.g., a main imaging device and a depth sensing imaging device), or even a triple rear imaging device configuration, a quadruple rear imaging device configuration, or a quintuple rear imaging device configuration on the front and rear sides of the client device 102. For example, these multi-imaging device systems may include a wide-angle imaging device, an ultra-wide-angle imaging device, a telephoto imaging device, a macro imaging device, and a depth sensor.

[0166] The location component 1034 includes a positioning sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure, from which altitude can be obtained), an orientation sensor component (e.g., a magnetometer), etc.

[0167] Various techniques may be used to implement communication. The I / O component 1038 also includes a communication component 1036 that is operable to couple the machine 1000 to the network 1020 or the device 1022 via a respective coupling or connection. For example, the communication component 1036 may include a network interface component or other suitable device that interfaces with the network 1020. In another example, the communication component 1036 may include a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, components (e.g., low power consumption), components, and other communication components that provide communication via other modalities. The device 1022 may be another machine or any of a variety of peripheral devices (e.g., a peripheral device coupled via USB).

[0168] In addition, communication component 1036 can detect an identifier, or include a component operable to detect an identifier. For example, communication component 1036 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., for detecting one-dimensional barcodes such as Universal Product Code (UPC) barcodes, multi-dimensional barcodes such as Quick Response (QR) codes, Aztec codes, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCC RSS-2D barcodes, and other optical codes, and other optical sensors), or an acoustic detection component (e.g., a microphone for identifying an audio signal of a tag). In addition, various information can be obtained via communication component 1036, such as a location via Internet Protocol (IP) geolocation, a location via signal triangulation, a location via detecting an NFC beacon signal that can indicate a specific location, etc.

[0169] Various memories (e.g., main memory 1012, static memory 1014, and the memory of processor 1002) and storage unit 1016 can store one or more sets of instructions and data structures (e.g., software) implemented or used by any one or more of the methods or functions described herein. These instructions (e.g., instruction 1008), when executed by processor 1002, cause various operations to implement the disclosed examples.

[0170] Instruction 1008 can be sent or received via network 1020 using a transmission medium and using any one of a plurality of well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)) via a network interface device (e.g., the network interface component included in communication component 1036). Similarly, instruction 1008 can be sent or received using a transmission medium via an interface with device 1022 (e.g., a peer-to-peer interface).

[0171] Software Architecture

[0172] Figure 11FIG. 1100 is a block diagram showing a software architecture 1104 that can be installed on any one or more of the devices described herein. The software architecture 1104 is supported by hardware of a machine 1102 such as a machine including a processor 1120, a memory 1126, and I / O components 1138. In this example, the software architecture 1104 can be conceptualized as a stack of layers, where each layer provides a specific function. The software architecture 1104 includes layers such as an operating system 1112, libraries 1110, frameworks 1108, and applications 1106. In operation, the application 1106 activates an API call 1150 through the software stack and receives a message 1152 in response to the API call 1150.

[0173] The operating system 1112 manages hardware resources and provides common services. The operating system 1112 includes, for example, a kernel 1114, services 1116, and drivers 1122. The kernel 1114 acts as an abstraction layer between the hardware layer and other software layers. For example, the kernel 1114 provides memory management, processor management (e.g., scheduling), component management, networking, and security settings, and other functions. The services 1116 can provide other common services to other software layers. The drivers 1122 are responsible for controlling or interfacing with the underlying hardware. For example, the drivers 1122 can include a display driver, a camera device driver, or a low-power driver, a flash drive, a serial communication driver (e.g., a USB driver), a driver, an audio driver, a power management driver, and the like.

[0174] The libraries 1110 provide common low-level infrastructure used by the applications 1106. The libraries 1110 can include system libraries 1118 (e.g., C standard libraries) that provide functions such as memory allocation functions, string manipulation functions, mathematical functions, and the like. In addition, the libraries 1110 can include API libraries 1124, such as media libraries (e.g., libraries for supporting the presentation and manipulation of various media formats such as Moving Picture Experts Group 4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer 3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), graphics libraries (e.g., the OpenGL framework for rendering graphics content in two-dimensional (2D) and three-dimensional (3D) on a display), database libraries (e.g., SQLite that provides various relational database functions), web libraries (e.g., WebKit for providing web browsing functions), and the like. The libraries 1110 can also include various other libraries 1128 to provide many other APIs to the applications 1106.

[0175] The framework 1108 provides a common high-level infrastructure for use by the applications 1106. For example, the framework 1108 provides various graphical user interface (GUI) functions, high-level resource management, and high-level location services. The framework 1108 may provide a wide range of other APIs that can be used by the applications 1106, some of which may be specific to a particular operating system or platform.

[0176] In an example, the applications 1106 may include a home application 1136, a contacts application 1130, a browser application 1132, a book reader application 1134, a location application 1142, a media application 1144, a messaging application 1146, a gaming application 1148, and various other applications such as an external application 1140. The applications 1106 are programs that execute functions defined in the program. One or more of the applications 1106 can be created using various programming languages, such as object-oriented programming languages (e.g., Objective-C, Java, or C++) or procedural programming languages (e.g., the C language or assembly language). In a particular example, the external application 1140 (e.g., an application developed using an ANDROID TM or IOS TM software development kit (SDK) by an entity other than the vendor of a particular platform) can be mobile software that runs on a mobile operating system such as IOS TM , ANDROID TM , , Phone or another mobile operating system. In this example, the external application 1140 can activate an API call 1150 provided by the operating system 1112 to facilitate the functions described herein.

[0177] Glossary:

[0178] In this context, a "carrier signal" refers to any intangible medium that can store, encode, or carry transient or non-transient instructions executed by a machine and includes digital or analog communication signals or other intangible media that facilitate the communication of such instructions. Instructions can be sent or received over a network via a transient or non-transient transmission medium and using any of a number of well-known transmission protocols.

[0179] In this context, a "client device" refers to any machine that interfaces with a communication network to obtain resources from one or more server systems or other client devices. A client device can be, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a PDA, a smartphone, a tablet computer, a Ultrabook, a netbook, a laptop, a multiprocessor system, a microprocessor-based or programmable consumer electronics product, a gaming console, a set-top box, or any other communication device that a user can use to access a network.

[0180] In this context, a "communication network" refers to one or more portions of a network, which can be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a portion of the Internet, a portion of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, a network, other types of networks, or a combination of two or more such networks. For example, a network or a portion of a network can include a wireless network or a cellular network, and the coupling can be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other types of cellular or wireless couplings. In this example, the coupling can implement any of a variety of data transfer technologies, such as single-carrier radio transmission technology (1xRTT), evolved data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rate GSM evolution (EDGE) technology, the third generation partnership project (3GPP) including 3G, the fourth generation wireless (4G) network, universal mobile telecommunications system (UMTS), high-speed packet access (HSPA), worldwide interoperability for microwave access (WiMAX), long term evolution (LTE) standards, other data transfer technologies defined by various standards-setting organizations, other long-distance protocols, or other data transfer technologies.

[0181] In this context, a "transient message" refers to a message that is accessible for a time-limited duration. A transient message can be text, an image, a video, etc. The access time of a transient message can be set by the message sender. Alternatively, the access time can be a default setting or a setting specified by the recipient. Regardless of the setting technique, the message is temporary.

[0182] In this context, a "machine-readable medium" refers to a component, device, or other tangible medium capable of storing instructions and data temporarily or permanently, and can include, but is not limited to, random access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical media, magnetic media, cache memory, other types of storage devices (e.g., erasable programmable read-only memory (EEPROM)), and / or any suitable combination thereof. The term "machine-readable medium" should be understood to include a single medium or multiple media capable of storing instructions (e.g., a centralized or distributed database, or associated caches and servers). The term "machine-readable medium" should also be considered to include any medium or combination of media capable of storing instructions (e.g., code) executable by a machine such that the instructions, when executed by one or more processors of the machine, cause the machine to perform any one or more of the methods described herein. Thus, a "machine-readable medium" refers to a single storage device or apparatus, as well as a "cloud-based" storage system or storage network comprising multiple storage devices or apparatuses. The term "machine-readable medium" does not include the signal itself.

[0183] In this context, a "component" refers to a device, physical entity, or logic having boundaries defined by function or subroutine calls, branch points, APIs, or other techniques that provide partitioning or modularization of a particular processing or control function. Components can be combined with other components via their interfaces to perform machine processing. A component can be an encapsulated functional hardware unit designed to work with other components and can be part of a program that typically performs a particular function among related functions. Components can constitute software components (e.g., code implemented on a machine-readable medium) or hardware components. A "hardware component" is a tangible unit capable of performing certain operations and can be configured or arranged in some physical manner. In various examples, one or more computer systems (e.g., a stand-alone computer system, a client computer system, or a server computer system) or one or more hardware components of a computer system (e.g., a processor or group of processors) can be configured by software (e.g., an application or part of an application) to operate as a hardware component performing certain operations as described herein.

[0184] Hardware components can be implemented mechanically, electronically, or in any suitable combination thereof. For example, a hardware component can include dedicated circuitry or logic that is permanently configured to perform certain operations. A hardware component can be a dedicated processor, such as a field programmable gate array (FPGA) or an ASIC. A hardware component can also include programmable logic or circuitry that is temporarily configured by software to perform certain operations. For example, a hardware component can include software executed by a general purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a particular machine (or a particular component of a machine) uniquely customized to perform the configured functions, rather than a general purpose processor. It will be appreciated that a decision can be made, for cost and time considerations, as to whether to implement a hardware component mechanically in dedicated and permanently configured circuitry or in circuitry that is temporarily configured (e.g., by software). Thus, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity, i.e., an entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate in some manner or to perform certain operations described herein. Considering examples in which a hardware component is temporarily configured (e.g., programmed), each of the hardware components need not be configured or instantiated at any one time. For example, in the case where a hardware component includes a general purpose processor that is configured by software to become a dedicated processor, the general purpose processor can be configured separately as different dedicated processors (e.g., including different hardware components) at different times. Thus, software configures one or more particular processors to, for example, constitute a particular hardware component at one time and different hardware components at different times.

[0185] Hardware components can provide information to, and receive information from, other hardware components. Accordingly, the described hardware components can be considered to be communicatively coupled. In cases where multiple hardware components are present simultaneously, communication can be achieved via signal transmission (e.g., via appropriate circuitry and buses) between or among two or more of the hardware components. In examples where multiple hardware components are configured or instantiated at different times, such communication between the hardware components can be achieved, for example, by storing information in a memory structure accessible to the multiple hardware components and retrieving the information from the memory structure. For example, one hardware component can perform an operation and store the output of the operation in a memory device communicatively coupled thereto. Then, another hardware component can access the memory device at a subsequent time to retrieve the stored output and process it.

[0186] The hardware component can also initiate communication with an input device or an output device and can operate on resources (e.g., a collection of information). Various operations of the example methods described herein can be performed, at least in part, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors can constitute a processor-implemented component operative to perform one or more of the operations or functions described herein. As used herein, a "processor-implemented component" refers to a hardware component implemented using one or more processors. Similarly, the methods described herein can be at least in part processor-implemented, where a particular one or more processors are examples of hardware. For example, at least some of the operations of the method can be performed by one or more processors or processor-implemented components. Additionally, one or more processors can also be operative to support the execution of relevant operations in a "cloud computing" environment or operate as "software as a service" (SaaS). For example, at least some of the operations can be performed by a group of computers (as examples of machines including processors), where the operations are accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., an API). The execution of certain operations of the operations can be distributed among the processors and can be deployed not only within a single machine but also across multiple machines. In some examples, the processor or processor-implemented component can be located in a single geographical location (e.g., within a home environment, an office environment, or a server farm). In other examples, the processor or processor-implemented component can be distributed across multiple geographical locations.

[0187] In this context, a "processor" refers to any circuit or virtual circuit (a physical circuit simulated by logic executed on an actual processor) that manipulates data values in accordance with control signals (e.g., "commands", "opcodes", "machine codes", etc.) and produces corresponding output signals for operating a machine. For example, a processor can be a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), or any combination thereof. A processor can also be a multi-core processor having two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously.

[0188] In this context, a "timestamp" is a sequence of characters or encoded information that identifies when an event occurred, e.g., so as to give a date and time of day, sometimes accurate to a fraction of a second.

[0189] Changes and modifications can be made to the disclosed examples without departing from the scope of the present disclosure. These and other changes or modifications are intended to be included within the scope of the present disclosure as expressed in the appended claims.

[0190] Modules, components, and logic

[0191] Certain examples are described herein as including logic or multiple components, modules, or mechanisms. A module can be a software module (e.g., code implemented on a machine-readable medium or in a transmitted signal) or a hardware module. A "hardware module" is a tangible unit capable of performing certain operations and can be configured or arranged in a particular physical manner. In various examples, one or more computer systems (e.g., a stand-alone computer system, a client computer system, or a server computer system) or one or more hardware modules of a computer system (e.g., a processor or a group of processors) are configured by software (e.g., an application or a portion of an application) to operate as a hardware module that performs certain operations described herein.

[0192] In some examples, the hardware module is implemented mechanically, electronically, or in any suitable combination thereof. For example, the hardware module can include dedicated circuitry or logic that is permanently configured to perform certain operations. For example, the hardware module can be a dedicated processor, such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). The hardware module can also include programmable logic or circuitry that is temporarily configured by software to perform certain operations. For example, the hardware module can include software contained within a general-purpose processor or other programmable processor. It will be appreciated that the decision to implement the hardware module mechanically in dedicated and permanently configured circuitry or in temporarily configured (e.g., software-configured) circuitry can be made based on cost and time considerations.

[0193] Accordingly, the phrase "hardware module" should be understood to include a tangible entity, i.e., an entity that is physically constructed, permanently configured (e.g., hard-wired), or temporarily configured (e.g., programmed) to operate in a certain manner or perform certain operations described herein. As used herein, "hardware-implemented module" refers to a hardware module. Considering examples in which a hardware module is temporarily configured (e.g., programmed), each of the hardware modules need not be configured or instantiated at any given time. For example, in the case where a hardware module includes a general-purpose processor that is configured by software to become a dedicated processor, the general-purpose processor can be configured to be different dedicated processors (e.g., including different hardware modules) at different times. Thus, software can configure one or more specific processors to, for example, constitute a particular hardware module at one time and different hardware modules at different times.

[0194] Hardware modules can provide information to other hardware modules and can receive information from other hardware modules. Thus, the described hardware modules can be considered to be communicatively coupled. In the presence of multiple hardware modules, communication can be achieved through signal transmission between or among two or more of the hardware modules (e.g., via appropriate circuitry and buses). In examples where multiple hardware modules are configured or instantiated at different times, communication between or among such hardware modules can be achieved, for example, by storing and retrieving information in a memory structure accessible to the multiple hardware modules. For example, one hardware module performs an operation and stores the output of the operation in a memory device to which it is communicatively coupled. Then, another hardware module can access the memory device at a later time to retrieve the stored output and process it. Hardware modules can also initiate communication with input or output devices and can operate on resources (e.g., collections of information).

[0195] The various operations of the example methods described herein can be performed, at least in part, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors constitute processor-implemented modules that are operative to perform one or more of the operations or functions described herein. As used herein, "processor-implemented module" refers to a hardware module implemented using one or more processors.

[0196] Similarly, the methods described herein can be implemented, at least in part, by processors, where one or more particular processors are examples of hardware. For example, at least some of the operations of the method can be performed by one or more processors or processor-implemented modules. Additionally, one or more processors can also be operative to support the execution of the relevant operations in a "cloud computing" environment or to operate as "software as a service" (SaaS). For example, at least some of the operations can be performed by a group of computers (as an example of machines including processors), where the operations are accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., APIs).

[0197] The execution of certain operations can be distributed among the processors, which can reside not only within a single machine but also be deployed across multiple machines. In some examples, the processors or processor-implemented modules are located in a single geographical location (e.g., within a home environment, an office environment, or a server farm). In other examples, the processors or processor-implemented modules are distributed across multiple geographical locations.

Claims

1. A method, comprising: Access mobile data including inertial measurement data and camera device data through an augmented reality (AR) device; Determine the three-dimensional (3D) movement of the AR device based on the mobile data; Present an AR object on the real-world environment being viewed using the AR device through the AR device; And Modify the AR object through the AR device in response to determining the 3D movement of the AR device.

2. The method according to claim 1, wherein, The mobile data includes gyroscope data.

3. The method according to any one of claims 1 to 2, wherein, The AR device includes an AR glasses device, and the method further includes: Determine the orientation, rotation, position, and velocity of the AR device in 3D space as the 3D movement of the AR device.

4. The method according to any one of claims 1 to 3, further comprising: Update one or more body parts of an avatar based on the 3D movement of the AR device.

5. The method according to any one of claims 1 to 4, further comprising: Synchronize the inertial measurement data with image data including the camera device data obtained from multiple camera devices; And Correct the position drift of the inertial measurement data in response to synchronizing the inertial measurement data with the image data.

6. The method according to claim 5, further comprising: Identify an anchor object in the image data; Periodically determine the movement of the position of the anchor object in the image data; And Synchronize the movement detected by the inertial measurement data with the movement of the position of the anchor object in the image data.

7. The method according to any one of claims 1 to 6, further comprising: Determine that the 3D movement of the AR device corresponds to a movement in a specific direction; And Move the AR object in the specific direction in response to determining that the 3D movement of the AR device corresponds to the movement in the specific direction.

8. The method according to claim 7, wherein, The specific direction includes at least one of moving forward, backward, left, or right.

9. The method according to any one of claims 1 to 8, further comprising: Determine that the 3D movement of the AR device corresponds to a movement at a specific speed; And Move the AR object in a direction at the specific speed in response to determining that the 3D movement of the AR device corresponds to the movement at the specific speed.

10. The method according to claim 9, further comprising: Determine that the movement at the specific speed exceeds a threshold speed; And Change the movement mode of the AR object from a first movement mode to a second movement mode in response to determining that the movement at the specific speed exceeds the threshold speed.

11. The method according to claim 10, wherein, The first movement mode includes walking, and wherein the second movement mode includes running, sprinting, or flying.

12. The method according to any one of claims 1 to 11, further comprising: Determine that the 3D movement of the AR device corresponds to a movement at a specific vertical height;And Move the AR object along the specific vertical height in response to determining that the 3D movement of the AR device corresponds to the movement at the specific vertical height.

13. The method according to claim 12 further comprises: Determine that the movement at the specific vertical height exceeds a threshold height and a threshold rate, wherein the specific vertical height is reached at the threshold rate; And Select a movement mode of the AR object in response to determining that the movement at the specific vertical height exceeds the threshold height and the threshold rate, wherein the specific vertical height is reached at the threshold rate.

14. The method according to claim 13, wherein The movement mode includes jumping or dodging.

15. The method according to claim 13 further comprises: Determine that the movement at the specific vertical height exceeds the threshold height and does not exceed the threshold rate; And Prevent the selected movement mode from being applied to the AR object in response to determining that the movement at the specific vertical height exceeds the threshold height and does not exceed the threshold rate.

16. The method according to any one of claims 1 to 15 further comprises: Determine that the 3D movement of the AR device corresponds to a 360-degree turn along a specified curve; And Rotate the AR object about an axis in response to determining that the 3D movement of the AR device corresponds to a 360-degree turn along the specified curve.

17. The method according to any one of claims 1 to 16, wherein The AR device is configured to apply one or more machine learning models to at least one of the 3D movement or the movement data to estimate a modification to be applied to the AR object.

18. The method according to any one of claims 1 to 17, wherein The AR object includes an avatar representing a user, and the method further includes: Determine an exercise goal associated with a fitness activity; and Animate the avatar to represent the achievement of the exercise goal based on the determined 3D movement of the AR device.

19. A system comprising: A storage device of an augmented reality (AR) device; And A processor coupled to the storage device and configured to perform operations including: Access movement data including inertial measurement data and camera device data through the AR device; Determine a three-dimensional (3D) movement of the AR device based on the movement data; Present an AR object on a real-world environment being viewed using the AR device through the AR device; and Modify the AR object through the AR device in response to determining the 3D movement of the AR device.

20. A non-transitory machine-readable storage medium, the non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations, the operations comprising: Access movement data including inertial measurement data and camera device data through an augmented reality (AR) device; Determine a three-dimensional (3D) movement of the AR device based on the movement data; Present an AR object on a real-world environment being viewed using the AR device through the AR device; And Modify the AR object through the AR device in response to determining the 3D movement of the AR device.

Images